Comfrey is a wonderful herb and a beautiful color and pleasant scent but the scent or color wouldn't really stick. You would need to do as spaceship mentioned and do a double infuse which IMO would be timely and pricey. I just use my infused oil with a bit of natural color if I'm going totally natural with no micas and such. I really only use herbal infusions such as calendula, chamomile, lavender, etc. for soaps but I do have comfrey infusion for other things.
I use OO for my infusions and substitute all the OO in the recipe for the infused. Try to do a recipe with around 40% or more for best shot at anything pulling through. And use EVOO to infuse as it helps with the color.