Generally speaking, I'm thinking that certain herbal substances are water soluble while others are oil soluble.
Haven't used chamomile, but these days I'm experimenting with mint, doing both - water and oil infusion. And maybe even a double infusion, if I find the time.
Worth considering...