Litsea has a lemony sort of scent to begin with, so it's fine as long as you don't go nuts with it. The patch/orange soap I smelled didn't smell like patchouli at all to me, it just smelled like orange with a tiny touch of "something" deeper to it. I'm not a fan of patch but liked the soap very much, go figure.
you can anchor stubborn scents with castor oil as well. i generally put my citrus scents directly into a jar with 2 oz. castor oil and let that sit while i put my other ingredients together. i lightly scent to begin with and i can still smell my cirtus scents for a long time after.