Depending on the EO's that I am using, I generally use 1oz per lb total batch weight if they are
milder scents, and 1oz per lb of oils if they are
stronger scents. Your 'scent to batch' volume ratio is relatively low. I would think you will be just fine. I have never had a customer come back and say they could not use the soap due to irritation. Most sensitivity arises from chemicals in a product, not the scent, (although there are the exceptions, and FO's
are chemical based) but even then, the ratio is so low for scent to batch it really doesn't create a problem unless someone is hyper-sensitive. This knowledge comes to me via of my cosmetic R&D background.
Even the mildest of commercial baby soaps have scent. People just generally blame the scent because it's the most tangible. They don't think about all the chemicals that the bars contain because they can not see or smell 'them'.
And that's where we come in
Not to worry, I think your soap sounds yummy and will be just fine.
Sygrid