What is your SF? I find that has a huge effect on how conditioning my soap is (soaps do not moisturize, BTW). I use 8-10%, 5% is too drying for me.
I would reduce the castor to 5-7%. More than that may make your finished soap sticky.
To get the beneficial properties of any oil, I'd use at least 5%. So I would up the babassu. Babassu is very similar to coconut oil chemically. For what it's worth, I don't see much difference between the two in CP soap but I do see a difference in leave on products like lotion and salves.