I use fresh raw goat milk. No billy goat smell, but my unscented goat milk castile soaps have a milk scent. Very similar to powdered dried milk, the one that is good for camping and emergencies.
Most people do not object, and the goat owner loves all of my goat milk soaps (I trade her soap for milk&yogurt). However, I have had customers object to the way scents get 'modified' by the milk scent, and my vegan customers stay away too, of course.
I personally prefer other types of soap (beer, I am looking at you) but I am happy to make goat milk soaps for my husband, sister, and goat owner friend/trade partner. I freeze it into ice cubes, keep oils and lye/milk at room temperature, and I get a creamy white soap (I do not gel my goat milk castiles).