This exact thing is is happening to me too. It feels like an exfoliant. I made 2 batches of the same recipe just different additives and essential oils. The recipe has 10% butter and another 30% hard oils.
The grainy one was SF at 5% lye C 27% and I took out 50% of the water and used cream...