For me, 4 weeks is still a little too soon. The difference between 4 and 6 weeks on my soaps is significant, and ideally I wait 8 weeks (I know, most people don't like to wait that long). Your soaps will become more mild, but let a bar or two sit for 6 weeks before you try it and see if that helps.
Another possibility is ash. Do you have any visible ash on your soaps? You could be getting some very fine sodium carbonate, which is quite caustic. That would explain why it would be drying the first time but not after that. Just a thought.
I normally superfat at 5% (other than salt bars). I find that is sufficient for me for a well-balanced bar and I prefer the better lather over that of a higher superfat, but a lot of people do superfat higher, and you may be one of the people that prefer it, so you can try that as well.