You talk about the soap leaving a residue on your skin, but you're not too clear what you mean by that. Would be nice to have a better idea of what you mean.
Honestly, soap shouldn't leave a residue on the skin. It can clean more strongly or it can clean more mildly, and that can alter how your skin feels after bathing. But an actual residue, like something sticky on the skin? Not so much.
What can create a residue is if you use lye-based soap in hard water. The soap reacts with the hard-water minerals to form soap scum that sticks to the skin. Is that maybe what is happening in your situation?
If so, the solution is to soften the water. The usual solution is to use a whole-house water softener.