I've noticed this, particularly with olive soap that isn't aged enough! If it's over 50% olive, I'm fine with it as long as it cured a year.
Castile I never like, but I can tolerate it at the 2 year mark. At that point, it's certainly not drying, but I dislike the stringy, snotty lather...