I don't recall having any adverse effects other than staining my clothing when gathering and arranging lilies. I grew up with calla lilies in our yard and my mom loved them in vases in the house. I have always enjoyed living Easter Lilies because they smell so nice and always are plentiful around my birthday. But I think Easter Lilies is one of the lily family not toxic. I don't recall; I could be wrong. I have had many different kinds of lilies growing in my yards over the years and have never had any adverse reaction with my skin from any of them. Now poison oak is another matter. Stinging nettles, another matter. Those two plants when they grew in my yard really played havoc with my skin.
But I never used any of the above in soap, so can't really address the issue of irritability after being subjected to lye.
P.S. Yes, I know stinging nettle is safe to eat (although not while quite raw and stinging), and makes a delicious soup. I like it in soup and have had it in tea, as well.