The sources that link perilla and atopic dermatitis are focusing on water based extracts. Do not assume the oil has the same properties as an aqueous extract.
A soap made with the oil is not going to be as effective as a leave on product such as a lotion, not to mention that lye can easily destroy the active ingredients.