The problem with adding vit E oil to soap is more about the type of vit E oil you are using. If it's from a vitamin capsule, it's probably more pure than if you are using the kind sold for rubbing onto the skin that has soybean or another carrier oil with only a small amount of vitamin E added. If the latter is used, it's mostly the carrier oil that is added and, yes that will saponify.
Take
this product for example. It is sold as vitamin E oil, but in addition to tocopherol, it also contains the following oils: sunflower, grapeseed, sesame, wheatgerm, lemon. I believe lemon oil is really and essential oil, but the others all have SAP values and will saponify.
When you say you poured the vitamin E oil thinking it was your Castor oil. Does that mean you left out your Castor oil altogether?