I don't see why anyone should be angry unless you're trying to misrepresent your product. As most people said proper labeling should take care of it.
You'll never make everyone happy. Some people are strict vegan (no products from any animal/insect source. ie beeswax, milk, etc), some are simply vegetarian, some only want organic ingredients, some don't like the use of palm. It's up to them to find the products they want to support. The best you can do is to make it simple for them to do so with your labeling and a good knowledge of your ingredients.