I'm sorry for your frustrations! You are brave to approach people though, I am sometimes too nervous and bad at conversation to approach people.
From the people I have mustered up the courage to contact about wholesale, their main concern, as you know, it that they are not sure of the market and do not want to risk the investment.
For this reason I have devoloped a "trial order deal" where I sell the items at a very low price (or even a free trial depending on your costs), to reduce the risk and do not impose minimums.
You could also give them a small trail display of your products for free, and if they don't sell within a set period of time (like a month) then you can get your products back - and if they do sell then hopefully they will want to make a regular purchase! I guess it is an investment for future sales, even if you may take a risk to start. You never know, it may pay off, and worse case you might loose a few soaps, but at least you will get some exposure.