Of course I wouldn't use latex mats myself because of my sensitivity to latex, but I would be concerned of the potential for latex deterioration leading to latex adhering to the product you are making. Latex allergies are certainly a problem for those who have them, and I would not want to come into contact with latex on my soap.
Latex rubber does deteriorate over time from light, ozone, heat. So if you sell, these may not be the best investment because latex allergies are more and more prevalent in the population, especially among healthcare workers and rubber industry workers.
If you choose to use them, I would strongly advise you to put a warning label to the effect of, "Caution: this soap has come in contact with latex".