Mixing Micas and Oxides
I mix my mica up with glycerin and my oxides with a bit of oil or melted butter. If I'm not adding glycerin or oxides, then I agree that mixing with the fragrances generally works very well.
If it's melt and pour, most likely it is from the MP base drying out. I find it happens with bases that contain SLS or SLES. You can add a bit of glycerin or oil with the base, remelt it, and it will be fine. It's not mold.
It sounds to me as if you are using too much color. Another way to tell is lather the soap. If the bubbles are not white, then you are using too much colorant. Hope this helps!