My experience with silicone molds and fragrance is that although the mold retains the scent of the last soap made, it doesn't transfer.
If I make peppermint soap in a silicone mold, even after washing the mold with hot, soapy water, the mold smells like peppermint.
However, if I next make lavender soap in that same mold, then the mold smells like lavender. And, the lavender soap has no trace of peppermint.
The only time it really bothers me is if I've made licorice soap, as anise EO makes me gag (it's a popular soap for me, though, so I just hold my nose and go). I try to make another soap quickly in that mold to get ride of the lingering anise smell.
Anita