I don't know why it would ruin clothes since the FO amount is going to be so small in comparison to the amount of water used for the wash. But I'm not sure how well the scent would remain in the clothes. If you want your clothes scented, it might be more economical to make sachets to put in your drawers and to hang in your closet. Oddly enough, I was just talking to a friend the other day who asked me if it would be okay to put some of the soap I gave her into her clothes drawers. I told her she could but she might want to wrap them loosely in cotton or muslin so they wouldn't be against any clothes. The bonus to this is the soap will also get a little harder and hopefully, last a little longer.
eta: I forgot to mention - I used to use unscented dryer sheets with a dab of peppermint EO on them. I'd toss it into the dryer and it did give the clothes a bit of a minty scent. I stopped doing it because I was getting low on it. It would probably be more effective if you waited until the cool down so the EO wouldn't be exposed to heat.