Yes, it will all fade away, and they will all turn some shade of brown.
If you want to try herbal teas, I strongly recommend you try rebatching, I've just done it with a berries infusion and the smell and colour stayed in the soap, which now has a wonderful texture and colour (and smell!).