That's why I let the towels sit for a few days before washing them. That way, there is little to no residual oil or grease - just saponified soap streaks.
For me, that works for the soap batter with a low SF, but not for the higher SF batter, and not for the vessels that have mostly oil. When I re-package my oils (large containers to smaller containers), those vessels never have soap batter in them, so it's only oil, and I do clean them prior to re-use after they are emptied, of course.