I typically stir my color in with a spatula, making sure it has been pre-mixed well with some of my liquid oils. Too much stick-blending will cause a thicker trace which is typically not desired when working with a swirl.
You can stick-blend though. I have seen many use the same blender, not rinsing or cleaning in any way. Usually, the small amount left on the blender won't cause a huge change in color. Just go from light colors to darker colors.
EDIT: Sorry, just realized you were talking about multiple batches, not just one batch with multiple colors. Could you wipe with paper towels between batches?