Since I've used wheat germ oil in soaps without any problem and at a higher percentage than you did, I'll say it isn't the wheat germ oil. We use wheat germ oil for food (packaged under nitrogen, always refrigerated). When I have extra, I add to soaps.
I think the problem with the recipe is the amount of milk that was added. You are at ~3:1 milk to lye. It's way too much liquid. Plus it could have added significantly to the superfat if you used whole milk. It lowered the pH in your cured soap down to a level that allowed the mold to grow (below 9.0). It also added other edibles.
Check the pH of the soap if you have the right pH strip.
The only change I would have made to your recipe is to reduce the liquid volume and use less milk (100 to 150 grams and use water for the rest). Better yet, skip the milk and use buttermilk instead. The combination of oils you used makes a lovely soap. I'm sorry you lost this batch after a long wait.
If you are using wheat germ, you need to grind it super fine in order for the lye to do its job.