As has been mentioned, it's not just about letting it dry, there's chemical changes going on for several weeks, which cause the bar to become milder, lather better, and just overall become a better bar of soap.
Additionally, rushing the drying process can cause uneven drying, warping, cracking, etc.