You can add it whenever, as the lye monster keeps eating after the soap is poured. With EOs that accelerate or rice, I add them into the oils so they are affecting the batter immediately, rather than turning the soap solid after trace.
When I make HP, I wait until the soap is around 120 degrees then add the EOs.