Edit: So, okay, I see you're asking about adding fragrance to finished liquid soap.
Yes, fragrance can do unpredictable things in liquid soap. It can thin, thicken, clump or do nothing. It's something you have to check when scenting liquid soap; there's no way to know this in advance. So next time, test the fragrance in a sample of the soap, not scent the whole batch. And then decide what you want to do from there.
Yes, if the fragrance thickens the soap, you need to dilute the soap more.
Another issue is 3-4% fragrance is likely too much scent for liquid soap -- try 0.5% to 1% and increase from there.
What do you mean that you used a "modifier"? The only modifier I can think of is the vanillin stabilizer people use to minimize discoloration caused by the vanillin that is in some fragrances. This stabilizer won't modify the behavior of the fragrance in the soap, however.
Original post: Fragrance can accelerate the rate of saponification in soap depending on the ingredients in the fragrance. Did you buy from a reputable fragrance supplier? Who did you buy the fragrance from? What is the fragrance? Do you know if they tested the fragrance for how it performs in cold process soap making? Did you read the reviews from customers to determine how the fragrance behaves in their soap?
Also how you make the soap can affect how your fragrance behaves. What is your recipe? How do you mix the ingredients? What was the temperature of your soap batter (or your starting ingredients)