Did you zap test this soap? That will confirm whether the soap is lye heavy or not. At this point, it sounds like the soap very well might be, but if it's not, you could be fixing a problem that doesn't exist.
Since you already diluted the soap, it may be tough to add fats to use up the excess KOH. I'd probably use boric acid or citric acid to neutralize instead. If you use citric acid, remember it is very easy to over neutralize with citric acid and the neutralization process can take a bit of time, so be patient. Over neutralizing will create a whole new set of problems.
To neutralize, I'd make a solution of, oh, say about 20% citric acid in water. Neutralize a sample of the diluted soap. Scale up the dosage needed for the sample to estimate the amount of solution needed to neutralize the rest of the soap. I'd not add the whole amount of solution, however. Add 1/2 to 3/4 of what you think you need and see how that works. You can always add more acid if needed.