I was surprised at these responses and ran upstairs to inspect my latest bar of orange soap complete with ground orange peel and a quarter slice of orange on top. Brown? Hmmm - maybe, or just dark orange - what's the difference? In my orange coloured soap (using 5 fold orange oil) it just looks like darker spots of orange. I also don't find it particularly scratchy. Madder root and rosemary leaves are scratchy. I guess it is all personal choice. I use orange peel in my soap to add a bit to the colour, provide a bit of texture, and I also mix my essential oils into it before adding to the final mixture. I have this theory that this helps "grab" the orange aroma and hold it in the soap. I couldn't conclusively say whether it does or not, but I have noticed my soaps do revive in use if they lose their scent in storage but this might happen anyway.
If you want a more intense orange colour without orange peel I have successfully used paprika. You can add at trace, but it will speckle the soap, I don't think it is very scratchy as its too fine. Or I have successfully added to the lye and sieved out prior to mixing with the oils. Gives a stronger slightly pinky orange more intense than just using folded oils. You only need a small amount. I think I used about a teaspoon to 600g oils weight so probably less than 1%