When I first started out I bought supplies mostly locally. EOs in small quantities, however, are HORRENDOUSLY expensive, so I quickly switched to buying online. I still buy BOGOS on oils when I see a good deal, but buy bulk oils primarily online. I bought lye at my local hardware store for years, but it was too often really static-ey, and I was always nervous about the 'flying lye' syndrome and having to mop the whole kitchen with every batch. I now buy it through Essential Depot and have had no problem with static.
When I was finally ready to sell, I looked at the profit margins of buying online vs. locally, and sadly concluded I couldn't come out ahead with local supplies. Even with shipping, it's much more cost-effective for me to buy online. My usual go-tos are Soapers Choice for oils and butters; Essential Depot for lye and some EOs; New Directions Aromatics for EOs, clays and a lot of other stuff, love them; Peak and Nature's Garden for FOs; Peak for SFIC glycerine base; and Brambleberry when I want treats I can't find elsewhere. I used to buy a lot of EOs from Camden Grey, but shipping's really high for me. I also look for deals on Ebay, there are some really good sellers there.