There are other Boudin locations, but quite a few are called "Boudin SF". The one in the basement of Macy's Union Square can sometimes be as busy as the main location at the demonstration bakery in Fishermans Wharf. The prices might even be a little bit less. My usual order is the "Best of Boudin" with clam chowder in a bread bowl (upcharge for the bread bowl) and a half sandwich. Often I can't finish it and take the sandwich home.
When I was working a summer job in San Francisco decades ago, I loved Boudin, which had a couple of (now gone) downtown locations. I might even get a sourdough loaf to take home.
For anyone visiting San Francisco, the Ferry Building is pretty nice. A lot of places there are super expensive, but I rather like Acme Bread, which is based in Berkeley and where this is their only retail location outside of their original location in Berkeley. This location usually has sandwiches. And there should be a farmers market on Saturdays where there will be lots of food booths.
And for the OP, there are some seedy parts of San Francisco. It's really odd because the notorious Tenderloin neighborhood of San Francisco is right next to Union Square where many of the nicest hotels are. I don't think it's particularly dangerous for tourists to accidentally end up there, but it's not a particularly pleasant thing. It's kind of been like that for decades. I'd just do a little bit of research about the Tenderloin in order to avoid it. That being said, some of the most interesting restaurants in San Francisco are there. But I'm a local and I kind of just accept it for what it is.