Doing it in the privacy of your own home is one thing, but using speaker phone when you’re out and about is considered a breach of the social contract by many.
The reasons are obvious — few of us like hearing other people’s conversations, with rare exceptions applying to gossip-lovers hoping to eavesdrop on something scandalous. It’s usually something far more mundane, however. Even so, speaker phone can be more convenient than holding your iPhone right up to your ear, especially on longer calls. Even if we know there’s no harm in it, sometimes it just *feels* like there is. Whether that justifies loudly talking about your personal life in the grocery store is up to you. And really, is it any different from hearing someone talking aloud to someone next to them?