We run a pipeline that ingests BLE beacons from a few hundred sites into an MQTT broker and fans them out to a handful of consumers. It’s not a large deployment in absolute terms, but it’s large enough that the naïve approach breaks.
Here’s what actually matters once you get past the hello-world.
Connection churn is the enemy
Our first outage, back in 2023, was caused by gateway firmware that reconnected to the broker every time it lost network for more than 30 seconds. The site had flaky LTE. The broker saw hundreds of connect/disconnect pairs per minute. Our auth provider rate-limited us. Everything stopped.
The fix was not more broker capacity. The fix was gateway firmware that keeps the TCP connection alive through brief network events and buffers messages locally when it can’t reach us. MQTT’s Keep Alive is not enough — you need your own supervision loop.
QoS 1 is almost always right
If you’re reading about MQTT, you’ve probably seen the three QoS levels. In practice:
- QoS 0 is fine for telemetry you’ll happily lose a few seconds of. Don’t use it for anything with “event” in the name.
- QoS 1 is the default for almost everything. At-least-once with idempotent consumers.
- QoS 2 is almost never worth the roundtrips. If you think you need it, you probably need a different message pattern entirely.
Our whole pipeline is QoS 1, and every consumer treats duplicate messages as a non-event. That’s cheaper than coordinating exactly-once delivery at the protocol level.
The retained message is a foot-gun
Retained messages are wonderful for status topics (“am I alive?”). They are terrible for event streams. If a new consumer subscribes to a topic with a retained message, they will immediately receive an event that already happened. Treat retained messages like the last-known-state of a thing, never like a log entry.
We found this out when a backfill job replayed a month of retained messages into our alert consumer. The alert consumer paged an on-call engineer 3,000 times in 40 seconds.
Instrument the broker, not just the app
Our most valuable dashboard isn’t Fleet. It’s the one that shows broker-level stats: message rate per topic, connection count, queue depth per client, TLS handshake failures. When something feels weird, that’s the first place we look — before the application logs, before the database, before anything else.
The broker is the choke point. Measure it accordingly.
What we’d tell someone starting today
- Write your reconnect logic before anything else.
- Use QoS 1 + idempotent consumers.
- Don’t put events on retained topics.
- Instrument the broker.
- Assume networks fail at the worst possible moment.
None of these are original. They’re all in books. But they’re also easy to skip when you’re shipping v1 and everything works in your dev environment.