The Echolo ingestion pipeline, roughly.

Every IoT company ends up running a data pipeline. Raw events from sensors come in, get normalized, stored somewhere durable, and pushed out to the consumers who care — dashboards, alert services, integrations with someone else’s data warehouse.

The industry talks about “streamlining data management” and “fully-managed pipelines” like this is a solved problem with a single correct off-the-shelf product. It isn’t. Every production pipeline we’ve seen is a patchwork of a broker, a queue, a schema enforcer, and at least one thing that was written in a hurry three years ago and still hasn’t been replaced.

Here’s what Echolo’s pipeline actually looks like. Notable absence: none of it is particularly glamorous.

The layout

[gateways] → MQTT broker → dispatch service → RabbitMQ fanout
                                                    ↓
                                 ┌──────────────────┼──────────────┐
                                 ↓                  ↓              ↓
                          MongoDB (time-series) PostgreSQL    WebSocket/SSE
                                                  (tenant data)  (live UI)

That’s the whole thing. Six boxes, five arrows.

  • MQTT broker: the single entry point for every beacon from every gateway. QoS 1, TLS, ACLs per tenant. We run EMQX but Mosquitto would be fine at our scale.
  • Dispatch service: small Go binary that subscribes to the broker, decodes the payload, validates against the schema, and fans out to RabbitMQ with typed routing keys.
  • RabbitMQ fanout: topic exchange with three queues: time-series storage, tenant metadata updates, live UI streaming. Each consumer only gets the messages it cares about.
  • MongoDB for telemetry. Sharded on tenant + asset. We ended up here instead of TimescaleDB because early schema churn was cheaper to absorb in a document store. This is a trade-off we revisit every year.
  • PostgreSQL for tenant data — users, alerts, rules, audit logs. Boring and correct.
  • WebSocket + Server-Sent Events for the Fleet UI. Phoenix Channels + PubSub handle this with very little ceremony.

Things that took us two rewrites to get right

Idempotency at ingestion. Gateways retry on any network hiccup. If you don’t dedupe at the broker boundary, the same beacon lands in your database three times and your CMH numbers drift. We stamp each message with a gateway-assigned event ID at publish time and dedupe on first-seen.

Dead-letter everything. If a message fails schema validation, it goes to a DLQ with the failure reason. Every morning we sample the DLQ. 90% of what ends up there is a customer running old firmware we haven’t blocklisted yet. The other 10% is a bug we need to fix.

Never lose the raw. Before any decoding or enrichment happens, the raw MQTT payload gets stamped with a timestamp and written to append-only storage. This has saved us exactly twice, and both saves were worth more than the whole storage cost.

Backpressure signals, not buffers. If the downstream slows, the system needs to tell the upstream. Buffering “temporarily” is how you discover at 4am that your Mongo connection pool is exhausted and the broker has six hours of queued messages it’s trying to deliver.

What we don’t do

  • Schema-less ingestion. Every topic has a schema. If a gateway publishes a payload that doesn’t match, it goes to the DLQ and nobody pretends it’s data.
  • “Managed” black-box pipelines. We evaluated several third-party “data pipe” SaaS offerings and walked away. When ingestion is broken at 3am, we need to own every layer of the stack. Managed pipelines trade debugging latency for setup speed — a trade-off that only looks good before you’ve had an outage.
  • One-size-fits-all consumer topics. Dashboards, alert engines, and reports have different latency requirements. The alert engine can’t wait on the reporting aggregation pipeline. Route each consumer to its own queue.

The integration surface

From a customer perspective, the pipeline exposes:

  • MQTT, if you want the firehose — subscribe to your tenant’s topics directly.
  • WebSocket, for in-browser live updates.
  • Webhooks, for push into your stack with signed payloads and retry-with-backoff.
  • REST, for historical queries.
  • Bridges to AWS IoT Core and Azure IoT Hub if you’re already running a cloud IoT stack and want ours to slot in.

Five integration points. Each one documented. That’s the whole surface.

The takeaway

Data pipelines are a solved problem in the sense that the primitives (brokers, queues, DBs, streaming) are mature. They are not solved in the sense that you can buy a complete one and skip having to think about idempotency, backpressure, schema validation, or what happens in an outage. Anyone who tells you otherwise is selling something.