Connectivity Dead Zones

One gateway, no tower in sight — the condition you design for, not against.

We’ve made the case that the cellular link is the unglamorous part of an IoT deployment — the thing that fails first and costs the most to fix. That post is about choosing the link well: multi-carrier SIMs, antenna placement, sane failover.

This one is about the other half. Because even a well-chosen link, at a real off-highway site, is going to be gone sometimes. The haul truck drops behind a bench. The gateway at the back forty is six miles from a tower with a hill in the way. A storm takes the tower down for three hours. On the kind of sites we get called about, the dead zone isn’t a corner case to handle gracefully. It’s a normal operating condition to design around.

Connectivity is a duty cycle, not a state

The mistake that shows up in year two is a system built as if the link is a switch — on, and occasionally off, where “off” is an error. At a remote site the link is a duty cycle: present most of the time, absent some of the time, and the absent stretches are not failures. They’re the network doing what that network does.

Once you accept that, the design questions change. Not “how do we make the link more reliable” — you already did that work — but “what is true about the system while the link is down, and what happens to all of it when the link comes back?”

Store-and-forward is the whole game

Every layer buffers. The asset tag’s beacon is seen by the gateway whether or not the gateway can reach the cloud. The gateway queues what it can’t send. Nothing upstream of a dropped link is allowed to lose data because the link dropped.

The only real design parameter is: how long must the buffer survive? Not the average outage — the worst realistic one. We size gateway storage for a multi-hour blackout as a floor, and for sites with a known bad season, longer. A buffer sized for the average outage is a buffer that overflows exactly when the outage is bad enough to matter.

Anything that drops data during a backhaul flap is a bug we’ve already shipped once and don’t intend to ship again.

When the buffer fills, you prioritize — you don’t panic

A buffer is finite, and a long enough outage will pressure it. The design decision is what to keep when you can’t keep everything.

The hierarchy we use:

• State changes and accrued hours are sacred. A machine started, a hub went offline, CMH ticked — those are the events the business runs on. They survive.
• High-frequency raw signal is expendable. Per-second RSSI is useful live and worthless backfilled three hours late. Under pressure it’s the first thing we downsample.
• Compress and summarize before you discard. An hour of “idle” collapses to one interval, not 3,600 samples. You keep the fact, you drop the redundancy.

This matters most where the fallback link is expensive. Iridium and Inmarsat work, but at a cost and payload size that makes “send everything, sort it later” a non-starter. Satellite is the last resort, and only the prioritized, compressed essentials go over it. A vendor whose offline story is “we’ll just push it all up when we reconnect” has never paid a satellite bill.

Time is attached at the source, not the arrival

The single most important rule for surviving outages cleanly: every event carries its own timestamp from the source, set when it happened — not when it arrived.

When the link comes back and the gateway dumps an hour of buffered events, those events arrive out of order, late, and in a burst. If the pipeline timestamps them on arrival, the history is a lie — everything looks like it happened at 2:47pm when the link recovered. Source-stamped events reorder correctly on ingest, and the timeline reflects what the machine actually did. Arrival time is metadata. Event time is the truth.

This also means the clocks have to be disciplined. A gateway with a drifting clock writes a confidently wrong history. We sync gateway time aggressively and treat a gateway whose clock has wandered as a fault, not a quirk.

Reconnection is a reconciliation, not a replay

When the link returns, the gateway doesn’t get to just stream its backlog into the system and hope. The ingest path has to be idempotent — keyed on device plus event identity, so that a retransmitted hour doesn’t double-count. The classic failure is a flapping link that delivers the same buffered batch twice and inflates CMH by an hour. The fix isn’t cleverness on the gateway; it’s an ingest layer that treats “I’ve seen this event already” as the normal case.

Get this right and a three-hour outage is invisible in the data after reconnection, except for the gap in the live view while it was happening. Get it wrong and every outage leaves a scar in the numbers that someone has to explain during a warranty conversation later.

Silence is ambiguous — don’t page on it blindly

Here’s the operational trap. A device that’s gone quiet could mean three different things: the asset is fine and nothing happened, the asset left coverage, or the device is dead. From the cloud they look identical — the same absence we described for a tag whose battery died.

If you alert naively on silence, a known dead zone wakes someone at 2am every time a truck parks behind the shop. They mute the alert within a week, and now you’ve trained the one person who matters to ignore the system.

So we separate stale from alarm. A device that’s quiet inside a known dead zone, within its expected return window, is stale — shown, not paged. A device that’s quiet when it should be reachable, past the window the site’s connectivity profile says is normal, is an alarm. The connectivity profile from the site survey is what lets the system tell those two apart. Without it, every outage is either a false page or a missed one.

The buffer needs power to survive the outage

One last dependency people miss: a gateway storing-and-forwarding through a three-hour outage is a gateway that has to stay powered for three hours of no backhaul — often during the same storm that took the tower down. A gateway that browns out mid-outage doesn’t just go quiet; depending on how it’s built, it can lose the buffer it was holding. Solar-plus-battery sized for the worst month, not the average one, is part of the connectivity design, not a separate line item.

The takeaway

The deployments that quietly work for years are the ones built on the assumption that the link will be gone, regularly, and that this is fine. Buffer everywhere. Stamp time at the source. Make ingest idempotent. Prioritize before you discard. Tell stale apart from alarm. Keep the gateway alive through the dark.

None of it is exotic. All of it is the discipline of building for the network you have, at the site you have, instead of the one in the coverage map.