Designing an IoT Ingestion Pipeline Where Every Stage Can Be Replayed

The goal

A multi-tenant IoT platform ingesting readings from cellular sensors on a raw TCP protocol and from a partner cloud pushing webhooks, feeding device health, an alerting engine, and notifications. Years earlier I had built the previous generation of this kind of system on a fully serverless stack, and it taught one lesson above all: keep the raw bytes. The goal this time was a pipeline where a decoder bug, a vendor quirk, or an alert-rule change is a data fix, never data loss.

The approach

Replayability as the organizing property. Ingress stores the raw delivery. Decoding is pure and separate from persistence. Every consumer is independently retryable, and every failure queue is listed with a replay command.

What I built

Ingress does one job. A long-lived TCP gateway terminates the wire protocol, puts the framed bytes on a queue verbatim, and answers the device with pending commands while the socket is open. The webhook receiver does the same with the request body. Neither decodes or persists.

Decode outside the transaction, commit once. A worker drains the queue in batches, decodes every record as a pure function first, then runs one transaction for attribution, an idempotent multi-row insert, and per-device snapshots. After commit it publishes an event per accepted reading; malformed frames are quarantined as their own event.

Consumers own their failures. Device health, alert evaluation, and notification fan-out each consume the bus with their own dead-letter queue. A registry declares every dead-letter queue in one place with operator scripts to list and replay.

Alerts from facts. A second evaluation path computes derived measures no single reading can answer, such as inventory shortfall across a zone or days since a device was last seen. Its query was profiled with the database's plan output and indexed, going from about 1.3 seconds to under 50 milliseconds at a few hundred thousand rows.

A fleet you can simulate. A seeded simulator built from the wire spec drives ten thousand virtual devices at the real gateway from a development stage, with mid-run injection of alarms, drops, and dying batteries, and a hard refusal to run against production.

The result

When a codec change mis-read one field, the fix shipped and the affected window was replayed from the raw queue records; corrected readings overwrote the wrong ones under the same idempotency keys, and nothing downstream needed to know. The alerting consumer has had its own bugs since; each time, its queue filled, the fix shipped, and the queue replayed while ingestion never noticed.

Takeaways

• Store the raw payload before you interpret it. Replay turns bugs into data fixes.
• Decode outside the transaction, commit once, publish after commit.
• Every consumer gets a dead-letter queue, and every queue is listed with a replay command.