← 5. Access rights · Contents · Next → 7. The fleet

6. Going offline¶

This is the heart of the project. Everything else exists so that this chapter can be short.

make outage   # cut the uplink, keep badging, replay: asserted end to end

What an outage actually changes¶

Almost nothing.

While the uplink is down
A badge is read	✅ works
The decision is taken	✅ works, from the local store
The door opens	✅ works
The passage is recorded	✅ works, in the local queue
A right changed in the cloud	⏸ queued in the cloud, delivered on reconnect
The dashboard shows the passage	⏸ shown when the passage arrives
The cabinet appears online	❌ marked offline after three missed heartbeats

The cabinet does not fall back to a degraded mode. It has been deciding locally all along; what stops is telling anyone about it.

Store & Forward¶

Every passage, granted, refused, replayed, or remote, goes into a local queue in the cabinet's SQLite database, and leaves it only when the broker has confirmed receipt.

flowchart TD
    D["A decision is taken"] --> Q["Append to pending_events<br/>status = pending"]
    Q --> T{"Drain tick<br/>every EVENT_PUBLISH_INTERVAL_MS"}
    T --> B["Take the oldest EVENT_BATCH_SIZE<br/>mark them 'sent'"]
    B --> P["PUBLISH each, QoS 1"]
    P --> A{"As many PUBACKs<br/>as we published?"}
    A -->|"yes"| C["Delete them from the queue"]
    A -->|"no"| R["Put the whole batch back<br/>as 'pending'"]
    R --> T

Every event goes through the queue, not just the ones taken during an outage. A live passage and a replayed one travel the same path, so the replay path is exercised constantly rather than only during an incident. An incident is exactly when you least want to discover it does not work.

Nothing is deleted before it is confirmed¶

An event leaves the queue on PUBACK and not a moment earlier. If the batch is only partly acknowledged, the whole batch goes back, including the events the publish loop never reached. Leaving them marked sent would strand them silently, which is exactly the loss this queue exists to prevent.

A process that stops between PUBLISH and PUBACK leaves rows marked sent that nobody received. On the next start they are all requeued.

The durability of the write¶

The store runs in WAL mode with synchronous = FULL.

Why FULL and not NORMAL? Under WAL, NORMAL lets the last commits go missing when the machine loses power rather than when the process crashes. On a door controller, losing power is the ordinary case, not the accident. A passage acknowledged to the driver and absent from the queue after a power cut would break the whole promise. The cost is one fsync per commit, and the queue write happens after the relay has already fired, so the driver waits for nothing.

The queue is bounded¶

PENDING_EVENTS_MAX (50 000 by default) is a real maximum.

An unbounded queue grows until the card fills, and a controller that cannot write is a controller that cannot record a passage at all. That is the exact failure this design exists to prevent. So the bound trades an unknown, unbounded loss for a known, counted one.

When the queue is full, the oldest events are dropped. Either end loses a passage; dropping the newest would mean refusing to record precisely while a car is at the door, and the oldest have already had the longest chance to be delivered.

Every drop is counted, kept across restarts, and reported in the telemetry as droppedEvents. A non-zero value raises an events_dropped alert.

A journal with a hole nobody knows about is worse than one that says where it tore.

The cabinet also reports queueSaturated once the queue crosses PENDING_EVENTS_WARN_RATIO (80 %) of its capacity, well before anything is lost. That raises a queue_saturated alert.

What an event looks like¶

{
  "eventId": "0d5b7c0e-3c4a-4c7d-9f22-2b3c1d5f7a91",
  "zoneId": "0b000000-0000-4000-8000-000000000001",
  "doorId": "0d000000-0000-4000-8000-000000000001",
  "deviceId": "edge-01",
  "readerId": "0e000000-0000-4000-8000-000000000001",
  "level": "info",
  "credentialType": "badgeNumber",
  "scannedValue": "BDG-0001",
  "userId": "0f000000-0000-4000-8000-000000000001",
  "credentialId": "1b000000-0000-4000-8000-000000000001",
  "accessGranted": true,
  "reason": "SUCCESS",
  "occurredAt": "2026-08-10T07:14:22.104Z",
  "isOfflineBatch": false
}

occurredAt is the reader's instant, not the moment the queue got round to it. A passage is timed when the credential was presented.

scannedValue is the only place a raw credential value exists outside the cloud's database. The local store holds its hash.

userId and credentialId are null when the credential is unknown: there is nobody to point at.

Deduplication¶

MQTT QoS 1 means at least once. A cabinet replays anything it did not see acknowledged, and its client may itself redeliver a buffered publish. Duplicates are expected traffic, not a fault.

They are made harmless on the receiving side:

CREATE UNIQUE INDEX idx_activity_logs_event
    ON activity_logs (event_id, occurred_at)
    WHERE event_id IS NOT NULL;

and the insert says ON CONFLICT DO NOTHING. A replayed event always carries its original occurredAt, so it lands in the same monthly partition and collides with itself.

Why deduplicate here rather than avoid duplicates there? Losing a passage is not recoverable; seeing one twice is. Chasing exactly-once delivery over an unreliable link buys a guarantee nobody can state simply; a unique index buys the same outcome in three lines.

The backend logs which it did, recorded or ignored duplicate, so a replay storm is visible rather than silent.

Telling a replayed passage from a live one¶

The dashboard marks passages that were held through an outage, because "this happened just now" and "this happened two hours ago and we are only hearing about it now" are different facts.

Two independent signals feed that flag:

The cabinet's claim. An event whose delivery has been attempted before is flagged isOfflineBatch: true as it goes out.
The backend's own measurement. If occurredAt is more than REPLAY_THRESHOLD_MS (30 s by default; 5 s in the local stack) behind the moment of receipt, it is recorded as held regardless.

The second exists because the first is not always available: the MQTT client buffers a publish internally and may deliver that first, unflagged copy once the link returns. Lateness is the observable fact, and the receiver is the one able to measure it.

The clock¶

A cabinet decides against validity windows and stamps every passage. A wrong clock therefore opens for a lapsed right, or turns a resident away, and does it silently, while dating records that will later be read as fact.

An offline controller cannot ask a time server. But there is one check that needs no network:

Real time never runs backwards.

The cabinet keeps a high-water mark of the latest instant it has already lived through, raised on every heartbeat. Wall time behind that mark by more than CLOCK_TOLERANCE_S (60 s) is proof the clock is wrong. The tolerance is wide enough to absorb a time daemon settling, and a dead battery lands years out, not seconds.

flowchart LR
    N["Wall clock now"] --> C{"now < mark − tolerance?"}
    C -->|"yes"| W["WentBackwards<br/>clockTrusted: false<br/>→ clock_untrusted alert"]
    C -->|"no"| T["Trusted"]
    T --> U["Raise the mark<br/>if now is ahead"]

The mark is deliberately not lowered on a bad verdict: the clock stays provably wrong until real time catches up with what was already lived, which is exactly how long the timestamps stay untrustworthy.

This cannot catch a clock that is merely slow or stopped. Only the cloud can, by comparing timestamps. So a Trusted verdict means not caught lying, not correct. The controller says which of the two it is in, at startup and in every heartbeat, rather than behaving as though a guarantee held.

The uplink coming back¶

sequenceDiagram
    participant E as Cabinet
    participant M as Broker
    participant B as Backend

    E->>M: CONNECT (clean session = false)
    M-->>E: CONNACK
    E->>M: SUBSCRIBE commands, rights/sync
    E->>M: PUBLISH rights/request
    Note over E: everything held is now drained,<br/>oldest first, flagged as replayed
    E->>M: PUBLISH events × N (QoS 1)
    M->>B: deliver
    B->>B: insert, ignoring duplicates
    B->>B: broadcast to open dashboards
    M-->>E: PUBACK × N
    E->>E: delete the confirmed events
    M->>E: the whole rights set

Replay is oldest first, so the backend receives passages in the order they happened.

Asking for the whole rights set on every connect is what lets a cabinet recover from any amount of missed traffic without tracking what it missed.

Why the cabinet also has a watchdog¶

Store & Forward covers the network. It does not cover the failure a supervisor cannot catch: a process that is still alive and no longer deciding. Nothing restarts it, and the door simply stops working.

The cabinet feeds /dev/watchdog every WATCHDOG_INTERVAL_S, and the feeder is deliberately not a bare timer: it kicks the board only while the local store still answers. If the store stops answering, the controller is no longer a controller, and the reset is the point.

A shutdown someone asked for disarms the watchdog first; a crash, a restart_service or a wedged process leaves it armed.

There is no /dev/watchdog in a container. The controller says so at startup rather than behaving as if it were protected.

Try it¶

make outage

The script freezes the broker rather than stopping it. Pausing leaves the TCP connections and every session intact, which is what a lost uplink looks like from the cabinet. Stopping the broker would wipe its sessions, a different failure altogether.

It then presents badges while the link is down, unfreezes, and checks that every passage arrives, with the right instants and flagged as replayed.

By hand:

docker compose pause emqx
make badge VALUE=BDG-0001      # the door still opens
docker compose logs edge-device | grep "event queued"
docker compose unpause emqx
docker compose logs edge-device | grep "events acknowledged"

The site survives on its own. Now let us watch it from a distance.

Next → 7. The fleet