← 6. Going offline · Contents · Next → 8. The dashboard

7. The fleet¶

Cabinets are on customers' sites, sometimes in another country, behind their router. Everything you can do to them, you do from a screen.

make fleet      # telemetry, remote log level, live debug: asserted
make commands   # open a lane, lock it, restart the service: asserted
make ota        # install a release, refuse a corrupt one, roll a broken one back

Every topic in this chapter is formalised in the MQTT reference.

The heartbeat is the report¶

Every TELEMETRY_INTERVAL_S (10 s), each cabinet publishes what it knows about itself. There is no separate ping.

Why one signal? Because "it is alive" and "here is how it is doing" can then never disagree. A cabinet that answers a ping while its queue is full and its readers are dead is a cabinet that looks healthy.

What a cabinet reports¶

{
  "firmwareVersion": "v0.2.0",
  "osVersion": "Debian GNU/Linux 12 (bookworm)",
  "hardwareModel": "UniPi Patron",
  "ipAddress": "10.42.0.17",
  "freeMemoryKb": 381244,
  "diskUsagePercent": 43.18,
  "certificateExpiresAt": "2028-11-02T09:31:00Z",
  "pendingEvents": 0,
  "queueSaturated": false,
  "droppedEvents": 0,
  "clockTrusted": true,
  "readersDown": [],
  "readerBusStopped": false,
  "accessRightsHash": "618c4da9…",
  "accessRightsCount": 128,
  "appliedRevision": 42,
  "logLevel": "info",
  "debugUntil": null,
  "timestamp": "2026-08-10T07:14:20.001Z"
}

Field	Why it is there
`firmwareVersion`	What it is actually running: the release it installed, not the one it shipped with
`osVersion`, `hardwareModel`	The distribution name and what the device tree says the box is
`ipAddress`	Discovered by asking the routing table which interface carries outbound traffic. No packet is sent
`freeMemoryKb`, `diskUsagePercent`	The partition holding the queue, which is the one whose fullness matters
`certificateExpiresAt`	The instant, not the days left. Counting days needs a clock, and that is the one thing the cabinet cannot vouch for
`pendingEvents`	How many passages are still waiting for the cloud
`queueSaturated`	The cabinet's own verdict: it knows its capacity
`droppedEvents`	Cumulative since commissioning. Non-zero means the record has a hole
`clockTrusted`	Whether wall time is consistent with what it has already lived through
`readersDown`	Which bus addresses stopped answering. The cabinet is the only party that knows what is wired to it
`readerBusStopped`	The poll loop is over. Distinct from the above: a stopped bus reports no reader down and would otherwise look like a site nobody is badging at
`accessRightsHash`, `accessRightsCount`	The fingerprint, so drift is visible without shipping the set back
`appliedRevision`	The last rights revision it confirmed applying
`logLevel`, `debugUntil`	What it is currently doing about diagnostics

The cloud stores the latest report per cabinet in edge_device_telemetry, marks the controller online, and broadcasts it so open dashboards update live.

Telemetry from a cabinet the cloud does not know is dropped: an unprovisioned box must not create rows by talking to us.

Going offline¶

A sweep runs every five seconds in every backend instance, coordinated by an advisory lock. Any controller in online or updating whose last heartbeat is older than DEVICE_HEARTBEAT_INTERVAL_S × DEVICE_OFFLINE_AFTER_MISSED (10 × 3 = 30 s) is flipped to offline, with a line in the technical journal and a controller_silent alert.

Why three missed heartbeats and not one? A single lost heartbeat on a 4G link says nothing, and a dashboard that flickers between online and offline teaches operators to ignore it.

Silence is the only evidence available: a cabinet that lost power, lost its uplink or crashed all look identical from here, and all three mean an operator should be told rather than left reading a stale "online".

The statuses¶

Status	Meaning
`provisioning`	Declared, not yet reporting
`online`	Reporting on time
`offline`	Silent past the threshold
`updating`	An OTA was pushed; it is expected to go quiet for a while
`maintenance`	Deliberately out of service

updating exists so that the silence of a firmware install does not read like a fault. A cabinet that never comes back from it is swept to offline like any other.

Alerts¶

The system raises eight named conditions. They are written to the technical journal (system_audit_logs with event_type = 'alert') and, when a DSN is configured, mirrored to Sentry, because the people on call do not read the database.

Kind	Raised when
`controller_silent`	A cabinet stopped reporting
`reader_error`	Readers stopped answering, or the bus stopped polling altogether
`queue_saturated`	The Store & Forward queue crossed its warning ratio
`events_dropped`	The queue was full and passages were lost
`clock_untrusted`	Wall time is behind an instant the cabinet already lived through
`sync_failed`	A rights delta was refused, or never delivered after ten attempts
`rights_drift`	The fingerprint still disagrees after repeated full syncs
`certificate_expiring`	A cabinet's certificate lapses within the warning window, or has

The same condition is raised once and then stays quiet for ALERT_REPEAT_MINUTES (60). A cabinet reports every ten seconds, so a saturated queue would otherwise raise three hundred and sixty alerts an hour and be indistinguishable from noise. Long enough to stop the flood, short enough that a condition nobody fixed says so again.

Deduplication is a query rather than in-memory state: any instance may notice the condition, and an in-memory guard would let each of them raise it once. Two instances noticing in the same millisecond can still double. An alert delivered twice is a nuisance; one held back because another instance thought it had sent it is a fault.

Reading them:

curl -H "Authorization: Bearer $TOKEN" \
     'http://localhost:3000/api/v1/alerts?limit=50'

Read-only on purpose. An alert is not acknowledged here; it stops being raised when the condition that raised it goes away. Anything else would mean two truths about the same cabinet.

An alert whose cabinet cannot be resolved has no tenant, so it reaches signed-in operators and is withheld from tenant-confined API keys. Withholding an alert is a nuisance; showing one customer's cabinet to another is a breach.

Commands¶

A command is published to the cabinet and acknowledged by it. Nothing waits for that acknowledgement: a cabinet on a 4G link may be a minute away, and an HTTP request must not be.

sequenceDiagram
    participant O as Operator
    participant B as Backend
    participant M as Broker
    participant E as Cabinet

    O->>B: POST /devices/{id}/actions/...
    B->>B: control plane audit: who asked
    B->>M: commands {commandId, action, parameters}
    B-->>O: 200, with the commandId
    M->>E: deliver (QoS 1, held while offline)
    E->>E: apply
    E->>M: ack {refId, status, error?}
    M->>B: technical journal: what became of it

The commands¶

Action	What it does
`set_log_level`	`debug` · `info` · `warn` · `error`, applied without restarting
`enable_debug` / `disable_debug`	Opens or closes the live diagnostic stream, always time-boxed
`full_resync`	Makes the cabinet ask for its whole rights set
`force_open`	Pulses a door's relay now
`set_door_mode`	`normal` · `unlocked` · `locked`
`restart_service`	Stops the process; the supervisor brings it back
`reboot_os`	Reboots the machine
`ota_update`	Fetches, verifies and installs a firmware release

An action this firmware does not implement is rejected, not retried: replaying it would fail the same way.

Through the API¶

DEVICE=0c000000-0000-4000-8000-000000000001
DOOR=0d000000-0000-4000-8000-000000000001

curl -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"level":"debug"}' http://localhost:3000/api/v1/devices/$DEVICE/actions/log-level

curl -X POST -H "Authorization: Bearer $TOKEN" \
  http://localhost:3000/api/v1/devices/$DEVICE/actions/full-resync

curl -X POST -H "Authorization: Bearer $TOKEN" \
  http://localhost:3000/api/v1/doors/$DOOR/actions/open

curl -X PATCH -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"mode":"locked"}' http://localhost:3000/api/v1/doors/$DOOR

A door is addressed by the door, and routed to whichever cabinet drives it: an operator on the phone with a driver knows which lane, not which cabinet.

Restarting versus rebooting¶

They are separate buttons because they are separate costs: seconds against a site with no controller for a minute. Collapsing them into one would make the cheap remedy as frightening as the expensive one.

restart_service stops the process and the supervisor brings it back, exactly as after a crash. The local rights survive, because they live in the cabinet's database and not in its memory.
reboot_os is refused where there is no OS to reboot. That is the case in a container. Degrading it into a service restart would let an operator believe a box came back clean when nothing rebooted.

In both cases the acknowledgement is published and given half a second to reach the broker before the process does what it was told: an operator must not see a cabinet go quiet with no idea whether their command arrived.

The door mode lives on both sides¶

The cloud holds it so the dashboard can show it. The cabinet holds it so an uplink outage cannot change how a door behaves. A locked lane stays locked with the network gone.

Changing it through the API updates the cloud row and sends the command. The cabinet is where it is applied.

The live debug console¶

open http://localhost:4200/fleet/<controller-id>/debug

The console streams raw reader frames, relay pulses, sync events and broker traffic as they happen, and writes none of it: not to a table, not to a file, not to a log.

flowchart LR
    E["Cabinet emits a signal"] --> D{"Is debug on<br/>right now?"}
    D -->|"no"| N["Nothing happens"]
    D -->|"yes"| Q["Bounded in-memory channel"]
    Q --> P["debug topic, QoS 0, never retained"]
    P --> B["Backend broadcasts it"]
    B --> W["/ws/debug?device=edge-01"]

Three properties, all deliberate:

It is time-boxed. The cloud caps whatever an operator asks for at DEVICE_DEBUG_MAX_DURATION_S (900 s), and the cabinet caps it again on arrival. It is the one paying for the 4G. The cabinet forgets about it on its own; nobody has to remember to turn it off.

It never blocks a decision. The channel is small (512 signals) and a full channel drops the signal. Waiting for a console to keep up would put a door's decision behind somebody's browser.

It is closed to API keys. Raw frames carry no tenant and cannot be confined to one; rather than confine what cannot be confined, the socket refuses a tenant-scoped caller outright with 403.

curl -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"enabled":true,"durationSeconds":120}' \
  http://localhost:3000/api/v1/devices/$DEVICE/actions/debug

Why store nothing? A debug stream that accumulated would become a second copy of everything a door sees, outside the retention rules and outside the audit trail.

Updating over the air¶

make ota

Publishing a release¶

A release is two objects in the wardn-ota bucket:

releases/v0.2.0/wardn-edge      the artifact
releases/v0.2.0/manifest.json    {version, artifact, sha256, sizeBytes, publishedAt}

The bucket is the catalogue: a release exists because its manifest is there, not because a row says so. One less place for two things to disagree, and publishing stays a matter of uploading two objects.

GET /api/v1/firmware      # what can be installed, newest first

Readable by any operator: seeing that a cabinet is two versions behind is part of watching a fleet.

Pushing it¶

curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" -H 'Content-Type: application/json' \
  -d '{"version":"v0.2.0"}' \
  http://localhost:3000/api/v1/devices/$DEVICE/actions/ota

This is the only action reserved to wardn-admin. Everything else in wardn is open to any operator and audited instead.

The backend signs a link valid for OTA_LINK_VALIDITY_S (900 s) and sends it with the digest. The artifact never travels through the API: the cabinet downloads it from the store on its own schedule, and the link dies on its own rather than being a credential somebody has to remember to revoke.

The trail records the version and the digest, not the link. The link is a credential; the version is the fact.

What the cabinet does¶

flowchart TD
    C["ota_update arrives"] --> Q["Queued on its own task<br/>(a 4G download outlasts<br/>the connection it arrived on)"]
    Q --> D["Stream to <binary>.staged,<br/>digesting chunk by chunk"]
    D --> V{"Digest matches<br/>what was announced?"}
    V -->|"no"| R["Delete the staged file,<br/>ack 'rejected'.<br/>The running firmware is untouched"]
    V -->|"yes"| P["chmod 755"]
    P --> B["Copy the current binary<br/>to <binary>.previous"]
    B --> M["Write <binary>.pending<br/>naming the release on trial"]
    M --> S["rename staged → current"]
    S --> A["ack 'applied', then exit"]
    A --> N["The supervisor starts the new firmware"]
    N --> OK{"Does it come up<br/>and serve doors?"}
    OK -->|"yes"| CF["It clears the marker.<br/>The release is confirmed"]
    OK -->|"no"| RB["The supervisor puts<br/>.previous back on the next start"]

Every step of that order is load-bearing:

The digest travels with the command, not with the artifact. An artifact that carries its own checksum proves nothing: whatever corrupted or replaced it would carry a matching one. A download truncated by a dropped link and one swapped by an attacker fail the same check.
Nothing is put in place before the digest matches, so a failed update leaves the cabinet deciding on the firmware it had.
The copy is taken before the swap. Once the rename lands, the bytes of the outgoing firmware are only reachable through it.
The marker is written before the swap, so a crash between the two leaves a rollback possible rather than a new firmware nobody is watching.
The switch is a rename inside one directory, so there is no moment at which the file the supervisor starts is half written.
Confirmation is the boot itself. The controller clears the marker only after it has opened its store, found its rights and declared its doors ready. A firmware that panics on a missing table never reaches that line.

The recorded firmwareVersion is set at confirmation, not at install: that is the version the cabinet is running, and it only becomes true once it has booted.

Only one update may be in flight; a second is rejected with "another update is already running".

What else a cabinet exposes¶

Its own metrics endpoint¶

Each cabinet serves Prometheus text on METRICS_LISTEN (:9090, published on 9101 locally):

relay_actuations_total          door openings driven since this process started
pending_events                  passages held by Store & Forward
osdp_reader_connected{address}  whether a reader is answering its polls

Read on the local network, by whatever monitors the site, not over the 4G link, which the telemetry already pays for. The two overlap on purpose: pending_events is the same number, taken from the same place, so an operator comparing a dashboard with a graph is never told two different things.

A cabinet that cannot count its own queue answers 500 rather than falling silent: silence would read as the box being down, which is a different and much louder problem.

Links and documents¶

GET /api/v1/devices/{id}/links           # 4G router admin page, monitoring board…
GET /api/v1/devices/{id}/documentation   # wiring diagram, hardware guide, network config

The address of the site's router at two in the morning is what somebody diagnosing a cabinet actually needs, and remembering each of them is not their job. Documents live in object storage; the rows point at them.

The fleet is watchable. Let us look at the screen it is watched from.

Next → 8. The dashboard