← 11. Personal data · Contents · Next → 13. Developing

12. Operating

Everything needed to keep wardn running: deploying it, watching it, backing it up, restoring it, and the complete list of knobs.

Deploying

The local stack is Docker Compose. Production replaces four things with managed equivalents and changes nothing in the code:

Component	Local	Production
Broker	EMQX container	AWS IoT Core, or any MQTT broker with shared subscriptions
Object storage	MinIO	S3
Database	PostgreSQL container	Managed PostgreSQL
Identity	Keycloak container	Keycloak, or any OIDC provider

A worked example of the right-hand column, on AWS with a Helm chart and Terraform, is 14. Deploying to production.

The database socle

db-init is the single source of truth for the schema, the roles, the grants, the partitions and the seed. It is idempotent and runs on every up.

make up          # migrations, partitions, seed
make verify      # assert the schema, the grants and the seed
make seed        # re-run migrations and seed
make reseed      # re-run the seed even if already applied

Set SEED_ENABLED=false for an empty database. PARTITION_MONTHS_BACK and PARTITION_MONTHS_AHEAD control how many monthly partitions are created up front; the retention sweep keeps two months ahead from then on.

Running more than one backend

The backend is stateless and leaderless.

make scale   # a second instance on port 3001, and the proof that it works

Two rules:

• Each instance needs its own INSTANCE_ID, stable across restarts. Two MQTT clients sharing an id kick each other off the broker. In Kubernetes this is the StatefulSet ordinal.
• Stable matters as much as unique. With INSTANCE_ID set, the broker keeps a persistent session and queues events for an instance that is restarting. Without it, the backend takes a clean session, says so in its log, and accepts that a restart misses whatever arrives while it is down.

Everything else coordinates by itself: MQTT shared subscriptions spread events, the rights queue is drained concurrently with leases, and the scheduled sweeps take a PostgreSQL advisory lock.

Certificates

tls-init issues a development CA and one identity per party on first start. In production the authority is the manufacturing one, and a cabinet's key never leaves its secure element. See 10. Security.

make tls-check   # how long every certificate in the stack has left

Exit code	Meaning
0	Nothing lapses within TLS_EXPIRY_WARNING_DAYS
1	Something lapses inside the window, or already has
2	The check could not be made (no openssl)

The 2 exists precisely so that an inability to check does not masquerade as a certificate alarm.

Watching day to day

Signal	Where to read it
A cabinet is offline	Fleet screen, or the controller_silent alert
A queue is rising	pendingEvents in telemetry, queue_saturated alert
Passages were lost	droppedEvents non-zero, events_dropped alert
A clock is wrong	clockTrusted: false, clock_untrusted alert
A reader stopped answering	readersDown, readerBusStopped, reader_error alert
Rights are not landing	sync_failed, rights_drift alerts; appliedRevision against rightsRevision
A certificate is dying	certificate_expiring alert, and make tls-check
Application errors	Sentry, when SENTRY_DSN is set
A specific request	The X-Trace-Id on its response
A cabinet, from the site's own monitoring	http://<cabinet>:9090/metrics
The backend is up	GET /health, and which instance answered
The database answers	GET /health/ready, or 503 when it does not

Alerts are read through the API:

curl -H "Authorization: Bearer $TOKEN" 'http://localhost:3000/api/v1/alerts?limit=50'

The eight kinds and what raises them are in 7. The fleet.

Backups

make backup                                    # database, artefacts, erasure ledger
make backup-verify BACKUP=./backups/<stamp>    # restore it elsewhere to prove it
make restore BACKUP=./backups/<stamp> CONFIRM=yes

flowchart TD
    B["make backup"] --> D["pg_dump, custom format"]
    B --> L["Erasure ledger<br/>written BESIDE the dump"]
    B --> O["Mirror the object storage<br/>(firmware, documents)"]
    D --> M["MANIFEST: what this can<br/>and cannot restore"]
    L --> M
    O --> M
    M --> ENC{"Off-site<br/>configured?"}
    ENC -->|"yes, with a passphrase"| E["Encrypt, upload,<br/>prune the remote"]
    ENC -->|"yes, no passphrase"| R["REFUSE: nothing<br/>leaves in the clear"]
    ENC -->|"no"| K["Stays on this machine"]

What is backed up

	Why
PostgreSQL	Rights, passages, the audit trail. The only copy
Object storage	Firmware artefacts and documents. Their digests are in the database, so a database restored without them points at files that do not exist
The erasure ledger	The list of people erased, written beside the dump. It is what makes this more than a pg_dump

What is not backed up

• The cabinets' local databases: derived from the cloud, no history. They rebuild themselves on the next connect.
• EMQX state: sessions and retained messages rebuild themselves.
• The development certificate authority: production uses the manufacturing PKI.
• ⚠️ Keycloak accounts created at runtime. The realm is imported from this repository, so what a backup would add is whatever was created afterwards. That is a real gap, stated rather than hidden.

The recovery point

One pg_dump a day. Up to 24 hours of passages can disappear.

That is not theoretical. The cabinet is built never to lose a passage locally. But once an event is acknowledged and cleared from its queue, last night's dump holds the only copy.

It is the accepted cost of not operating a write-ahead log archive. If that cost becomes unacceptable, point-in-time recovery is the way out, and it is an operations project rather than a script change.

A backup that has never been restored is not a backup

make backup-verify BACKUP=./backups/<stamp>

This is the step everyone skips, and the reason backups are found to be worthless on the one day they are needed.

It restores into a throwaway database beside the real one, never over it, then asserts three things a corrupt or truncated dump cannot satisfy: the schema comes back whole, checked by the same verify.sql the socle uses; the tables carrying the business are not empty; and the audit trail came back with them, since a restore that loses the trail restores the data without the accountability.

Off site

A backup on the same machine as the database is not a backup: the fire, the theft and the failed disk take both.

BACKUP_REMOTE_ENDPOINT=https://s3.eu-west-3.amazonaws.com
BACKUP_REMOTE_BUCKET=wardn-backups
BACKUP_REMOTE_ACCESS_KEY=…
BACKUP_REMOTE_SECRET_KEY=…
BACKUP_ENCRYPTION_PASSPHRASE=…

Encrypted before it leaves, never after. With a destination configured and no passphrase, make backup refuses, rather than sending names, credentials and the history of every passage through a site to somebody else's disk in the clear because a variable was forgotten.

⚠️ A lost passphrase makes the backup as useless as its absence. It belongs neither in this repository nor in the remote store.

The SHA-256 of the content before encryption stays on site, where the remote store never saw it. Comparing a copy that comes back against that value proves what returns is what left. The cipher alone does not prove that.

The same run prunes the remote store beyond BACKUP_KEEP_DAYS. Retention off site is not the local find: without this, a 30-day commitment would hold everywhere except the one place the data had been copied to.

Backups compose with retention rather than coinciding with it. The newest backup already holds logs 30 days old, so by the time it is deleted at 30 days it has been holding data up to 60 days old. That belongs in a register of processing activities.

Restoring

make restore BACKUP=./backups/<stamp> CONFIRM=yes

Destructive, and deliberately hard to run by accident: without CONFIRM=yes it refuses and prints the command it wanted.

It stops the backend, restores the dump, re-runs db-init (which is the source of truth for the schema, the roles and the append-only grants, and is idempotent), then starts the backend again.

The two steps that are not automatic

The script prints them instead of doing them, because neither is safe to automate.

flowchart TD
    R["Restore finished"] --> S1["1. Replay the erasures<br/>recorded after this backup"]
    R --> S2["2. Run the retention sweep"]
    S1 --> A["Collect the ids from every<br/>LATER backup's ledger"]
    A --> B["Re-issue each erasure"]
    B --> C{"Response?"}
    C -->|"404"| D["Already erased,<br/>nothing to do"]
    C -->|"2xx"| E["That person HAD come back.<br/>Count them: this number<br/>cannot be recovered later"]
    S2 --> F["Data purged after the backup<br/>is back, and out of policy<br/>until the next sweep"]

1. Replay the erasures. Restoring rolls back later erasures and the record that they happened. The ledgers written beside the following backups are the only things that know who must stay erased. The script prints the list of ledger files with the real paths filled in:

cut -d, -f2 backups/<later>/erasures.csv … | sort -u

Then, for each id, re-issue the erasure and read the answer:

• 404: already erased, nothing to do;
• 2xx: that person had come back, and has just been erased again.

That count is what goes into the incident record and to the data protection officer. After this step it is no longer recoverable from anywhere.

If no later ledger is reachable, stop: nothing in the world knows who had to be erased.

2. Run the retention sweep. A backup taken before a purge restores the data the purge removed, putting the system back outside its own announced window until the next sweep.

Recovering a backup that went off site

The scenario off-site backup exists for is the one where nothing local is available, so the command cannot live only in the script that disappeared with the machine. It is copied into the manifest inside the archive:

mc alias set offsite "$BACKUP_REMOTE_ENDPOINT" "$BACKUP_REMOTE_ACCESS_KEY" "$BACKUP_REMOTE_SECRET_KEY"
mc cp "offsite/$BACKUP_REMOTE_BUCKET/<stamp>.tar.enc" .

openssl enc -d -aes-256-cbc -pbkdf2 -iter 600000 \
  -in <stamp>.tar.enc -out <stamp>.tar \
  -pass env:BACKUP_ENCRYPTION_PASSPHRASE

mkdir -p backups/<stamp> && tar -C backups/<stamp> -xf <stamp>.tar
make backup-verify BACKUP=backups/<stamp>

A wrong passphrase fails loudly (bad decrypt); it never returns a half-correct file.

Commissioning a cabinet

1. Create the zone, then the edge_controllers, doors and identification_devices rows. Today this is SQL or the seed. There is no write API for hardware.
2. Issue the cabinet an identity whose common name is its device_identifier. The broker turns that name into the topic subtree it may reach.
3. Set DEVICE_ID to the same value, point MQTT_HOST / MQTT_PORT at the broker, and give it the three TLS paths.
4. Give it a BOOTSTRAP_RIGHTS_PATH describing its wiring (which zone it serves, its doors, its readers) plus a fallback set of rights. The file is read only when the local store is empty; the cloud replaces the rights on the first full sync.
5. Start it. On its first connect it asks for its whole rights set.

Check it landed:

curl -H "Authorization: Bearer $TOKEN" http://localhost:3000/api/v1/devices/fleet

status: online, an appliedRevision matching rightsRevision, and an accessRightsHash the cloud agrees with.

Common situations

Symptom	What to look at
A cabinet is grey on the fleet screen	It has missed three heartbeats. Power, uplink or a crash all look the same from here. controller_silent was raised
appliedRevision lags rightsRevision	Deltas queued or in flight. sync_failed says if delivery gave up; POST /devices/{id}/actions/full-resync forces the remedy
rights_drift keeps being raised	The cabinet has been sent its whole set three times and still disagrees. That is far more likely a fault in the fingerprint comparison than a cabinet losing the same rights every time. The nightly pass still covers it meanwhile
pendingEvents climbing	The uplink is down, or the broker is refusing. Passages are safe until the queue reaches PENDING_EVENTS_MAX
droppedEvents non-zero	The record for that cabinet has a hole. The number is cumulative since commissioning
clockTrusted: false	The cabinet's wall clock is behind an instant it already lived through. Its decisions and its timestamps cannot be trusted until real time catches up
A door behaves oddly	Check its mode. A lane left in unlocked after roadworks opens for everybody, and still logs every read
reboot_os is refused	The controller runs in a container. It refuses rather than degrading into a service restart
The dashboard cannot reach the API	CORS_ORIGINS and WARDN_API_URL must both name the address the browser dials
Retention never runs	MAINTENANCE_DATABASE_URL is unset. The backend says so once at startup and keeps serving

Configuration reference

Everything is an environment variable with a documented default. make env creates your .env from .env.example.

Backend

Variable	Default	Effect
PORT	3000	HTTP port
DATABASE_URL	required	The application role's connection
DATABASE_POOL_MAX	10	Connections per instance
DATABASE_CONNECTION_TIMEOUT_MS	5000	How long a request waits for a free connection
DATABASE_IDLE_TIMEOUT_MS	10000	How long an unused connection is held
DATABASE_STATEMENT_TIMEOUT_MS	15000	Enforced by PostgreSQL, so it releases the connection even when the client stopped waiting
MAINTENANCE_DATABASE_URL	(empty)	Retention and erasure only. Empty means those two are unavailable
MQTT_URL	required	mqtt:// or mqtts://
MQTT_SHARED_GROUP	wardn-backend	Every instance joins the same group
MQTT_USERNAME	wardn-backend	Must match the certificate's common name on the mTLS listener
MQTT_TLS_CA · _CERT · _KEY	(empty)	Read only when MQTT_URL is mqtts://. Missing one is fatal
INSTANCE_ID	wardn-backend-<hostname>	Stable and unique per instance. Setting it deliberately enables a persistent session
KEYCLOAK_ISSUER_URL	required	Exactly as it appears in tokens
KEYCLOAK_JWKS_URL	derived from the issuer	Where this process can reach the realm, which may differ
CORS_ORIGINS	http://localhost:4200	Listed, never reflected
LOG_LEVEL	info
API_DEFAULT_PAGE_SIZE	50
API_MAX_PAGE_SIZE	500	Ceiling on limit
DEFAULT_ZONE_TIMEZONE	Europe/Brussels	The timezone a zone created without one gets
RATE_LIMIT_WINDOW_MS	60000
RATE_LIMIT_MAX	600	Requests per window, applied before authentication
WS_HANDSHAKE_LIMIT	60	Live-socket handshakes per address per window
REPLAY_THRESHOLD_MS	30000	Lateness past which a passage is recorded as held
DEVICE_HEARTBEAT_INTERVAL_S	10	The cadence the cloud measures silence against
DEVICE_OFFLINE_AFTER_MISSED	3	Missed heartbeats before a cabinet is called offline
DEVICE_OFFLINE_SWEEP_MS	5000	How often the sweep looks
DEVICE_DEBUG_MAX_DURATION_S	900	Ceiling on a debug stream, enforced here and again on the cabinet
ALERT_REPEAT_MINUTES	60	Silence after a condition has been raised once
TLS_EXPIRY_WARNING_DAYS	30	Notice before a cabinet's certificate lapses
SYNC_TICK_MS	1000	How often the rights queue is drained
SYNC_TASK_BATCH	20	Deltas claimed per tick
SYNC_TASK_LEASE_S	30	How long a claim is held before another instance may take it
SYNC_TASK_MAX_ATTEMPTS	10	Attempts before giving up and alerting
RIGHTS_FULL_SYNC_LOCAL_HOUR	3	Nightly full sync, in each zone's own timezone
RIGHTS_FULL_SYNC_JITTER_MINUTES	60	Window the fleet is spread over
RIGHTS_FULL_SYNC_TICK_MS	60000	How often the nightly schedule is examined
RIGHTS_VERIFY_INTERVAL_S	7200	How often fingerprints are compared. 0 disables the check
RIGHTS_MAX_FINGERPRINT_MISMATCHES	3	Disagreements before alerting instead of resynchronising
RETENTION_SWEEP_MS	21600000	Six hours
ACTIVITY_LOG_RETENTION_DAYS	30
SYSTEM_LOG_RETENTION_DAYS	30
AUDIT_LOG_RETENTION_DAYS	365
ACCESS_RIGHT_RETENTION_DAYS	30	How long a revoked right is kept
SYNC_TASK_RETENTION_DAYS	7	How long an acknowledged sync task is kept
S3_ENDPOINT	http://minio:9000	Signed into the links handed to cabinets, so it must be an address a cabinet can dial
S3_REGION	us-east-1
S3_ACCESS_KEY · S3_SECRET_KEY	(empty)	Scoped credentials; never the root account
S3_OTA_BUCKET	wardn-ota
OTA_LINK_VALIDITY_S	900	How long a presigned firmware link lives
SENTRY_DSN	(empty)	Empty means no error reporting
SENTRY_ENVIRONMENT	development
OTEL_EXPORTER_OTLP_ENDPOINT	(empty)	Empty means spans are created but not exported
OTEL_TRACES_CONSOLE	false	Print spans where the logs are

Cabinet

Variable	Default	Effect
DEVICE_ID	required	The name it publishes under, and the common name of its certificate
SQLITE_PATH	/data/edge.db
SQLITE_ENCRYPTION_KEY_SOURCE	secure_element	env · none · secure_element (⚠️ not implemented: the process refuses to start)
SQLITE_ENCRYPTION_KEY	(none)	Required when the source is env. No default, ever
BOOTSTRAP_RIGHTS_PATH	(none)	Wiring and fallback rights, read only when the store is empty
READER_BUS	simulated	simulated (text over TCP) or osdp (RS-485)
READER_LISTEN	0.0.0.0:9000	Where the simulated bus listens
OSDP_SERIAL_PORT	/dev/ttyV0
OSDP_BAUD_RATE	9600
OSDP_ADDRESSES	1	Which peripheral addresses to poll
OSDP_POLL_INTERVAL_MS	100
METRICS_LISTEN	0.0.0.0:9090	Prometheus endpoint, on the local network
LOG_LEVEL	info	Changeable at runtime from the cloud
DECISION_TIMEOUT_MS	300	The budget a decision is measured against
RELAY_PULSE_MS	2000	Default pulse; a door may carry its own
PENDING_EVENTS_MAX	50000	Store & Forward capacity. A real maximum
PENDING_EVENTS_WARN_RATIO	0.8	Depth at which the cabinet reports itself saturated
CLOCK_TOLERANCE_S	60	How far wall time may run backwards before the clock is called wrong
EXPIRED_RIGHT_GRACE_DAYS	7	How long a lapsed right stays distinguishable from an unknown one
MQTT_HOST · MQTT_PORT	emqx · 1883
MQTT_TLS_CA · _CERT · _KEY	(empty)	All three or none; a half-configured identity fails at startup
MQTT_KEEP_ALIVE_S	30
EVENT_BATCH_SIZE	100	Passages published per drain
EVENT_PUBLISH_INTERVAL_MS	1000	How often the queue is drained
TELEMETRY_INTERVAL_S	10	Heartbeat and state report
DEVICE_DEBUG_MAX_DURATION_S	900	The cabinet's own ceiling on a debug stream
HARDWARE_MODEL	read from the device tree	What the box is
REBOOT_COMMAND	/sbin/reboot	Refused outright inside a container
WATCHDOG_PATH	/dev/watchdog	Absent in a container; the controller says so
WATCHDOG_INTERVAL_S	10
FIRMWARE_PATH	/opt/wardn/bin/wardn-edge	What the supervisor starts, and what an update replaces
SENTRY_DSN · SENTRY_ENVIRONMENT	(empty) · development

Dashboard

Read at container start and written into config.js. These are the addresses a browser dials, never the service names the containers use.

Variable	Default
WARDN_API_URL	http://localhost:3000
WARDN_KEYCLOAK_URL	http://localhost:8080
WARDN_KEYCLOAK_REALM	wardn
WARDN_KEYCLOAK_CLIENT_ID	wardn-spa
WARDN_SENTRY_DSN	(empty)
WARDN_ENVIRONMENT	development

Backups

Variable	Default	Effect
BACKUP_DIR	./backups
BACKUP_KEEP_DAYS	30	Applied locally and in the remote store
BACKUP_REMOTE_ENDPOINT	(empty)	Empty keeps backups on this machine
BACKUP_REMOTE_BUCKET · _ACCESS_KEY · _SECRET_KEY	(empty)
BACKUP_ENCRYPTION_PASSPHRASE	(empty)	Mandatory once an endpoint is set; make backup refuses without it

Local infrastructure

Used by Compose and by db-init, not by the applications.

Variable	Default
POSTGRES_DB · POSTGRES_USER · POSTGRES_PASSWORD	wardn · wardn_owner · wardn_owner_dev
APP_DB_USER · APP_DB_PASSWORD	wardn_app · wardn_app_dev
MAINTENANCE_DB_USER · MAINTENANCE_DB_PASSWORD	wardn_maintenance · wardn_maintenance_dev
PARTITION_MONTHS_BACK · PARTITION_MONTHS_AHEAD	3 · 3
SEED_ENABLED · SEED_FORCE	true · false
SEED_ACTIVITY_DAYS · SEED_ACTIVITY_EVENTS_PER_DAY	30 · 12
MQTT_PORT · MQTT_TLS_PORT	1883 · 8883
EMQX_DASHBOARD_PORT · EMQX_DASHBOARD_PASSWORD	18083 · wardn_dev_dashboard
KEYCLOAK_PORT · KEYCLOAK_ADMIN · KEYCLOAK_ADMIN_PASSWORD · KEYCLOAK_REALM	8080 · admin · admin · wardn
MINIO_PORT · MINIO_CONSOLE_PORT · MINIO_ROOT_USER · MINIO_ROOT_PASSWORD	9000 · 9001 · wardn · wardn_dev_secret
MINIO_DOCS_BUCKET · MINIO_OTA_BUCKET	wardn-docs · wardn-ota
MINIO_APP_ACCESS_KEY · MINIO_APP_SECRET_KEY	wardn-backend · wardn_backend_dev_secret
TLS_DAYS · TLS_CA_DAYS	825 · 3650
BACKEND_PORT · BACKEND_2_PORT · FRONTEND_PORT	3000 · 3001 · 4200
EDGE_READER_PORT · EDGE_METRICS_PORT	9100 · 9101

---

wardn is operable. Last chapter: contributing to it.

Next → 13. Developing