← 12. Operating · Contents · Next → 14. Deploying to production

13. Developing¶

Repository layout¶

db/
├── migrations/     schema, partitions, indexes, grants: the source of truth
├── seed/           the development dataset
├── checks/         assertions run by `make verify`
└── init/           the migration and seed runner
backend/
└── src/            NestJS: API, auth, provisioning, audit, MQTT consumer, live feed
edge/
├── wardn-domain/  decision logic: no I/O, no clock, no database
├── wardn-osdp/    the OSDP protocol, both roles: no_std, shared with a node firmware
├── wardn-node/    a bus node's logic, on any board: configuration, its side of OSDP. no_std
├── wardn-node-runtime/  what a node does with its hardware, on any chip: bus, reader, keypad, LED, screen, relay, contact
├── wardn-rp2040/  a node on a Pico: which pin can do what, where the configuration sits in flash,
│                   and (feature `embassy`) opening its devices
├── wardn-node-config/  turns a node's TOML configuration into a UF2 file for the Pico
├── wardn-node-pico/    the node firmware's binary, outside the workspace: Pico target only
├── wardn-edge/    adapters: encrypted store, reader bus, relay, forwarder, heartbeat, OTA
└── bootstrap/      wiring and fallback rights for a controller that boots with nothing
frontend/           four pnpm workspace packages (see below)
├── dashboard/      @wardn/dashboard: the Angular app, core (auth, API, live socket),
│                   shared, one folder per screen
├── packages/
│   ├── ui/         @wardn/ui: the shared component library, built with ng-packagr
│   └── tokens/     @wardn/tokens: design tokens, exported as CSS and a Tailwind theme
└── storybook/      the component library's own docs site, built in CI
dev/                everything specific to running wardn locally: docker-compose.yml,
                    Makefile, .env.example, infra/ (idp realm, MinIO seed, TLS CA),
                    scripts/ (the executable scenarios, all of them run in CI)
infra/aws/          an AWS installation: IoT Core, Cognito and S3 in Terraform, and a
                    compose file running the server half locally against them (chapter 14)
docs/               this documentation

Where things live¶

Looking for	Look in
The decision rules	`edge/wardn-domain/src/decision.rs`
Value normalisation	`edge/wardn-domain/src/credential.rs` and `backend/src/common/credential.ts`
The controller's local store	`edge/wardn-edge/src/store.rs`
Store & Forward	`edge/wardn-edge/src/store.rs` and `mqtt.rs`
OSDP framing	`edge/wardn-osdp/` (pure, both roles) and `wardn-edge/src/osdp_bus.rs` (I/O)
What a reader or door node answers	`edge/wardn-node/src/node.rs` and `door.rs`, run by `edge/wardn-node-pico/`
Where a credential comes from	the `ReaderInput` port, `edge/wardn-edge/src/reader_input.rs`
The rights queue	`backend/src/sync/`
The fingerprint	`backend/src/sync/fingerprint.ts` and `store.rs::rights_fingerprint`
Authentication	`backend/src/auth/`
Workspace confinement	`workspaceScope()` in `backend/src/auth/principal.ts`, and every repository query
The three journals	`db/migrations/0051_activity_logs.sql` to `0053`, `backend/src/audit/`
The schema, roles and grants	`db/migrations/`

Running the suites¶

make edge-test        # cargo test + clippy -D warnings + cargo fmt --check
cd backend && pnpm install && npx tsc --noEmit && npx jest
cd backend && pnpm test:integration   # needs `make up`; see below
cd frontend && pnpm install
pnpm --filter @wardn/ui run build       # the dashboard depends on this package's dist/
pnpm --filter @wardn/dashboard run build
pnpm --filter @wardn/dashboard run test

@wardn/ui has to build first. The dashboard imports its compiled output, not its source. pnpm build on the dashboard is the type check: Angular compiles templates, so a signal renamed in a component fails there rather than in a browser.

dev/Makefile's dashboard-build target runs the same two builds, and is what CI's dashboard job uses.

Three kinds of test¶

Unit tests, next to the code¶

Rust tests live in the module they exercise. TypeScript tests sit beside their subject as *.spec.ts. They cover the things that are cheap to get wrong and expensive to notice:

Every branch of the decision function, including the carpool case, the half-open validity interval, and the fact that candidate ordering does not change the outcome.
The queue: that it is bounded, that it drops the oldest, and that drops are counted.
The clock: that it catches a board that came back in the past, tolerates a small backwards slew, and keeps saying so until real time catches up.
The rights delta guards: a stale revision is inert, an incomplete upsert is rejected, an empty full sync does not empty a populated controller.
The fingerprint, pinned on both sides by the same golden vector.
pagination cursors, RFC 9457 problem shapes, Sentry scrubbing, workspace scoping.

The OSDP decoder is additionally held to property tests: a decoder is the one part of a reader adapter where a mistake is silent. A bad CRC reads as a broken cable, a misread bit count as somebody else's badge.

These run with no database at all. A backend unit test hands its subject a fake that records the SQL it was given, which pins the shape of a statement and nothing about what the statement does.

Database tests, against a real schema¶

*.it.spec.ts, run by pnpm test:integration, separately from npx jest so the ordinary suite stays offline. They exist for the claims a recorded statement cannot support:

A row lock ordered two callers. Two requests replacing the same right's credentials leave one version in force, and the loser is refused rather than adding a second.
A predicate matched the data, not merely the intended text — a jsonb containment, an ILIKE pattern, a partial index.
A role was allowed, or was not. Erasure and retention connect as the maintenance role, which is the whole point of there being two.
Two statements were read at one instant, such as a revision and the topology it carries.

The suite creates its own wardn_it database on the server the dev stack runs, replays db/migrations into it and tracks them the same way, so a second run costs one query. It does not seed: a test that leans on demo data is a test that changes meaning when the demo does. What it does not create either is the login roles — those belong to the cluster make up sets up, and inventing them would let a test pass against privileges production does not grant. IT_DATABASE_URL points it somewhere else when the port differs, which locally it usually does.

Between tests every business table is emptied. The three catalogs a migration ships — transformers, identifier types, reader formats — survive it, so a fixture names a builtin format rather than inventing one.

They run one file at a time (--runInBand): they share one database, and a test that truncates a table while another reads it proves nothing.

Scenarios, against the real stack¶

Each script in dev/scripts/ starts from a running stack, exercises one claim end to end, prints what it checked, and exits non-zero when something did not hold.

Script	The claim
`demo.sh`	Every decision branch behaves as documented
`matrix.sh`	Every frame the seed can make lands where it should, and no generated one opens a door
`offline.sh`	An uplink outage loses no passage, and replays are flagged
`sync.sh`	A right changed through the API changes what the door does
`api.sh`	Authentication, workspace confinement, immutability, an attributed audit trail
`dashboard.sh`	A passage reaches an open feed live, and a second workspace hears silence
`fleet.sh`	The heartbeat is real, a log level changes without a restart, debug streams and stores nothing
`commands.sh`	A click moves a physical door. A mode change changes how the controller decides
`ota.sh`	A release installs. A corrupt one is refused. A broken one is rolled back
`compliance.sh`	A request can be traced, a person exported and erased, retention purges what expired
`two-instances.sh`	One passage, one row, both dashboards
`mtls.sh`	Both ends prove who they are. A stranger is refused. A controller cannot reach another's topics
`load.sh`	Every passage decided inside the budget and accounted for
`backup.sh` / `backup-verify.sh` / `restore.sh`	A backup is taken, restores, and says what it cannot restore
`tls-check.sh`	What every certificate in the stack has left

They are run with make <target>. See 2. Getting started.

Why scripts and not a test framework? Each of these asserts something that a green HTTP response would not establish: that a physical door moved, that a second workspace heard nothing, that a controller came back on the firmware it had. They are also the fastest way for a newcomer to watch a feature work.

matrix.sh is the one built for breadth rather than for a single claim, and it carries three oracles because one would have to lie about part of it. The palette declares what each situation should produce, and walks every cell of the decision table. The format vectors reuse the fixtures the unit tests already pin, so the DSL is never re-implemented in the harness — the run only checks that the live controller agrees with what two implementations already agree on. The generated frames predict nothing at all, and are held to invariants instead: chiefly that none of them ever opens a door.

Correlation is by frame, not by position. Every recorded passage carries raw_frame, the frame as received before any format ran, so a run of thousands joins back to its own cases in one query. The two outcomes that record nothing — a line that is not a frame, and a frame from an address nobody declared — are read off the controller's log and confirmed by the absence of any row at all.

Continuous integration¶

flowchart TD
    I["images<br/>build backend, controller, dashboard once"]
    subgraph NB["No image needed"]
        E["edge: fmt, clippy, cargo test"]
        B["backend: tsc, jest"]
        BD["backend: the same code against a real schema"]
        D["dashboard: build, test"]
        S["socle: migrations, grants, idempotent seed"]
        K["idp: the provider issues the right roles"]
    end
    I --> ES["edge scenarios"]
    I --> OU["outage and replay"]
    I --> AP["API behaviour"]
    I --> RS["rights reach the door"]
    I --> DL["a passage reaches the dashboard"]
    I --> FL["fleet, heartbeat and debug"]
    I --> RC["a command reaches the door"]
    I --> LD["a burst of passages"]
    I --> MT["a controller proves who it is"]
    I --> TI["two backends, one event"]
    I --> CO["retention, tracing and erasure"]
    I --> OT["a firmware release reaches the controller"]

Two things about the shape are worth knowing:

The images are built once and handed around. Building the controller means compiling SQLCipher and a vendored OpenSSL from source. Every scenario job paying that separately spent most of a run rebuilding what the job next door was building at the same moment. It also means every scenario tests the same binary, which the old arrangement only assumed.

The socle job proves idempotence rather than asserting it. It applies the migrations and seed, verifies, re-runs them, re-runs the seed under SEED_FORCE, verifies again, and then counts rows: a broken idempotency claim otherwise shows up as duplicated demo data weeks later.

Every job dumps the relevant container logs on failure and tears the stack down whatever happens.

Conventions¶

Code¶

Code must be simple, and as functional as it can reasonably be.
Types express intent. RawValue and NormalizedValue are distinct types so a raw read can never be used as a lookup key by accident. Principal is a union rather than one type with an optional workspace, so confinement cannot be forgotten.
Self-explanatory code over comments. A precise name and a small, single-purpose body cannot drift from the code. A comment can.
Comments carry the non-obvious "why": an invariant the types cannot express, an ordering constraint, a deliberate deviation. Never a paraphrase of the line below.
Document contracts, not usage. A function's documentation describes its inputs, its result, its guarantees and its failure modes. It never describes where it happens to be called from today.
No early-return or early-continue for its own sake.

The two implementations that must agree¶

Two pieces of logic exist on both sides of the wire, and each is pinned by a shared vector:

What	Controller	Server
Value normalisation	`wardn-domain/src/credential.rs`	`backend/src/common/credential.ts`
The rights fingerprint	`wardn-edge/src/store.rs`	`backend/src/sync/fingerprint.ts`

Changing either one without the other surfaces in production as a credential that never matches, or a controller resynchronised on every pass. Both have a test asserting the same literal on each side. Change both, or change neither.

Adding a feature¶

flowchart TD
    A["Decide where it belongs:<br/>domain, adapter, backend, dashboard"] --> B["Write the code and its test"]
    B --> C["Add or extend a scenario<br/>if a green response would not prove it"]
    C --> D["make edge-test · jest · tsc · pnpm build"]
    D --> E["Update the chapter this documentation<br/>describes it in"]

The last step is not optional. This documentation is meant to be readable on its own and to describe what the software actually does. A feature that changes behaviour and leaves the chapter behind has made the documentation wrong, not merely incomplete.

Adding an environment variable¶

Three places, always:

Read it with a documented default in config.ts or config.rs.
Add it to .env.example with the comment saying why the default is what it is.
Add it to the configuration reference in 12. Operating.

Changing the schema¶

db/migrations/ is a baseline, not a history: one file per table, each holding the table as it is today with its constraints, indexes, grants and, for a catalog, its built-in rows. db-init applies them in order on every start, and the number orders them so a table comes after the ones it references.

Files	Hold
`0000`	Extensions, the application role, and `wardn_app_may()`, the one way a table states what that role may do with it
`0010`–`0012`	The three catalogs: transformers, identifier types, reader formats
`0020`–`0027`	Workspaces, API keys, zones, controllers, doors, readers and their links
`0030`–`0032`	Users, access rights, credentials
`0040`–`0044`	What the fleet reports and the queues that serve it
`0050`–`0053`	The partition functions retention runs on, then the three journals
`0060`–`0061`	Alert read markers and realtime tickets

A schema change edits the file that owns the object — a new column goes into the CREATE TABLE it belongs to, never into an ALTER TABLE in a new file. A new table is a new file, numbered among the ones it belongs with. Reading the schema is then reading the table's own file, not replaying a history.

The cost is that an existing database cannot be migrated forward: db-init skips a version it has already applied, so editing a file changes nothing until the volume is wiped. Locally that is make reset.

This holds until the first production deployment, at which point the baseline freezes and changes start being appended again.

If the change touches grants or partitioning, extend db/checks/verify.sql so make verify covers it.

Working on one piece at a time¶

# The controller, on its own: it needs nothing else in the stack.
make edge && make edge-logs
make badge VALUE=BDG-0001

# The backend, against the local socle.
(cd dev && docker compose up -d --wait postgres emqx idp)
(cd dev && docker compose up db-init)
cd backend && pnpm start:dev

# The dashboard, from source with live reload.
make dashboard-dev

The controller is the easiest to develop against: it depends on nothing but its own database, which is the whole point of the design.

Debugging¶

make logs                       # everything
make edge-logs                  # the controller
(cd dev && docker compose logs backend)  # the server
make psql                       # a database shell
curl localhost:9101/metrics     # the controller's own metrics
open http://localhost:3000/docs # the generated API reference

Raise the controller's verbosity without restarting it. Restarting to read a log loses the moment being investigated:

curl -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"level":"debug"}' http://localhost:3000/api/v1/devices/$DEVICE/actions/log-level

And to see raw frames as they arrive, open the debug console on /controllers/<id>/debug. It is time-boxed and stores nothing.

To follow one request through the backend:

(cd dev && OTEL_TRACES_CONSOLE=true docker compose up -d backend)
curl -i -H "Authorization: Bearer $TOKEN" http://localhost:3000/api/v1/zones
# quote the X-Trace-Id from the response in the backend logs

The stack is understood end to end. One more chapter: running it for real.

Next → 14. Deploying to production