# Hosted realm on Fastly Compute: implementation plan *Plan, 2026-09-26. C++ on Fastly Compute (fastlyc), state in Fastly KV. Builds on [`hosted-realms.md`](/hosted.md) (the why) and the protocol spec (the what).* ## 1. Push or pull? Both, and the store is the truth Fastly Compute runs a request handler per HTTP request. It cannot hold a listening socket, keep a TCP connection between requests, or run a background loop. Three consequences: 1. **Every message rests in KV first.** Whoever delivers it later reads it from there. This is the email property and it is what makes offline laptops work: the store is the queue. 2. **Push is possible, through Fanout.** Fastly Fanout is the edge holding long-lived WebSocket, SSE or long-poll connections on Compute's behalf. A handler hands a connection off to Fanout with a channel name; later, any handler publishes into that channel through Fastly's publish API and Fanout writes to the open socket. So a connected relay is pushed to within milliseconds of a message landing in KV, and Compute never holds the socket itself. This is the pattern Slack, Firebase and push notification services use: durable queue, notify if connected, sync on connect. 3. **Pull remains for everyone.** A relay that just connected (or a client without Fanout) fetches its backlog by prefix from KV. Push is an optimisation on top of a correct pull. There is a third delivery mode that costs nothing extra on Compute: **webhook**. A workspace that *is* reachable (a home server with a port) registers an HTTPS endpoint, and Compute POSTs each message to it as it arrives, which is exactly the HTTPS peer binding below. So per workspace, delivery is one of: `fanout` (connected), `webhook` (reachable), `pull` (neither), all backed by the same KV queue, all acked the same way. ## 2. A second binding of the protocol: HTTPS The spec's only binding today is a JSONL stream. Compute cannot originate or accept one, so the hosted realm needs an **HTTPS binding**, and because a federated peer must be able to reach airc.dev and vice versa, the binding belongs in the spec, not in the app. Proposed for draft 0.3: | purpose | request | body | response | |---|---|---|---| | deliver a message (peer face) | `POST /airc/v0/msg` | one `msg` frame (JSON) | one `ack` frame; HTTP 200 for delivered/queued, 4xx for failed | | capabilities | `GET /.well-known/airc` | — | `{"realm","proto","bindings":["https","stream"],"kid":[…]}` | | client send | `POST /airc/v0/send` | `msg` frame with local-form addresses | `ack` | | client bind / delivery mode | `POST /airc/v0/bind` | `{"names":[…],"mode":"fanout"\|"webhook"\|"pull","webhook":url}` | `bound` | | pull backlog | `GET /airc/v0/inbox?since=&max=50` | — | `{"msgs":[…],"cursor":…}` | | ack pulled/pushed messages | `POST /airc/v0/ack` | `[{"id","status","code","reason"}]` | 204 | | push connection | `GET /airc/v0/ws` (WebSocket-over-HTTP via Fanout; SSE at `/airc/v0/events`) | — | frames as WS text messages | | links, passports, subscribe | `POST /airc/v0/link`, `/passport`, `GET /airc/v0/events` | the client-face frames unchanged | their replies | Authentication maps one-to-one onto what the stream binding already has: - **Peer requests** carry the signed `msg`; no hello is needed because each request is authenticated by the message signature (§9.3). The `welcome` signature's job (proving the destination holds the realm key) is done by a signed capabilities document: `GET /.well-known/airc` returns the realm's kids and a signature over the document and a request-supplied nonce. - **Client requests** carry a signed header: `Authorization: AIRC-Key :::` with `sig` over `"airc-client\n\n\n\n\n\n"`, verified against the workspace's registered key, 300 s skew, nonce kept in KV (`ADD` mode, short TTL) against replay. This is the "key-signed client hello" of [`hosted-realms.md`](/hosted.md) §3.1 in per-request form. Discovery: `_airc-https._tcp.` SRV → host:443, or a `t=https` field in the `_airc.` TXT record; a server that publishes both bindings lets the peer choose. **Decision for Rafael:** whether spec 0.3 makes the HTTPS binding mandatory for federated servers (recommended: yes, since a Compute realm cannot reach a stream-only realm, and HTTPS is what every hosting platform can do), with the stream binding staying for the client face and for servers that prefer it. The Python `aircd` implements both sides of the HTTPS binding first. That gives the C++ service an interoperability target and lets oroboro.com talk to airc.dev on day one. ## 3. Architecture on Compute ``` relay/laptop ──HTTPS/WSS──▶ Fastly edge ── Compute handler (C++) ──┬── KV: msgs, queues, seen, links, passports, workspaces, nonces other realm ──HTTPS──────▶ │ ├── Secret Store: realm key, Fastly API token │ ◀── Fanout publish API ◀──────┤── Config Store: policy rules, limits ▼ └── Log endpoint: events.jsonl → our ingest Fanout (WS/SSE held at the edge) ``` One Compute service, one realm (`airc.dev`), workspaces as namespaces (`//airc.dev/joe/…`). The handler is a router over the paths in §2, in the style of the existing C++ Compute apps (single `main()`, route by path, `fsut*` helpers). ### 3.1 KV data model KV is eventually consistent and has no transactions, but fastlyc exposes what a queue needs: `ADD` (insert only if absent), `APPEND`, `TTL`, `IF_GENERATION_MATCH` (compare-and-swap), prefix `LIST`, `DELETE`. | key | value | notes | |---|---|---| | `m/` | the `msg` frame as JSON | TTL = message ttl (default 86400); written once | | `q//-` | `{"id","to"}` | the namespace's queue; listed by prefix `q//`; deleted on `delivered`; TTL = ttl | | `seen/` | `""` | `ADD` mode = atomic dedupe; TTL 172800 | | `nonce//` | `""` | `ADD` mode; TTL 300; replay guard | | `ws/` | `{"key":…,"mode":"fanout"\|"webhook"\|"pull","webhook":…,"binds":[…],"created":…}` | workspace record; CAS on updates | | `key/` | `{"ns":…,"pub":…}` | client key → namespace | | `link/`, `lx//` | link record; index for "links touching X" | CAS on state changes; index value = state | | `pp/`, `px//` | passport record; index | CAS on `used`/`state` | | `rw//` | last time a→b passed | reply window; TTL = window | | `cur/` | pull cursor | | Guarantees: at-least-once delivery with dedupe at the receiver (`seen/` at the destination server, plus the relay's own dedupe on the id header) and exactly-once *acknowledgement* (deleting `q/…` is idempotent). Ordering: per namespace, by key, which is by origin timestamp then id; good enough for messages, and the relay's one-in-flight rule keeps delivery ordered. ### 3.2 Request flows **Inbound from a peer** (`POST /airc/v0/msg`): parse; canonicalize; check `to.realm == airc.dev`; fetch peer's keys (`_airc.` TXT through a DNS backend, or the peer's `/.well-known/airc`, cached in KV with TTL); verify signature; `ADD seen/` (exists → 200 `delivered duplicate`); policy (config rules → links → default); write `m/`; write `q//…`; then by workspace mode: `fanout` → publish the frame to channel `ns:` and answer `queued` with code `pushed` (the relay's ack will arrive on `/ack`); `webhook` → POST to the workspace URL with a short timeout, answer its ack (delete `q/…` on delivered); `pull` → answer `queued endpoint_offline`. Log an event. Note the change from the stream binding: a Compute handler cannot wait 15 s for a relay's ack over a socket it does not hold, so **push mode acks `queued`**, and `delivered` is recorded when the relay acks. Senders already treat `queued` as success; the audit stream shows the delivery. (A future option: hold the request up to N seconds and poll for the ack; costs compute time, buys a `delivered` in the synchronous reply.) **Relay connects** (`GET /airc/v0/ws`): authenticate; hand off to Fanout on channel `ns:`; the relay then calls `GET /inbox` to drain the backlog, acks each, and thereafter receives pushes. **Relay sends** (`POST /airc/v0/send`): authenticate; `from` must be inside the workspace; local destination → same as inbound; remote → sign with the realm key, resolve the peer, `POST` to its `/airc/v0/msg` (dynamic backend), return its ack; on failure write `q//…` and answer `queued`. **Retries and expiry.** With no background loop, retries happen when something pokes the realm: a relay connecting drains its queue; a peer becoming reachable is retried by a **tick** request (`POST /airc/v0/tick`, authenticated, called every minute by an external scheduler such as our `co cron --exec`) that lists `q//` prefixes and re-sends. Expiry is free: TTLs on `m/` and `q/`. ### 3.3 Push with Fanout - Fanout must be enabled on the service (Fastly add-on). fastlyc needs one wrapper for the hand-off hostcall (`fastly_http_req_redirect_to_grip_proxy` / `handoff_fanout`); everything else is HTTP (GRIP headers on the hand-off response, `POST https://api.fastly.com/service//publish/` with the API token from the Secret Store). - WebSocket-over-HTTP: each WS event from the relay (open, text, close) arrives as an HTTP request to the handler, so client frames (`ack`, `link`, `passport`, `ping`) work over the socket unchanged. - If Fanout is unavailable or refused, the relay falls back to pull with long-poll (`GET /inbox?wait=20`) which the handler serves by re-listing the queue a few times before answering empty; acceptable for a trial, wasteful at scale. ### 3.4 Crypto and libraries (as built in H1) - **ed25519**: added to `libs/crypto` (`ed25519.cpp`, TweetNaCl-derived, constant-time field arithmetic, SHA-512 from kx). Verified against RFC 8032 and the spec §18.2 vectors, all regenerated with python-cryptography. Release speed is about 1.6 ms per sign and 3.2 ms per verify, fine for a message service; a ref10 port behind the same three functions is the upgrade if volume demands it. Two kx bugs surfaced and were fixed on the way: SHA-512 encoded the padded rather than the message length (every digest wrong; HMAC-SHA512 and any HS512 JWT were affected), and `KxCbufBase::operator==` was inverted. - **Frames**: dl atoms (`Dl`, `DlAtom`), parsed with `DlParser` in JSON mode. **Known limit:** dl holds numbers as float32 and s32, so a message's fractional `ts` cannot be read exactly from the tree; the core scans the raw JSON literal for `ts` (`aircFrameTs`), creates its own frames with whole-second timestamps, and forwards received frames verbatim (never re-serialised), so signatures always survive. Link records' `created` ordering has the same precision limit. The proper fix is a 64-bit number type in dl (an out-of-line double, as objects and arrays are stored), which also removes the 2038 limit for integer timestamps; proposed to Rafael. - **Strings and containers**: interned `KxSymbol` for every recurring name (realms, segments, keys, kids, signatures), `KxCbufBase` for built text, `KxVector` / `KxHashDict` for records; no STL. - The protocol logic has no Fastly dependency and builds for wasi (`wkjam -s PLATFORM=wasi` produces `airc_test.wasm`); H2 adds the Compute app on top of it. ### 3.5 Limits to design against Compute request time (tens of seconds), no threads, 128 MB; KV values up to 25 MB (frames are ≤ 1 MiB), KV rate limits per key; eventual consistency (seconds at worst) so a pull right after a write may miss it once, which the next push or poll covers; publish API latency (~100 ms). Costs scale with requests, not idle workspaces: an idle Fanout connection is cheap, a message is roughly five KV operations and one publish. ## 4. Registration and identity - A workspace is `POST /airc/v0/workspaces` with a public key and a requested name; the realm answers with the namespace. Signup policy is a config choice: open with rate limits and a proof-of-work or e-mail step, or invitation by passport-like token. Keys rotate through `POST /workspaces//keys`. - **Registrar mode** ([`hosted-realms.md`](/hosted.md) §4) is a separate small API that writes DNS records for `.airc.dev` through the DNS provider's API; not a Compute concern beyond hosting the endpoint. ## 5. What changes elsewhere - **Spec 0.3**: the HTTPS binding (§2 here), per-request client authentication, the `pushed` ack code, capabilities document, discovery field. Everything else is unchanged. - **Python aircd**: server and client sides of the HTTPS binding (the server side as a small HTTP/1.1 handler on the existing asyncio loop or `http.server` in a thread; stdlib either way), a `--server https://…` mode for `airc relay`/`airc send` with a workspace key, and the interop tests against a local mock of the Compute app's routes. - **Site**: the spec update, and a "run your agents from a laptop" page once airc.dev exists. ## 6. Phases | phase | deliverable | depends on | |---|---|---| | H0 | Spec 0.3: HTTPS binding + client key auth; Python aircd speaks both; interop tests | done 2026-09-26 | | H1 | C++ protocol core: addresses, frames, canonical strings, ed25519, policy, links, passports; native tests against the spec vectors | done 2026-09-26 (`src/aircore`, `airc_test`) | | H2 | Compute app `airc.dev`: `/msg`, `/send`, `/bind`, `/inbox`, `/ack`, `/link`, `/passport`, `/tick`, KV model, log events; pull + webhook delivery; registration API | done 2026-09-26 (`src/aircd_edge`); interop with the Python server verified both ways under viceroy | | H2b | Cross-realm links and passport redemption (the control messages the Python server exchanges: link offers to another realm, `_passport` redemption + grant); rate limits on signup and send; `/tick` on a schedule | H2 | | H3 | Fanout push: hostcall wrapper in fastlyc, `/ws` hand-off, publish on arrival; relay `--server` mode uses it | H2, Fanout enabled | | H4 | Registrar / dynamic DNS mode | DNS provider API | | H5 | End-to-end body encryption between endpoints | H0 | H0 and H1 can run in parallel; H2 is the bulk. A first useful milestone is H2 without Fanout: a laptop relay that polls every few seconds already gives a person offline-tolerant agent messaging under `//airc.dev//…`. ## 6a. As built (H2) `src/aircd_edge/` is the realm: `aircd_edge.cpp` (routes), `edge_store.cpp` (KV model, policy, delivery), `edge_net.cpp` (DNS over HTTPS discovery, peer verification, peer and webhook posting). Findings that changed the plan: - **Queues are one KV key per message** (`q//-`, value = the raw frame as received, `qi/` pointing back) rather than an appended log: viceroy 0.16 rejects the KV append mode, per-key TTLs expire messages individually, and settlement is a delete. Events use the same shape (`ev//...`). Listing is done in **strong** mode; an eventual listing can lag a write by seconds, which is wrong for a queue. - **Received frames are stored and forwarded byte for byte.** Signatures therefore survive dl's float32/s32 numbers; only frames this realm originates are built here (whole-second `ts`). - **Discovery runs over DNS-over-HTTPS** (a `doh` backend, cloudflare-dns.com) because Compute has no resolver; results are cached in KV for an hour. A `peers` entry in the config store overrides discovery for static or plaintext peers (tests, private federations). - **Outbound connections are dynamic backends**, one per host:port, registered at request time (`fsutRegisterDynamicBackend`, new in fastlyc). - KV keys may not contain `|`; key parts are separated with `/`. - Two fastlyc additions: `FsutKvStore::list` (prefix listing) and `fsutRegisterDynamicBackend`. ## 7. Decisions needed 1. HTTPS binding mandatory for federated servers in spec 0.3? (recommended) 2. Fanout: enable the add-on for the airc.dev service, or start with pull and webhook only? 3. Domain: `airc.dev` (to acquire) or a subdomain we already own for the first version. 4. Signup policy for workspaces: open with limits, or invitation only. 5. Push-mode acks: `queued pushed` immediately (recommended), or hold the request for a few seconds to return `delivered`.