Deploying klock v2

Turning on klock v2 in production: the root key, the operational keys, the
ORM prerelease, the klock service, the API migrations, and the capability
enforcement rollout, with the rollback for each step.

This page is for whoever operates api.prod-orm.io and klock.prod-orm.io.
Nothing in the repository deploys any of it for you: every step below is a
person running a command. For what klock v2 is, read the
klock guide and the
capabilities guide first.

The order, and why

v2 fails closed at every stage. Until the root is pinned and a keyset that
certifies the grant key is published, the API answers every v2 route with
503 klock_v2_unavailable and v1 keeps working untouched. So the order is
chosen to make each step safe on its own:

Steps 4 and 5 can swap. Step 6 must come before step 7: a service binding to
a Worker that does not exist fails wrangler deploy outright.

Where does it stand?

Each stage has an answer you can see from outside, so a half-finished rollout
tells you which step it stopped at:

1. The root key ceremony

The root is an offline Ed25519 key. It signs only keyset documents, and
the CLI, the MCP Worker, the console and Go pin its public half. Losing it
means a CLI release to re-pin, so make two (a primary and a backup kept
separately) and pin both.

The ceremony is api/scripts/klock-keyset.ts. Run it from api/ on an
offline machine: bun run scripts/klock-keyset.ts <command>. It never
uploads anything, never touches a database, opens no network connection and
starts no child process. It makes keys and signs documents on this machine,
and prints the SQL and wrangler commands for you to run later, where you
choose.

Command	Makes	Flags
`root`	An offline Ed25519 root. Prints `{kid, x}` to pin.	`--kid root-2026 --out FILE`
`keygen`	An operational key: `grant` (`KLOCK_GRANT_KEY`), `klock` (`KLOCK_SERVICE_KEY`), or an RPC key (`--service api`: `API_RPC_KEY`; `--service klock`: `KLOCK_RPC_KEY`). Prints the public half.	`--use grant\|klock\|rpc [--service api\|klock] [--kid …] --out FILE`
`sign`	A keyset `{seq, iat, expires, keys}`, signed by the root and verified with `verifyKeyset` before it is printed	`--root-key FILE --key PUB.json [--key PUB.json …] [--seq 1] [--days 89] [--out FILE]`
`rotate`	The next keyset, `seq + 1` (rotating keys)	`--current KEYSET --root-key FILE [--add PUB.json]… [--retire KID[=days]]… [--drop KID]… [--days 89] [--out FILE]`
`verify`	Checks a keyset against the given roots, or the pinned ones	`--keyset FILE [--root KID:X]… [--min-seq N]`

How it handles keys:

A private key is a 32-byte Ed25519 seed, b64url, written to --out with
mode 0600 and O_EXCL, so an existing file is never overwritten.
--out - is refused, and a seed is never printed. A key file with group or
other permission bits is refused when read (chmod 600 it).

The public half is written beside it as <out>.pub.json. That is what
sign --key and rotate --add take.

stdout carries only public, machine-readable output: a {kid, x}, a
public key's JSON, the keyset JWS, or a verify summary. stderr carries
the instructions: the pin lines, the wrangler secrets-store commands, the
vars to set and the SQL that publishes a keyset.

Make the roots

$ bun run scripts/klock-keyset.ts root --kid root-2026 --out root-2026.key
{"kid":"root-2026","x":"…"}
$ bun run scripts/klock-keyset.ts root --kid root-2026-b --out root-2026-b.key
{"kid":"root-2026-b","x":"…"}

stderr prints the two pin lines for src/klock/keys.ts and
outh/internal/klockv2/roots.go. Move each seed to a hardware token or a
password manager, keep the backup apart from the primary, and delete these
copies once stored.

Make the operational keys

$ bun run scripts/klock-keyset.ts keygen --use grant --out grant.key
$ bun run scripts/klock-keyset.ts keygen --use klock --out klock.key
$ bun run scripts/klock-keyset.ts keygen --use rpc --service api   --out api-rpc.key
$ bun run scripts/klock-keyset.ts keygen --use rpc --service klock --out klock-rpc.key

A grant key gets a kid g-YYYYMM-xxxx and a klock key k-YYYYMM-xxxx
(or pass --kid). For the klock key, stderr also prints the
KLOCK_SERVICE_KID var. For each RPC key, it prints the var the other
side pins: API_RPC_PUB on the klock Worker for svc:api, and
KLOCK_RPC_PUB on the API for svc:klock. RPC keys are not in the keyset.

Sign keyset seq 1

$ bun run scripts/klock-keyset.ts sign --root-key root-2026.key \
    --key grant.key.pub.json --key klock.key.pub.json --seq 1 --out keyset-1.jws

Every key is current from now until the keyset expires (89 days by default,
90 at most). The document is signed with typ: klock-keyset+jws and the
root's kid, and verified before it is printed. stderr prints the
INSERT INTO klock_keysets … SQL and the bun run d1 --command "…" line for
step 5, and notes when the root is not pinned in
the build that ran it.

Verify it

$ bun run scripts/klock-keyset.ts verify --keyset keyset-1.jws --root root-2026:<x>

Without --root, it verifies against the roots pinned in this build
(PRORM_KLOCK_ROOTS in a non-production shell). --min-seq refuses a
rollback. The API's TypeScript verifyKeyset and Go's
klockv2.VerifyKeyset both accept the tool's keysets.

Rules the verifier enforces (src/klock/keys.ts):

A root kid matches root-[a-z0-9-]{1,32}.

An operational kid is g-YYYYMM-<4 hex> for use: 'grant' and
k-YYYYMM-<4 hex> for use: 'klock'; the prefix must match the use.

A keyset lives at most 90 days, and seq starts at 1 and only grows:
clients refuse a lower seq than they have seen.

[!CAUTION] The root seed never touches a networked machine
Run the ceremony offline, move each root key file to a hardware token or a password
manager, and keep a second root in a different place. A leaked root lets
whoever holds it publish a keyset with their own grant key, which every
client would trust.

Then pin the root:

src/klock/keys.ts: add { kid, x } to KLOCK_ROOTS. It is empty today,
which is why every keyset fails verification.

-export const KLOCK_ROOTS: readonly KlockRoot[] = Object.freeze([]);
+export const KLOCK_ROOTS: readonly KlockRoot[] = Object.freeze([
+  { kid: 'root-2026', x: '<the primary root x>' },
+  { kid: 'root-2026-b', x: '<the backup root x>' },
+]);

outh/internal/klockv2/roots.go: add the same entry to Roots.
roots_test.go holds the two lists equal, so a mismatch fails the Go tests.

Release the CLI and outh with the pinned roots (the ORM release in step 3).

Development and staging use their own root through PRORM_KLOCK_ROOTS, a
JSON array of {kid, x}. Production builds ignore it.

Rollback. Nothing is live yet. If a root leaks before release, generate a
new one and re-pin.

2. Keys and secrets

Name	Kind	Held by	Bound in	Read by
`KLOCK_GRANT_KEY`	Secret: Ed25519 seed, b64url (or hex), 32 bytes	API	`wrangler.jsonc` `secrets_store_secrets`, commented out until the store holds it	`controllers/klockGrants.ts`; loaded by `worker-entry.ts` `SECRETS`
`API_RPC_KEY`	Secret: Ed25519 seed for `svc:api`	API	`wrangler.jsonc` `secrets_store_secrets`, declared and commented out until the store holds it	`controllers/klockGrants.ts` (signs calls into `KlockInternal`); loaded by `worker-entry.ts` `SECRETS`
`KLOCK_RPC_PUB`	Public key (b64url, 32 bytes) of `svc:klock`	API	`wrangler.jsonc` `vars` (empty)	`controllers/klockGrants.ts`: empty means `ApiInternal` refuses every call (`klock_unconfigured`), including `compiledRules` and `recordFindings`
`KLOCK_SERVICE_KEY`	Secret: Ed25519 seed, use `klock`	klock Worker	`wrangler.klock.jsonc` `secrets_store_secrets`, commented out	`klock/env.ts`; signs sync answers
`KLOCK_SERVICE_KID`	Var: the `k-…` kid of that key in the keyset	klock Worker	`wrangler.klock.jsonc` `vars` (empty)	`klock/lib/trust.ts`: empty means sync answers `503` (`klock_key_unpublished`)
`KLOCK_RPC_KEY`	Secret: Ed25519 seed for `svc:klock`	klock Worker	`wrangler.klock.jsonc` `secrets_store_secrets`, commented out	signs calls to `ApiInternal`; its public half is the API's `KLOCK_RPC_PUB`
`API_RPC_PUB`	Public key (b64url, 32 bytes) of `svc:api`	klock Worker	`wrangler.klock.jsonc` `vars` (empty)	`klock/env.ts`: empty means every `KlockInternal` call is refused
`SUPPORT_DROP_DEVICE_ID`	Var: the staff device (`dev_…`) that `support:` transfers are sealed to	klock Worker	`wrangler.klock.jsonc` `vars` (empty)	`klock/lib/transfer-policy.ts`: empty means `support:` answers `503 klock_unconfigured` (step 6)
`KLOCK_AUTHORITY_TOKEN`	Secret: a second bearer for the Kerberos authority	the authority (`api/kerberos`)	the authority's environment; optional	admitted on `POST [/{team}]/v1/csr` only (step 10)

The two RPC key pairs come from the ceremony tool
(keygen --use rpc --service api|klock, step 1).
They are not in the keyset: each side pins the other's public key directly,
so each var must be the public half of the other side's secret:

Secret	Held by	Its public half is the var	On
`API_RPC_KEY` (`svc:api`)	API	`API_RPC_PUB`	klock Worker
`KLOCK_RPC_KEY` (`svc:klock`)	klock Worker	`KLOCK_RPC_PUB`	API

To put a seed in the Secrets Store, paste the key file's contents at the
prompt of the command the tool prints; never put it on a command line:

$ wrangler secrets-store secret create <STORE_ID> --name KLOCK_GRANT_KEY --scopes workers --remote

For the keys the API holds, bun run secrets set <NAME> < <file> also works.

The API signs grants only when three things line up (controllers/klockGrants.ts):
KLOCK_GRANT_KEY is set; the newest klock_keysets row verifies against the
pinned roots; and that keyset has a current use: 'grant' entry whose x is
the key's public half. Anything else is 503 klock_v2_unavailable, and
nothing is cached negatively, so publishing a keyset takes effect on the next
request.

Rollback. Removing KLOCK_GRANT_KEY (or its binding) turns v2 signing off:
v2 routes answer 503 klock_v2_unavailable and v1 is untouched. Grants
already issued stay valid until their exp (at most 7 days); to end them
early, publish a keyset without their kid (see rotation).

3. Publish the ORM prerelease

The API depends on the published package (api/package.json:
"ts-prorm-orm": "^4.9.2"), and until that version exists it resolves
the new subpaths through the ordered paths fallback in api/tsconfig.json:
the installed dist first, then ../src.

Pin the roots (step 1) and set the ORM version to 4.9.2.

Check that package.json#exports has "./mcp/registry" (it does now):
the API imports ts-prorm-orm/mcp/registry for the MCP forward.

Publish under a prerelease dist-tag, so latest is untouched.

In api/, run pnpm install so pnpm-lock.yaml records it.

Drop the ../src fallbacks from api/tsconfig.json paths. For
ts-prorm-orm/mcp/registry the order is source first today (the installed
4.7.0 build has a registry without needs, and the forward fails closed with
503 on one), so swap it to dist first once the new release is installed,
then remove the source entry.

Rollback. Pin the API back to the previous version and reinstall. v1
clients are unaffected either way.

4. API migrations 0026 to 0030

All five are additive: every column is nullable or defaulted, and there is no
down (database/migrations/types.ts: a migration may only add). They are
applied in the order of database/migrations/index.ts.

Version	File	Adds
`0026`	`klock-sessions.ts`	`sessions`: kind, binding, device, team, capabilities, limits, step-up and check-in, rotation, freezing, revocation reason and actor, usage, TLS fingerprint. `audit_log`: level, category, device, session, actor. `passwordless_sessions` and `transfer_jobs`: the linked session row and protocol. Backfills each session's `kind` (`assumed`, `cli-legacy` or `browser`).
`0027`	`access-rules.ts`	The `access_rules` table, and the change-log triggers re-issued. Existing `database_access.blockedIps` are not copied.
`0028`	`behaviour-profiles.ts`	`behaviour_profiles`, `behaviour_baselines`, `behaviour_findings`.
`0029`	`klock-keys.ts`	`klock_keysets` and `klock_revocations`.
`0030`	`team-ownership.ts`	`ownerKind` and `teamId` on `databases` and `data_stores`, `ownerKind` on `transfer_jobs`, a billing account on `teams`, `teamId` on `subscriptions` and `invoices`, and an index on `project_resources (targetKind, targetId)`. Existing multi-team attachments are reported, not changed.

Before applying, check that production has transfer_jobs (migration
0022). 0026 adds columns only to tables that exist and lack them, so a
missing transfer_jobs is skipped silently rather than failing.

$ cd api
$ bun run migrate:d1 -- --status
$ bun run migrate:d1 -- --rehearsal
$ bun run migrate:d1 -- --dry-run
$ bun run migrate:d1

In order: what is applied; the whole run against prorm-rehearsal; the plan
against production, applying nothing; and the real run.

A quick check that transfer_jobs is there before you start:

$ bun run d1 --command "SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'transfer_jobs'"

migrate:d1 runs the migrator through a Worker that is never deployed
(wrangler.migrate.jsonc), with --env-file /dev/null so the repository's
.env is not used.

Rollback. There is no down migration. The new columns and tables are
unused until the code that reads them is deployed, so roll the code back and
leave the schema. Restore from a D1 Time Travel point only if a migration
itself damaged data.

5. Publish the keyset

Insert the root-signed keyset document from step 1 into the API's D1. sign
printed this exact statement on stderr:

Run it with bun run d1 --command "…" (wrangler d1 execute prorm --remote).
doc is stored and served exactly as signed; verifiers check the bytes and
never re-serialise it.

Check it:

GET https://api.prod-orm.io/.well-known/klock-keys.json serves the highest
seq.

GET https://api.prod-orm.io/.well-known/klock-configuration advertises v2.

$ curl -sD - -o /dev/null https://api.prod-orm.io/.well-known/klock-keys.json | grep -i klock-keyset-seq
klock-keyset-seq: 1
$ curl -s https://api.prod-orm.io/.well-known/klock-configuration | jq .v2
false

v2 is false here, as it should be: KLOCK_GRANT_KEY is bound in step 7.
It turns true on the first request after that deploy, provided the key's
public half is the keyset's current grant key.

The klock Worker does not need its own copy: it fetches the keyset from the
API over RPC and caches it (its own klock_keysets table, and in memory for an
hour), and serves it at GET https://klock.prod-orm.io/v2/keys.

[!CAUTION]
Never delete or lower a seq: clients that saw it refuse anything older.

Rollback. To withdraw a key, publish a higher seq without it.

6. The klock Worker

The klock service is api/klock/ with its own config, api/wrangler.klock.jsonc.

Create its D1 database

$ cd api
$ wrangler d1 create prorm-klock

Put the printed id in database_id (it is all zeros today):

"d1_databases": [
  {
    "binding": "DB",
    "database_name": "prorm-klock",
    "database_id": "<the id wrangler printed>",
    "migrations_dir": "klock/migrations"
  }
],

Apply its migrations

File	Does
`klock/migrations/0001_klock_init.sql`	The device registry (`klock_devices`), the credential ledger, the sync log, pairings, the event chain, commands, the keyset cache and the ticket and RPC nonce tables
`klock/migrations/0002_klock_transfers.sql`	Recreates `klock_transfers` (0001 declared it ahead of the protocol, and nothing ever wrote it), and adds `klock_transfer_chunks` and `klock_transfer_nonces` for file transfer

$ wrangler d1 migrations list prorm-klock -c wrangler.klock.jsonc --remote
$ wrangler d1 migrations apply prorm-klock -c wrangler.klock.jsonc --remote

0002 begins with DROP TABLE IF EXISTS klock_transfers. That loses
nothing: every /v2/transfers route answered 501 before it, so the table
is empty. A Worker already on 0001 needs only the second command.

Bind its keys

Put KLOCK_SERVICE_KEY and KLOCK_RPC_KEY in the Secrets Store, then
uncomment the secrets_store_secrets block. Set the vars:

"vars": {
  "NODE_ENV": "production",
  "API_ORIGIN": "https://api.prod-orm.io",
  "KLOCK_SERVICE_KID": "k-202609-7c2e",   // the klock key's kid in the keyset
  "API_RPC_PUB": "<b64url public half of API_RPC_KEY>",
  "SUPPORT_DROP_DEVICE_ID": ""            // the staff drop device; see below
},

The relay bucket and the support drop

File transfer keeps sealed chunks (ciphertext only, at xfer/<id>/<n>) in
the R2 bucket prorm-klock-relay, bound as RELAY. The binding is commented
out in wrangler.klock.jsonc, because binding a bucket that does not exist
fails the deploy. Until it is bound, every /v2/transfers route answers
503 klock_unconfigured and the CLI says "coming soon". To turn it on,
create the bucket with a 24-hour lifecycle rule, then uncomment the block:

$ wrangler r2 bucket create prorm-klock-relay
$ wrangler r2 bucket lifecycle add prorm-klock-relay expire-24h --expire-days 1

"r2_buckets": [
  { "binding": "RELAY", "bucket_name": "prorm-klock-relay" }
],

The lifecycle rule is the backstop: klock deletes a transfer's objects when
it ends, and the hourly sweep (step 9) expires
transfers after 24 hours.

For the support drop (prorm klock cp … support:), enrol a staff device,
then set its id as SUPPORT_DROP_DEVICE_ID. Its session needs
admin:support or admin:dev to open the drop. Left empty, support:
answers 503 klock_unconfigured; transfers between devices still work.

Deploy

$ wrangler deploy --dry-run -c wrangler.klock.jsonc
$ wrangler deploy -c wrangler.klock.jsonc
$ curl -s -o /dev/null -w '%{http_code}\n' https://klock.prod-orm.io/v2/keys
200

It serves klock.prod-orm.io as a custom domain. /v2/keys answers 503
(no_keyset) instead of 200 until the Worker can reach the API over RPC,
which needs step 7. The deploy also registers the hourly cron
(step 9).

The Worker already binds API to the control plane's ApiInternal
entrypoint. Pairing is live: the PairingRoom Durable Object (one per
pairing) holds its state. Set the optional var CONSOLE_ORIGIN if the console
is not at https://app.prod-orm.io; it is the origin of the #/pair link a
pairing hands the new device.

Rollback. Unbind KLOCK in the API first (step 7), then roll the Worker
back or delete it. Devices keep their grants until checkInBy and exp; the
API remains the authority for sessions and revocation.

7. Bind the klock service to the API

In api/wrangler.jsonc:

Uncomment the services entry
{ "binding": "KLOCK", "service": "prorm-klock", "entrypoint": "KlockInternal" }.

Uncomment the KLOCK_GRANT_KEY and API_RPC_KEY entries, which are
declared (commented out) at the end of secrets_store_secrets, once the
store holds both.

Set KLOCK_RPC_PUB in vars (declared, empty).

Deploy the API.

The result, for the parts this step touches:

"services": [
  { "binding": "KLOCK", "service": "prorm-klock", "entrypoint": "KlockInternal" }
],

"secrets_store_secrets": [
  // … the existing entries …
  { "binding": "KLOCK_GRANT_KEY", "store_id": "<store id>", "secret_name": "KLOCK_GRANT_KEY" },
  { "binding": "API_RPC_KEY", "store_id": "<store id>", "secret_name": "API_RPC_KEY" }
],

"vars": {
  // … the existing vars …
  "KLOCK_RPC_PUB": "<b64url public half of KLOCK_RPC_KEY>"
},

[!WARNING]
KLOCK_RPC_PUB must be the public half of the klock Worker's
KLOCK_RPC_KEY, and the klock Worker's API_RPC_PUB the public half of
API_RPC_KEY. Crossing them, or leaving either empty, fails closed: every
RPC call is refused and device management answers 503 klock_unconfigured.

Then check it end to end, from a machine with a CLI built with the pinned
root:

$ curl -s https://api.prod-orm.io/.well-known/klock-configuration | jq .v2
true
$ prorm login
$ prorm klock sync
$ prorm devices list

prorm login should sign in with v2 (it prints

klock v2 interactive
session

); if it falls back to a legacy token, v2 was still false when it
asked. klock sync proves the klock Worker, and devices list the RPC keys in
both directions.

The DEVICE_GUARD Durable Object (single-use request nonces) is already bound
and migrated (v3).

The device-management routes (/security/devices…, the team and staff
device routes) answer 503 klock_unconfigured until both the KLOCK binding
and API_RPC_KEY are in place; the CLI and console show that as "coming
soon".

The klock Worker also calls ApiInternal.compiledRules and
ApiInternal.recordFindings, to evaluate access rules and behaviour
profiles at sync, enrolment and credential issue. It fails closed: while
KLOCK_RPC_PUB is wrong or empty, sync gives no grant
(grantError: "policy_unavailable"), and enrolment and credential issue
answer 503 policy_unavailable. A healthy prorm klock sync therefore also
proves this direction.

The MCP Worker. workers/mcp/wrangler.toml binds API to the control
plane (prorm-control-plane), which it uses for /me/capabilities, the
delegated-token routes and /internal/mcp/forward. API_TRANSPORT = "url"
(or no binding) falls back to plain fetch to API_URL. Set the optional
secret MCP_CONFIRM_KEY (the HMAC key for the confirmation tokens it gives
legacy bearers) with wrangler secret put MCP_CONFIRM_KEY. Never set
PRORM_KLOCK_ROOTS there in production. Until a keyset is published it
answers klock callers 503 no_keyset, like the API and the klock service.

Rollback. Comment the binding out again and redeploy. v2 sessions that
need the klock service (enrolment, device lists) then answer 503.

8. The enforcement rollout

Capability checks are controlled by three variables, read in
api/middleware/capabilities.js:

Variable	Values	Default
`CAPABILITY_ENFORCEMENT`	`off`, `report`, `enforce` (or an area list)	`off`
`CAPABILITY_ENFORCEMENT_AREAS`	a comma list of `area` or `area=mode` for `resource`, `team`, `admin`, for example `resource=enforce,team=report`	empty
`CAPABILITY_E3`	`1` (also `true`, `yes`, `on`)	off

off skips the ceiling checks entirely, so an existing account sees exactly
what it saw before.

report records access.would-deny in the audit log and lets the request
through.

enforce answers 403 capability_missing with {needs, unlock}, or
step_up_required.

Each area has its own mode. area=mode in the list wins; a bare
area takes CAPABILITY_ENFORCEMENT (or enforce when that is not a
mode); an unlisted area takes CAPABILITY_ENFORCEMENT when it is a mode,
report when only a list is set, and off when nothing is set. The full
precedence is in the capabilities guide.

The admin area is held at report even under enforce until
CAPABILITY_E3=1. A legacy CLI admin token carries no admin:* by design,
so enforcing admin earlier would lock every CLI admin out.

A member who reaches a resource through a team is checked against their
role in every mode. Team access is new, so there is no older behaviour to
keep.

[!WARNING] Changed in wave S2
CAPABILITY_ENFORCEMENT=enforce with CAPABILITY_ENFORCEMENT_AREAS=admin
used to mean "admin enforced, everything else off". Unlisted areas now take
the base mode, so that pair enforces every area. Write
CAPABILITY_ENFORCEMENT_AREAS=admin=enforce instead, which leaves the rest
at report. Check your production vars before you deploy this API.

The order:

Report everywhere

CAPABILITY_ENFORCEMENT=report, no areas. Watch access.would-deny for
about two weeks and fix what it shows. The routes and capabilities that
would be refused, most frequent first:

$ bun run d1 --command "SELECT json_extract(detail, '$.area') AS area, json_extract(detail, '$.route') AS route, json_extract(detail, '$.needs') AS needs, json_extract(detail, '$.unlock') AS unlock, COUNT(*) AS hits FROM audit_log WHERE action = 'access.would-deny' GROUP BY area, route, needs, unlock ORDER BY hits DESC LIMIT 20"

Rows with unlock step-up under the admin area are admins still on
legacy tokens; rows with upgrade are plan limits working as designed.

Enforce resource

CAPABILITY_ENFORCEMENT_AREAS=resource=enforce (unset
CAPABILITY_ENFORCEMENT). Team and admin keep reporting.

Enforce team

CAPABILITY_ENFORCEMENT_AREAS=resource=enforce,team=enforce.

Turn on E3 and enforce admin

When every admin has signed in with a v2 session (prorm login) and
confirmed a TOTP factor (prorm security mfa enable):
CAPABILITY_E3=1 and
CAPABILITY_ENFORCEMENT_AREAS=resource=enforce,team=enforce,admin=enforce.

CAPABILITY_E3=1 also changes behaviour outside the admin area
(E3):

every /admin/* route (and /proxy/status) refuses an admin without a
confirmed TOTP factor, 403 mfa_required, with the enrolment routes;

POST /security/sessions/cli and the v1 sign-in relay answer
410 login_v2_required, except an explicit CI token
(purpose: "token", 30 days at most). Update CI that mints tokens through
that route to send purpose: "token";

GET /provision/databases and /stores default to scope=all.

[!WARNING] Do not enforce admin early
Setting CAPABILITY_E3=1 before every admin has a v2 session and a
confirmed TOTP factor refuses them: an admin without a factor gets
mfa_required on every staff route (they can still enrol, since
/security/mfa/* is their own), and one still on a legacy CLI token has no
admin:* capability. Customers are not affected.

Rollback. Set the previous values and redeploy; the change takes effect on
the next request. CAPABILITY_ENFORCEMENT=report is always safe.

9. The klock cron

The klock Worker has an hourly cron, triggers.crons: ["17 * * * *"] in
wrangler.klock.jsonc, and wrangler deploy registers it. Its scheduled
handler runs maintain (klock/lib/maintenance.ts), two independent passes
(one failing does not stop the other):

Pass	Removes
Retention (`prune`)	`klock_sync_log` rows older than 30 days; spent RPC nonces older than the 5-minute replay window; redeemed enrolment-ticket nonces a day after the ticket expired
Transfer sweep (`sweepTransfers`)	Expires transfers past their 24 hours, cancels those whose sender or recipient device was revoked or wiped, deletes ended transfers' R2 objects (nothing while `RELAY` is unbound), and prunes spent transfer nonces

Deletes are batched (5,000 rows a statement, at most 20 statements per table
a pass), so what one pass leaves the next one takes. The event chain, the
device registry and the credential ledger are never pruned. Each run logs a
line beginning klock: pruned N sync log rows; watch for it with
wrangler tail -c wrangler.klock.jsonc.

The API's own tables are pruned by the API:

klock_revocations (API D1) is kept for 8 days and pruned by the API's
cron and by the feed route.

Rollback. Remove the triggers block and redeploy. Nothing breaks
without it; the sync log and nonce tables just grow.

10. Credentials: the authority's CSR route

The TLS and Kerberos providers issue through the Kerberos authority
(api/kerberos, at KERBEROS_AUTHORITY_URL), which the API calls with
KERBEROS_AUTHORITY_TOKEN. Deploy the authority build from this release,
which adds two routes:

Route Does

POST [/{team}]/v1/csr Signs a device's CSR. Body { csr, subject, uris, spkiSha256, hours }. It checks the CSR's self-signature, that the key is P-256, the exact common name (device/<id> or session/<uuid>), the exact URI SANs and the attested key hash, and refuses any extension but subjectAltName and any extra subject field. hours is 1 to 48 (12 by default). 201 { cert, ca, chain, serial, fingerprint, subject, issuer, notBefore, notAfter }: no key is returned, because none is made. 422 { code } for csr_invalid, csr_key_type or csr_subject_mismatch.

POST [/{team}]/v1/principals/{session}/rotate A new random key for a principal (the kvno rises) and its keytab. 404 for a principal that does not exist: rotation never creates one.

Route	Does
`POST [/{team}]/v1/csr`	Signs a device's CSR. Body `{ csr, subject, uris, spkiSha256, hours }`. It checks the CSR's self-signature, that the key is P-256, the exact common name (`device/<id>` or `session/<uuid>`), the exact URI SANs and the attested key hash, and refuses any extension but subjectAltName and any extra subject field. `hours` is 1 to 48 (12 by default). `201 { cert, ca, chain, serial, fingerprint, subject, issuer, notBefore, notAfter }`: no key is returned, because none is made. `422 { code }` for `csr_invalid`, `csr_key_type` or `csr_subject_mismatch`.
`POST [/{team}]/v1/principals/{session}/rotate`	A new random key for a principal (the kvno rises) and its keytab. `404` for a principal that does not exist: rotation never creates one.

POST /v1/certificates is deprecated: it makes the client's key on the
server. It stays only for v1 passwordless --auth tls.

The optional KLOCK_AUTHORITY_TOKEN (authority environment) is a second,
separately revocable bearer that the CSR route admits beside
AUTHORITY_TOKEN. It cannot make principals, realms or server-keyed
certificates, nor check tickets. Leave it unset unless something other than
the control plane must reach /v1/csr.

The API's side needs nothing new beyond the klock service
(step 6, step 7).
Check it from a signed-in CLI:

$ prorm klock certs request
$ prorm klock certs list

The dev authority (authority -dev) returns bytes that are not a keytab, so
Kerberos issuance refuses them (authority_unavailable); try Kerberos
against a real KDC.

CA key custody. The CA key stays on the authority's volume. The
mitigations for its theft are the hours-long leaves, the fingerprint binding
at the API, and the attestation, which the API re-verifies against the
session's own device key, so a certificate for a key the device never vouched
for cannot be bound.

Rollback. Roll the authority back. /v2/credentials/* then answers
authority_unavailable; v1 --auth tls keeps working on /v1/certificates.

11. Team billing (Stripe)

Team billing (a team's billing account)
uses the API's existing Stripe settings; there is no team-specific key.

Setting	What	Without it
`STRIPE_SECRET_KEY`	Secret, in the Secrets Store	Checkout, portal, card and plan routes answer `503` (account and team alike)
`STRIPE_WEBHOOK_SECRET`	Secret, the webhook's signing secret	The webhook refuses events, so a team's plan never moves
`STRIPE_PRICE_USAGE`	The usage price id	Checkout answers `503` ("no billing cycle can be started")
`STRIPE_PRICE_MEMBERSHIP`	The membership price id	Membership checkout and `plan: membership` answer `503`
`STRIPE_FAKE`	Tests only. An in-process Stripe fake that logs `[stripe:fake] …`. It refuses to switch on beside a live key (`sk_live_`/`rk_live_`).	Never set it in production

STRIPE_SECRET_KEY and STRIPE_WEBHOOK_SECRET are not bound in
wrangler.jsonc today; add them to secrets_store_secrets once the store
holds them. The webhook endpoint (POST /billing/webhook) must receive:

checkout.session.completed (including setup mode, for new cards);

customer.subscription.created, .updated and .deleted;

invoice.created, invoice.paid and invoice.payment_failed.

A team's Stripe customer is created on the team's first checkout, never from
a member's. Team-owned transfers need a card on the team's customer (402
without one), and team-owned pools count against the team's plan.

Rotating keys

Planned: publish a keyset with the old and new keys (the new one with
nbf now), switch signing to the new key after 24 hours, and drop the old
key after 7 days (the longest grant). With the ceremony tool:

$ bun run scripts/klock-keyset.ts keygen --use grant --out grant-2.key
$ bun run scripts/klock-keyset.ts rotate --current keyset-1.jws --root-key root-2026.key --add grant-2.key.pub.json --retire g-202609-3fa1 --out keyset-2.jws
$ bun run scripts/klock-keyset.ts verify --keyset keyset-2.jws --min-seq 2

rotate verifies the current keyset with the root first (an expired one
can still be rotated from), then raises seq by one, extends the kept keys
to the new expiry, starts --add keys now, and ends a --retired key after
a grace (7 days by default; --retire g-202609-3fa1=3 for three). Keys
that have already expired are dropped, and it warns when no grant or klock
key stays current. Publish the result as in
step 5, bind the new seed as KLOCK_GRANT_KEY
after 24 hours, and the old key leaves the keyset's validity when its grace
ends.

The last grant signed with the old key is minted just before the switch and
lives at most 7 days, so the old key can leave the keyset once the switch is
a week old.

Emergency: publish a keyset without the compromised kid
(rotate … --drop <kid>, which removes it at once). Clients pick it up on
their next sync, and the klock Worker within its one-hour keyset cache.

Root: a new root means a CLI and outh release that pins it. Keep a
second root pinned so a lost primary is not an emergency.

klock: v2 files, grants, sessions and the klock service

Capabilities: tiers, roles and enforcement modes

Access rules and behaviour

MCP server: delegated tokens and the forward

Credential rotation

Security in the console: the team billing tab and the admin MFA prompt

Deploying klock v2

The order, and why#

Where does it stand?#

1. The root key ceremony#

Make the roots#

Make the operational keys#

Sign keyset seq 1#

Verify it#

2. Keys and secrets#

3. Publish the ORM prerelease#

4. API migrations 0026 to 0030#

5. Publish the keyset#

6. The klock Worker#

Create its D1 database#

Apply its migrations#

Bind its keys#

The relay bucket and the support drop#

Deploy#

7. Bind the klock service to the API#

8. The enforcement rollout#

Report everywhere#

Enforce resource#

Enforce team#

Turn on E3 and enforce admin#

9. The klock cron#

10. Credentials: the authority's CSR route#

11. Team billing (Stripe)#

Rotating keys#

Related reading#