Operational runbooks

What to do when an application running on prorm misbehaves in production.

These are written for the person on call for your application — not for
operating any particular cloud. They assume you have a database somewhere, a
pool in front of it, and an incident in progress.

Start with the incident checklist if you do not yet know which one you need.

Incident checklist — Something is broken and you need to classify it fast.

Connection failures — Queries are failing to reach the database at all.

Failover — The database failed over, or you are about to make it.

Credential rotation — A password must change without dropping traffic, or one has leaked.

Migration rollback — A migration made it to production and should not have.

Deploying klock v2 — For prod-orm operators, not application on-call: the root key ceremony, keys, the klock service, migrations and capability enforcement.

What prorm does and does not do for you

It retries a narrow, well-defined set of failures — timeouts, lock
contention, and connections that timed out or could not reach the host — and
nothing else. A constraint violation is never retried, because retrying it
would fail identically.

[!IMPORTANT]
It does not fail over, re-resolve DNS on its own schedule, or hold
traffic while a database restarts. Those are the pool's and the driver's
job, and the runbooks above say what to configure so they do it well.

Error handling — the error classes in full

Connection pooling — sizing and lifecycle

Replication — splitting reads from writes

Audit levels and escalation — the policy that pages someone, and prorm logs escalations

The security toolkit — prorm audit db and prorm audit code when an incident might be a breach

Operational runbooks

What prorm does and does not do for you#

Related reading#

What prorm does and does not do for you

Related reading