Deployment

This page is for whoever runs the Custodian server. It covers the configuration the server reads from its environment at boot, and the route you use to check it is alive.

Running the server

Custodian runs as a standard long-running web service. Deploy and start it the way your hosting platform starts any web application. Once its environment is set (below) and its data store is reachable, it listens for HTTP and serves /v1, /mcp and /healthz; point your platform's health check at /healthz so an instance that has lost its data store is taken out of rotation.

The database connection

The server keeps every vault's data in a database and needs a connection to it. Set the standard database connection-string variable your hosting platform provides; the server reads its connection from there. Without a reachable database the server starts but reports itself unhealthy — see Liveness below.

Environment variables

Variable	Sets
`CUSTODIAN_BOOTSTRAP_CREDENTIAL`	The credential that authorises `POST /v1/vaults`. A deployment that configures none refuses every vault bootstrap.
`CUSTODIAN_SINGLE_ROLE`	Set to a truthy value (`1`/`true`/`yes`/`on`) on a deployment where the server cannot create a second database role for itself; leave it unset — the default — everywhere it can.
`CUSTODIAN_ENROLMENT_APPROVAL_MAX_AGE_SECONDS`	How long a signed device-enrolment approval stays valid, in seconds. Defaults to `300` (five minutes).
`CUSTODIAN_SERVER_ENVELOPE_CACHE_SIZE`	How many vaults' unwrapped data keys the server holds in memory at once. Defaults to `1024`.
`CUSTODIAN_KMS_PROVIDER`	Which key-management service protects stored values. Set it to `aws` to use AWS Key Management Service; it defaults to a built-in stub for non-production use, and to `aws` in production.
`CUSTODIAN_RECOVERY_RATE_LIMIT_MAX`	How many recovery-envelope fetches a vault may make within the window before `GET /v1/recovery` is refused with `recovery_rate_limited`. The count is taken from the audit rows every fetch attempt writes. Defaults to `5`.
`CUSTODIAN_RECOVERY_RATE_LIMIT_WINDOW_SECONDS`	The window, in seconds, over which `CUSTODIAN_RECOVERY_RATE_LIMIT_MAX` is counted. Defaults to `60`.
`CUSTODIAN_FILE_MAX_BYTES`	The largest a single file write may store, in bytes. A write that declares or sends more is refused with `413`. Defaults to `10485760` (10 MiB).
`CUSTODIAN_FILE_OBJECT_LOCAL_ROOT`	The directory the server keeps a file field's ciphertext in when it stores files on local disk rather than in a bucket. A production deployment stores in a bucket and does not read this.
`CUSTODIAN_FILE_OBJECT_SWEEP_GRACE_SECONDS`	How old an unreferenced file object must be before `rake file_objects:sweep` removes it, in seconds. A write stores the object just before its row, so the grace period stops the sweep taking the bytes of a write still in flight. Defaults to `3600` (one hour).

Maintenance tasks

Removing orphaned file objects

A file field's ciphertext is stored as an object before the row that names it, and replacing a file writes a new object rather than overwriting the old one. An object that no live value names is therefore either a write that failed between those two steps or a file that was replaced — safe to remove once it is old enough that it cannot be a write still in flight.

rake file_objects:sweep removes those orphaned objects. It skips any object younger than CUSTODIAN_FILE_OBJECT_SWEEP_GRACE_SECONDS, and prints what it did — the number of objects removed and their total bytes, how many recent objects it left, and one line per removed object:

$ rake file_objects:sweep
file_objects:sweep removed 2 orphaned objects (37888 bytes); left 1 unreferenced object younger than the grace period.
  removed 42/MFRGGZDFMZTWQ2LKNNWG23TPOBYXE43U (20480 bytes)
  removed 42/NB2HI4DTHIXS653XO4XHIZLTOQXHG5DP (17408 bytes)

Run it on a schedule, the way you run your deployment's other periodic work. It is safe to run at any time: it never removes an object a live value names.

Liveness

GET /healthz reports whether the server and its data store are alive. It takes no credential — it is the one route outside /v1 — and it reports healthy only while the data store answers a query, not merely that the process started.

GET /healthz

When healthy, 200 OK:

{ "status": "ok", "database": "connected" }

When the data store does not answer, 503 Service Unavailable:

{ "status": "error", "database": "unreachable", "error": "..." }

Point your platform's health check at /healthz, so an instance that has lost its data store is taken out of rotation rather than left serving errors.