custodian file

custodian file stores a file in a record's file field and gets it back. A file field is a field the schema declares type: file. It is always client_encrypted: the server holds bytes it cannot read. The file is sealed on an enrolled device before it leaves, and opened on one.

What protects the file on each machine depends on the store that holds that machine's device key. Each store's guarantee, and what it does not defend against, is quoted in the store that holds the key. An unattended machine's device is a key in AWS KMS, and Custodian prints this line for it: AWS KMS under a task role — the device key is an AWS KMS key the container's task role may use; unwrapping happens inside KMS and the key itself never enters this tool's memory. The downgrade, stated plainly: AWS can unwrap this copy, so this device is protected by a cloud provider's access control rather than by hardware anyone holds — materially weaker than a hardware module, and the price of running unattended — not yet exercised on real hardware — validate on a machine with this store before relying on it. See headless devices.

custodian file put seals a file on this machine and stores it. It prints the file's token and a one-line command, the retrieval line, that you can send to the person who needs the file. custodian file get runs that line on an enrolled machine, opens the file there and writes it to disk.

A file's token is tok_ followed by sixteen characters, for example tok_AOWVT36VNVCIESCL. It is the identifier Custodian gives the stored file, in the same form as the token of any sensitive value. It is derived from a random value, not from the file's bytes, so two identical files do not share a token. It is not a sign-in token. On its own it does not open the file: the person running the retrieval line must be signed in with a grant to read the field, on an enrolled device in the vault.

Before you start

• Be signed in. For get, your credential needs a read-the-plaintext grant (--plaintext) on the file field, and on its filename field if the schema names one. For put, it needs a --write grant on the collection and the same --plaintext grant: put reads the record before it stores anything, and a write grant does not permit reading. Without a read grant, put reports the record as not found.
• This machine must be an enrolled device in the vault. The command unwraps the vault's master key through it, so on a machine with a presence prompt (Touch ID or a passcode on macOS) you are asked to approve.
• The schema must declare the field type: file. See the schema file.
• For put, the record must already exist. Create it with custodian records create first.

custodian file put

custodian file put <record> --collection <name> --field <name> --input <path>

<record> is the record identifier, the same UUID custodian records get takes.

Options:

• --collection <name> — the collection the record lives in (required).
• --field <name> — the field the schema declares a file (required).
• --input <path> — the file to seal and store (required).
• --url <api> — select the deployment.
• --vault <slug> — select the vault.
• --json — emit machine-readable JSON. It never contains a byte of the file.

custodian file put checks that the schema declares the field a file and that the record exists in the collection before it reads the file. It then reads the file, seals it on this machine and sends only the sealed bytes to the server.

On success it prints the field and record, the token, and the retrieval line:

Stored in field "passport_scan" on record 018f4e2a-7c1b-7a3e-b9d4-2f6c8a1e5b70.
  tok_AOWVT36VNVCIESCL

Retrieve it on an enrolled device with:
  custodian file get tok_AOWVT36VNVCIESCL

If the field already held a file, the new file replaces it, and the first line says so:

Replaced the file in field "passport_scan" on record 018f4e2a-7c1b-7a3e-b9d4-2f6c8a1e5b70. The previous retrieval line has stopped working.

Each stored file gets a new token. Send the new retrieval line to anyone who needs the new file.

With --json, it prints one object:

{"stored":true,"replaced":false,"token":"tok_AOWVT36VNVCIESCL","record":"018f4e2a-7c1b-7a3e-b9d4-2f6c8a1e5b70","field":"passport_scan"}

When put is refused

The field is not in the applied schema:

error: The applied schema declares no field "passport_scan" in collection "policyholders", so it cannot hold a file. Apply the schema with `custodian schema apply` or correct the field name.

The field is not a file field:

error: Field "policy_number" in collection "policyholders" is declared "string", not a file, so a file cannot be stored in it. Only a field the schema declares a file carries a file.

The record is not in that collection:

error: No record "018f4e2a-7c1b-7a3e-b9d4-2f6c8a1e5b70" was found in collection "policyholders".

The input file cannot be read. The message ends with the reason the system gave:

error: Cannot read the input file "scan.pdf": <reason>.

This machine has no enrolled device:

error: Cannot seal the file: this machine holds no enrolled device to unwrap the master key, and a file is sealed on this device or nowhere. Enrol one with `custodian device enrol`, then re-run.

Anything the server refuses is shown as the server worded it. See record files for those refusals. The one you are most likely to meet is a file over the deployment's size limit. When the file is too large, the server's message is followed by the file's size on this machine:

error: <the server's refusal> (The file "scan.pdf" is <n> bytes on this device.)

Rarely, the server refuses a write because the storage location it was given for the file is already in use. The file already stored there is left untouched. Each run of put picks a new location, so run the command again.

custodian file get

custodian file get <token>

The token is the only argument. Custodian looks up the collection, record and field from it, fetches the sealed file, opens it on this machine, and writes it.

Options:

• --output <path> — where to write the file: a file path, or a directory to write it into. Without it, the file is written into ~/Custodian/<vault>/.
• --url <api> — select the deployment.
• --vault <slug> — select the vault.
• --json — emit machine-readable JSON. It never contains a byte of the file.
• The global --yes answers the overwrite question below.

On success it prints the name, the size and the full path:

Wrote passport.pdf (412 KB) to /home/alex/Custodian/acme/passport.pdf

Where the file is written

Without --output, the file goes into ~/Custodian/<vault>/ in your home directory. With --output, a path that ends in / or names an existing directory is treated as a directory, and the file is written inside it. Any other path is used as the file's full path.

When Custodian writes into a directory — the default one or one you named — it creates the directory if it is missing, with permissions 0700 (only you can open it). It refuses a directory that already exists but is not yours, or that other users can open, and writes nothing:

error: The destination directory /home/alex/Custodian/acme is owned by uid 0, not by you (uid 1000); refusing to write a decrypted file there. Nothing was written.

error: The destination directory /home/alex/Custodian/acme has permissions 0755, which admit users other than you; refusing to write a decrypted file there. It must admit only its owner (0700).

To fix the second one, run chmod 700 on the directory, then run the command again.

The file itself is written with permissions 0600. It is first written to a temporary file next to the destination and then renamed into place, so a failed write leaves no partial file behind.

If a file of that name already exists

Custodian asks before it overwrites:

A file already exists at /home/alex/Custodian/acme/passport.pdf. Overwrite it? [y/N]

Any answer other than y or yes leaves the existing file alone:

Left the existing file at /home/alex/Custodian/acme/passport.pdf in place; nothing was written.

The global --yes answers yes without asking. Without --yes, when standard input is not a terminal, Custodian cannot ask, so it refuses:

error: Refusing to continue without confirmation: standard input is not a terminal, so there is nobody to answer "A file already exists at /home/alex/Custodian/acme/passport.pdf. Overwrite it?". Pass --yes to answer yes non-interactively.

How the file is named

A file field can name a sibling field that holds the file's name, with filename_field. When it does, and the record holds a name there that you are allowed to read, the file is written under that name. Only the last part of the name is used, so a name cannot place the file outside the destination directory.

custodian file put does not set that name. The filename field is an ordinary string field of the record, and you set it yourself with custodian records create or custodian records update. If the field is client_encrypted, read the value from standard input rather than putting it on the command line:

printf 'passport.pdf' | custodian records update 018f4e2a-7c1b-7a3e-b9d4-2f6c8a1e5b70 --collection policyholders --set passport_scan_name=@-

Otherwise the file is written under its token, and a note under the result says why. The three notes are:

  The schema names no field for this file's name, so it was written under its token.

  You are not granted the field that holds this file's name, so it was written under its token.

  The record holds no name for this file, so it was written under its token.

Machine-readable output

With --json, it prints one object:

{"written":true,"path":"/home/alex/Custodian/acme/passport.pdf","bytes":421888,"name":"passport.pdf","name_source":"filename-field","overwrote":false}

written is false when you declined to overwrite an existing file. overwrote is true when an existing file was replaced. name_source says where the name came from:

When get is refused

The argument is not a token:

error: "passport.pdf" is not a Custodian token. A token is `tok_` followed by sixteen base32 characters, and is the only argument `custodian file get` takes.

The token belongs to an ordinary field, not a file:

error: Token tok_AOWVT36VNVCIESCL names a field that is not a file — it has an ordinary value, not a file's bytes. Read it with `custodian resolve` (the command that does read it), not `custodian file get`.

The token does not resolve to a file you can read. The server answers an unknown token and one you have no grant on in the same way, so the message names all three causes:

error: Token tok_AOWVT36VNVCIESCL resolves to no file you can read: it does not exist, it belongs to another vault, or you hold no read_plaintext grant on its field. Nothing was written.

The file is no longer stored, or its record was deleted:

error: The file this token named is no longer stored, or its record was removed. Nothing was written.

This machine has no enrolled device:

error: Cannot open the file: this machine holds no enrolled device to unwrap the master key, and a file is opened on this device or nowhere. Enrol one with `custodian device enrol`, then re-run.

You declined the presence prompt:

error: Cannot open the file: the presence prompt was declined, so this device did not unwrap the master key. Re-run and approve the prompt (Touch ID, a passcode, or your device PIN) to open it.

Nothing is written in any of these cases.

Retrieval lines

A retrieval line is custodian file get followed by the file's token. It holds the token, not the file. Running it needs everything listed under Before you start: a signed-in credential with a grant to read the field, and an enrolled device in the vault.

A retrieval line stops working when:

• the file is replaced — custodian file put, or a capture with custodian exec --capture-output, stores a new file in the same field. The new file has a new token.
• the vault's master key is rotated — custodian key rotate recomputes every file's token.

In both cases the old line can never work again, so send a new one, built from the file's new token:

• After a replacement, the command that stored the new file printed the new token and retrieval line.

• After a rotation, nothing prints the new tokens. Read the record with custodian records get; the file field's line shows its current token, which needs a read grant on the field:

  custodian records get 018f4e2a-7c1b-7a3e-b9d4-2f6c8a1e5b70 --collection policyholders

  passport_scan       tok_K3QZ7W2M5VJXRT4N  tokenised
  passport_scan_name  tok_P6HCY2DQ5LMNWB3F  tokenised

  The new retrieval line is custodian file get tok_K3QZ7W2M5VJXRT4N. You do not need the original file.

For the whole procedure from one machine to another, see sending a file.