Operations

Tokens and secrets

Every credential in morca has one job and a short reach. Postgres holds only hashes of tokens and the names of secrets, a worker never holds database credentials, and a pipeline's spec names its secrets without carrying them.

Overview

The server keeps only the SHA-256 of a session, an API token or a worker token, so a copy of the database doesn't let anyone in. The orca_ prefix lets a secret scanner recognise a token that leaked.

UI logins

Anyone who can reach the UI can read it. Its actions (start, backfill, resume, rerun, cancel, rebuild) need a logged-in user as soon as any user exists, or with ORCA_UI_LOGIN=required before one does, so a stack with users can't be left open by forgetting the variable.

orca user add alice          # reads the password from standard input
orca user tokens alice       # her API tokens
orca user revoke alice 3     # or @all
orca user remove alice       # and ends her sessions

• Passwords are bcrypt hashes made by Postgres's pgcrypto, so neither morca nor its logs see one after it is checked.
• Sessions are an HttpOnly, SameSite=Strict cookie, Secure with ORCA_UI_SECURE_COOKIES=1 behind HTTPS.
• Failed logins are audited and counted: after 5 for one name, or 20 from one address, in 15 minutes, /login and /api/login answer 429 without checking the password. Behind your own proxy, ORCA_UI_TRUST_PROXY=1 takes the address from X-Forwarded-For.
• API tokens. orca login <url> gets a token for the orca command on another machine, kept there mode 600, and sent as Authorization: Bearer. Changing a user's password or removing the user revokes all of her tokens.

Every action a user takes is a row of orca.audit (The UI).

Worker tokens

orca worker-token create --pool bgent --label bgent   # prints the token, once
orca worker-token list
orca worker-token revoke 3

A worker token is scoped to one pool. It can claim that pool's runs and make the state calls of the runs it claimed, and nothing else; a user's token gets nothing from the worker API. There is no read-only or read-write variant. A revoked token stays listed as revoked, and the worker using it is refused and exits. Each claim is audited as worker:<label>.

A run on a worker gets the worker's token as ORCA_STATE_TOKEN, with ORCA_STATE_URL, in place of a database. A worker with ORCA_DATABASE_URL set refuses to start, and it needs only outbound HTTPS (--remote takes http only to its own machine), so Postgres stays reachable only on the server (Remote workers).

Boot tokens of autoscaled machines

A machine of a pool morca grows gets a worker token of its own, made for it and revoked when it is removed. Its startup script holds that token and no other secret. With it, the machine asks the UI's /api/worker/boot over TLS for the image, the registry login to pull it, and the bucket's variables. /api/worker/boot answers only a token the autoscaler made for a machine that is live: one from orca worker-token create gets a 403.

The bucket's variables come from the UI's own environment, so on Hetzner a pool machine gets the same S3 keys as the server. On EC2, give the instances a profile whose role reaches the bucket (ORCA_POOL_EC2_PROFILE) instead. The machine keeps them in a file only root can read, and deletes the registry login's file as soon as it has read it.

Cluster tokens

From 0.24.0, a run spread over several machines of a pool (workers = N) gets a token for each start. The worker that starts the run makes it and gives it, as DISTA_TOKEN, to the run's dista head, its driver and its own dista worker, and reports it to the UI. A member gets it only in the answer to its claim, over the worker API's TLS. It is never in a log or a page.

dista checks it on every connection to a head or a worker's object server: a missing or wrong token closes the connection unanswered, and the comparison takes the same time wherever the bytes differ. So nothing else on the network can join the run, submit to it or fetch its results. It authenticates and doesn't encrypt, so keep the pool's machines on a private network, as an autoscaled Hetzner pool does (dista's DESIGN.md).

Pipeline secrets

[pipelines.gharchive]
secrets = { GITHUB_TOKEN = "orca/github-token" }

A pipeline's secrets map an environment variable to a secret's ID, which starts with orca/. Postgres and the run's spec hold only the IDs, and a run's page shows its secrets by name only. orca.toml refuses an env name that looks like a secret (*TOKEN*, *SECRET*, *PASSWORD*, *DATABASE_URL*), so values don't end up there.

• On EC2, each instance reads them from Secrets Manager at start with its role, which may read only orca/.
• With ORCA_SECRET_FETCH, the scheduler runs that command with the ID and uses what it prints. On the Hetzner server it is sh /etc/orca/secret-fetch, which prints /etc/orca/secrets/<id> (mode 600, root only) and must refuse ...
• On a remote worker, the worker's own ORCA_SECRET_FETCH reads them there. The spec it claims names the secrets and never carries their values.

Each run gets ORCA_SECRETS, the names of its secrets, and morca masks their values in every log it writes.

Secrets on deploy

On the Hetzner stack, hetzner.env holds no secret itself: it names the secret files. A file named *.sops or *.sops.<ext> is decrypted with sops -d on its way to the server and never written to a file on the laptop, so the laptop keeps one age key instead of plain tokens. SECRETS_FILE names one SOPS file with every secret, decrypted once per deploy, so a Secure Enclave key asks for Touch ID once. The server gets plain files under /etc/orca, mode 600. Every deploy brings them in line and prints the names of what changed, never a value (Settings and secrets, hetzner.md).

Cloud credentials

The Hetzner token is sent only as a bearer token and never put in an error, and the registry login is never in one either. The firewall each Hetzner pool machine gets lets in ssh from ADMIN_IP only (hetzner.md).