Kamal - Deploying from clean checkouts and worktrees

Deploying "from a clean checkout" (a fresh clone, a git worktree, or a CI runner) is good release hygiene: the image can't accidentally include your local uncommitted changes. But there's a trap in how Kamal resolves secrets that will cost you a failed deploy the first time you try it.

The trap: gitignored key files resolve silently EMPTY

Your .kamal/secrets file typically sources secrets from local files:

RAILS_MASTER_KEY=$(cat config/master.key)
DB_PRIMARY_PASSWORD=$(cat config/db.key)

Both files are gitignored, so a pristine checkout doesn't have them. And here's the nasty part: the $(cat …) substitution does not fail the deploy when the file is missing. It resolves to an empty string, the image builds fine, the push succeeds, and the container ships with empty credentials.

You then get failures that look nothing like "missing key file":

• An empty RAILS_MASTER_KEY means the app can't decrypt credentials.yml.enc, so the boot dies at whatever reads credentials first: often a custom guard or an initializer, with an error message about that feature rather than about the key.
• An empty database password dies later still, as PG::ConnectionBad: fe_sendauth: no password supplied, during db:prepare, after credentials already loaded fine.

Each failure is downstream of the real cause, so you can burn a full build-push-boot cycle (minutes) per missing file if you debug them one at a time.

The 5-second preflight that catches all of it

Before deploying from ANY checkout that isn't your usual working directory:

kamal secrets print

Eyeball the output for empty values. Every secret should have one. If anything is blank, fix the sourcing before you deploy. You just saved yourself a failed cycle per empty line.

The pre-flight that runs by itself: bin/check-deploy-secrets

RailsFast apps (template v1.2.0 and later) ship this check as a script, and Kamal runs it for you. It reads the same secrets files that Kamal reads (.kamal/secrets-common, then .kamal/secrets or .kamal/secrets.<destination>), with Kamal's own parser, but it does not run their commands. For each secret that your config/deploy.yml uses, it finds the files that its $(cat …) or < file commands read. Then it sorts what it finds in two groups:

• Errors: a file that is missing or empty. Either way the secret would reach the containers as an empty string. Errors stop a deploy.
• Warnings: a file that still holds a template placeholder. bin/setup writes YOUR_… and your-… values into config/backups/*.key. They look filled in, but the backups cannot upload with them. Placeholders never stop a deploy, so a new app's first deploy still goes through before its backups are set up.

bin/check-deploy-secrets                              # production (.kamal/secrets)
bin/check-deploy-secrets -d staging                   # the staging destination (.kamal/secrets.staging)
bin/check-deploy-secrets -c config/deploy.worker.yml  # another config file

The deploy secrets are not ready (.kamal/secrets):
  config/master.key (RAILS_MASTER_KEY) is missing
Kamal sends these template placeholders as they are (.kamal/secrets):
  config/backups/s3_bucket.key (BACKUP_MONITOR_BUCKET, BUCKET) still holds a template placeholder

Each line names the file and the secrets that read it. It prints names, never values. A secret built from another one (POSTGRES_PASSWORD=$DB_PRIMARY_PASSWORD) carries the other secret's file. Only a literal fallback makes a file optional: one command, then ||, then echo, printf, true or : with no file of its own. For example $(cat config/backups/monitor_key_id.key 2>/dev/null || echo ""). Such a file may be absent. It may be empty only when the fallback prints nothing either, because cat succeeds on an empty file and the fallback never runs. A fallback that reads another file (|| cat other.key), or a list before it (a ; b || echo), makes every file required.

The pre-connect hook

.kamal/hooks/pre-connect runs bin/check-deploy-secrets --hook before Kamal connects to a host. As a hook it checks only the secrets that the current command sends, for the destination Kamal is using. It also sees every secret as Kamal evaluated it, so it stops on a used secret that came out empty from a source it cannot follow, such as an ENV variable or a password manager.

Placeholders only print a warning, whatever the command. To skip the check once, add --skip-hooks (-H) to the Kamal command. This also skips your other hooks. The hook cannot see --reuse: an app exec without it starts a new container with the same wrong secrets, so read the warning.

The fix

Either copy the gitignored key files into the clean checkout before deploying:

cp /path/to/working-copy/config/master.key config/master.key
cp /path/to/working-copy/config/db.key     config/db.key

…or, better long-term, source those secrets from a password manager (1Password's op, etc.) in .kamal/secrets, so any checkout on an authenticated machine resolves them identically. The commented examples at the top of the generated .kamal/secrets show the pattern.

Related failure modes worth knowing

• Registry 403 on push (unexpected status from HEAD request … 403 Forbidden): registry auth tokens expire (AWS ECR's last 12 hours). Run kamal registry login (or for ECR directly, aws ecr get-login-password | docker login --username AWS --password-stdin <registry>) and retry. The build cache survives, so the retry is fast.
• A failed deploy is traffic-safe, but read the right logs. When the new container fails its health check, Kamal stops it and the previous container keeps serving, so users see nothing. But Kamal's own output only says target failed to become healthy within configured timeout. The actual error is in the stopped container's logs on the host:

ssh <deploy-host> "docker logs <service>-web-<git-sha> 2>&1 | tail -50"

That's where the real exception lives. Read it before changing anything.