Restoring your database backups — the runbook

A backup you've never restored is hope, not disaster recovery. This page is the verified restore runbook for RailsFast's default backup setup (configured here) — every command below was proven in a live restore drill: seed real data, take a backup, destroy the data, restore it, and prove it back through the app (Devise passwords authenticating, new writes succeeding), not just with row counts.

What you're restoring

RailsFast runs four databases: myapp_production plus _queue, _cache, and _cable. Only myapp_production holds your business data. The other three belong to Solid Queue/Cache/Cable and should be rebuilt, not restored — restoring the queue resurrects stale jobs. After restoring the primary, just run bin/kamal app exec --reuse 'bin/rails db:prepare' and the ephemeral three recreate themselves in seconds.

Backups land in S3 as s3://<bucket>/<app>/<YEAR>/<Month>/PG_<database>.<DD-Month-YYYY-HH-MM>.dmp.gz, one file per database, timestamped one minute apart, plus one globals.sql at the app root (roles — only needed if you lost the whole postgres accessory).

Step 1 — Find the dump

aws s3 ls s3://YOUR-BUCKET/YOUR-APP/ --recursive --human-readable

Pick the file whose name contains the database you want — the one named exactly ..._production is your business data.

Step 2 — Restore into a scratch database first

Never point your first restore at production. Save this locally as restore.sh — it runs inside the pg-backup container, which already has the S3 credentials, network access, and client tools:

set -uo pipefail
KEY="${1:?s3 key relative to $BUCKET}"; DB="${2:?target database}"
WORK=/data/manual-restore; mkdir -p "$WORK"; rm -f "$WORK/dump" "$WORK/dump.gz"

cat > /root/.s3cfg <<EOF
[default]
host_base = ${HOST_BASE}
host_bucket = ${HOST_BUCKET}
bucket_location = ${DEFAULT_REGION}
use_https = ${SSL_SECURE}
access_key = ${ACCESS_KEY_ID}
secret_key = ${SECRET_ACCESS_KEY}
signature_v2 = False
EOF

s3cmd get --force "s3://${BUCKET}/${KEY}" "$WORK/dump.gz" || { echo "DOWNLOAD FAILED"; exit 1; }
gunzip -f "$WORK/dump.gz"

export PGPASSWORD="${POSTGRES_PASS}"
PG="-h ${POSTGRES_HOST} -p ${POSTGRES_PORT} -U ${POSTGRES_USER}"

dropdb   ${PG} --if-exists --force "${DB}"
createdb ${PG} -O "${POSTGRES_USER}" "${DB}"
pg_restore ${PG} -d "${DB}" -j 4 "$WORK/dump" 2> "$WORK/err.log"
echo "pg_restore exit=$?"
sort -u "$WORK/err.log" | head
psql ${PG} -d "${DB}" -At -c "SELECT 'users='||count(*) FROM users"

Run it against a throwaway database name (note: kamal accessory exec can't pass stdin/env like this, so go over plain ssh + docker exec):

ssh -i ~/.ssh/YOUR_KEY docker@YOUR_SERVER_IP \
  'docker exec -i YOURAPP-pg-backup bash -s -- "2026/July/PG_yourapp_production.28-July-2026-02-00.dmp.gz" "yourapp_restore_check"' \
  < restore.sh

Confirm the row counts look right before going anywhere near production.

Step 3 — Cut over to production

Same command, real database name:

ssh -i ~/.ssh/YOUR_KEY docker@YOUR_SERVER_IP \
  'docker exec -i YOURAPP-pg-backup bash -s -- "2026/July/PG_yourapp_production.28-July-2026-02-00.dmp.gz" "yourapp_production"' \
  < restore.sh

You don't have to stop the app: dropdb --force terminates its connections and Rails reconnects on its own (verified — the site served 200s immediately after). Requests arriving mid-restore will error, so stop the app first if you want a clean window, but it's not required for correctness.

Step 4 — Prove it through the app

Row counts prove presence; the app proves usability:

bin/kamal app exec --reuse 'bin/rails runner "puts User.count"'
# and for real confidence, check something cryptographic survived:
bin/kamal app exec --reuse 'bin/rails runner "puts User.first&.email.inspect"'

Step 5 — Rebuild the ephemeral databases

bin/kamal app exec --reuse 'bin/rails db:prepare'

Done. Fresh queue/cache/cable, restored business data, live app.

Practice this before you need it

The whole drill — scratch restore and all — takes about fifteen minutes against a live app, and it's the only way to know your backups are real. Do it once after configuring backups (this replaces "list the bucket" as the meaningful verification), and again after anything that changes the backup pipeline. If you lost the entire postgres accessory (volume and all), apply globals.sql first to recreate the role, then restore per database.