Kamal - A second origin with its own container, port and hostname

Sometimes one part of your app must not share the app's origin. Examples: an internal API that other services call, an approval or payment page that must not share cookies or a Content Security Policy with the main app, or a small Rack service with its own secrets. Kamal can run it as an extra role of the same image: its own container, its own command and port, its own hostname with a TLS certificate from kamal-proxy, and its own secrets. You need no second repository, no second image and no second server.

RailsFast apps (template v1.2.0 and later) have this recipe as a commented internal role in config/deploy.yml. This page explains each step.

1. Write the service

The role runs the same image with another command. For a small Rack service, commit a rackup file, for example config/internal.ru. It must answer GET /up with 200 and without a database call, because kamal-proxy calls it to decide that a new container is healthy:

# config/internal.ru
run lambda { |env|
  case env["PATH_INFO"]
  when "/up" then [200, { "content-type" => "text/plain" }, ["ok"]]
  else [404, { "content-type" => "text/plain" }, ["not found"]]
  end
}

A real service checks its callers too: a shared key from its own secret, mutual TLS, or an allow-list.

2. Point the hostname at the server

Add a DNS record for the new hostname, the same way as for the app's own hostname. With Cloudflare:

cf dns records create -z example.com --body '{"type":"A","name":"internal.example.com","content":"1.2.3.4","proxied":true,"ttl":1}'

kamal-proxy gets a Let's Encrypt certificate for the hostname at the first deploy. Behind Cloudflare's proxy, use the same SSL mode as for the app (Full (strict)). Until the first deploy, Cloudflare shows error 521 or 526 for the new hostname. This is expected.

3. Give it its own secret

umask 077
openssl rand -hex 32 > config/internal.key   # config/*.key is in .gitignore and .dockerignore

Then add one line to .kamal/secrets:

INTERNAL_SERVICE_KEY=$(cat config/internal.key)

4. The role in config/deploy.yml

servers:
  web:
    - 1.2.3.4
  internal:
    hosts:
      - 1.2.3.4
    cmd: bundle exec rackup config/internal.ru --host 0.0.0.0 --port 3001
    options:
      memory: 384m
    proxy:
      host: internal.example.com
      app_port: 3001
      healthcheck:
        path: /up
    env:
      secret:
        - INTERNAL_SERVICE_KEY
      clear:
        RACK_ENV: production
        SOLID_QUEUE_IN_PUMA: false

RACK_ENV: production matters. The image sets RAILS_ENV but not RACK_ENV, and rackup without it runs in development mode, where Rack::ShowExceptions answers any error with the stack trace and the request headers, on a public hostname.

Then bin/kamal deploy, and check:

curl -sS -o /dev/null -w '%{http_code}\n' https://internal.example.com/up   # 200

The three Kamal rules that bite

1. The root env reaches every role. Kamal merges the root env (both clear and secret) into each role's env. So with the template's default layout, the internal role also gets RAILS_MASTER_KEY, the database password and the other root secrets. If the service must not hold them, move the root env.secret list into the web role (and the job role), and leave only plain, shared values at the root:

servers:
  web:
    hosts:
      - 1.2.3.4
    env:
      secret:
        - RAILS_MASTER_KEY
        - POSTGRES_PASSWORD
        # ... the rest of the old root list

Check the result with Kamal itself. bin/kamal config lists the roles but not their secrets, so ask the configuration directly:

bundle exec ruby -rkamal -e 'c = Kamal::Configuration.create_from(config_file: Pathname("config/deploy.yml")); c.roles.each { |r| puts "#{r.name}: #{r.env(r.hosts.first).secret_keys.join(", ")}" }'

2. The role's proxy merges over the root proxy, and must use the same host key. The role inherits ssl: true and forward_headers from the root, and its own host replaces the root's. If the root uses host: and the role uses hosts: (or the other way round), the merged proxy has both keys, and Kamal refuses it with "Specify one of 'host' or 'hosts', not both".

3. rackup starts Puma, and Puma reads config/puma.rb. That file starts the Solid Queue supervisor when SOLID_QUEUE_IN_PUMA is on, and the role inherits that variable from the root env. Set it to false on the role, as above. Since template v1.2.0, config/puma.rb reads false, 0, off, no, f, n, or an unset or empty value as off, and any other value as on. Older templates started the plugin for any value, false included: if your config/puma.rb still has plugin :solid_queue if ENV["SOLID_QUEUE_IN_PUMA"], take the template's line first.

Memory

Each role is its own container, so give it its own limit (options: memory:). During a deploy, Kamal starts the new container of a role before it stops the old one, so the role uses its memory two times for a moment. The limits of every container on the host, plus the largest limit once more, must fit in the host's memory, with room for the system. This assumes Kamal's default order, one role after another; with boot: parallel_roles: true, add every role that it replaces at the same time. A container that goes over its limit stops with exit code 137: lower its concurrency before you raise the limit.