Notes to self

Deploying Rails on a single server with Kamal

, updated on

Here’s one way of a cloud-independent deployment of Rails, Sidekiq, PostgreSQL, and Redis on single virtual server with Kamal.

TIP: Checkout Kamal Handbook, my new book on Kamal.

What’s Kamal

Kamal is a new deployment tool from 37signals (authors of Basecamp and Hey). It’s a philosophical successor to Capistrano but made for the container era.

Kamal keeps things simple by employing a push agentless model without any declarative orchestration. The model has its downsides, but on the plus side it uses the plain Docker you likely know. Conceptually, Kamal deployment is a combination of Bash (deployment hooks), Ruby (Kamal’s source), YAML (configuration), Traefik (load balancer), and Docker (process manager).

It will automatically install Docker, (cross) build your containers, push them to a container repository, and deploy the new versions by fetching it on the servers. Zero-downtime deployment is done by booting the new version before switching over with Traefik.

One thing to note about Kamal is that while it can run your containers, it doesn’t concern itself with further server management.

Single server

A lot of great projects can run on a single server. I’ll almost always consider starting this way as it simplified a lot of things. Kamal was primarily designed as a multi-node deployment tool, and as such it misses good documentation on a single-server deployment. Nevertheless it’s still possible.

The three core ideas to achieve that are:

  • Making Traefik issue the Let’s Encrypt certificates which will require a DNS challenge and a local directory.

  • Introducing a private bridge Docker network for container communication without the need of using IP addresses or exposing ports.

  • Having a mounted volume or a simple directory for a local disk service to avoid a 3rd-party object storage.

In this post we’ll look at how to do that for a very typical Ruby on Rails application with a PostgreSQL relational database, Sidekiq, and a Redis key-value store. Please make sure you are already a bit familiar with Kamal.

Pre-requisities

To write a desired configuration, we’ll need to prepare a couple of things beforehand.

  • Buy a domain name for the SSL/TLS setup.

  • Set up emails on the domain name for the Let’s Encrypt email contact.

  • Set up a Docker registry.

  • Creat an SSH key.

    You can create one like this:

    $ ssh-keygen -t ed25519 -C "support@imagebinary.com"

  • Buy a virtual private server and set it up with the SSH key from previous step.

  • Update DNS A and CNAME entries so that the domain leads to the public IP of the VPS.

  • Create an .env file where you’ll put your secrets.

  • Prepare RAILS_MASTER_KEY for Rails Encrypted Credentials.

    Once you are done, install Kamal and run init:

    $ gem install kamal
$ kamal init

This should install some files including config/deploy.yml.

Configuration

Your config/deploy.yml configuration starts with your application name and image:

# config/deploy.yml

# Name of your application. Used to uniquely configure containers.
service: myapp

image: username/imagename

Kamal’s assumption is that one application is served by one single image. The service name will be prepended to your application and accessory container names.

Registry

Once you specify your image, you’ll need to tell Kamal where to push it and when to pull it from. Provide your Docker registry server, username, and password:

# config/deploy.yml
...
# Credentials for your image host.
registry:
  # Specify the registry server, if you're not using Docker Hub
  # server: registry.digitalocean.com / ghcr.io / ...
  username: strzibnyj

  # Always use an access token rather than real password when possible.
  password:
    - KAMAL_REGISTRY_PASSWORD

If you use Dockerhub you won’t need the server directive as it’s default. Then put the actual value of KAMAL_REGISTRY_PASSWORD inside the .env file.

Traefik

Traefik is the bread and butter of our single server setup. It will handle TLS termination, https: redirect, and zero downtime deployments. As a special component it has its own entry within config/deploy. Traefik takes options which let us mount the acme.json file required for Let’s Encrypt and specify the Docker private network, and args which lets us specify the entrypoints as well as the details for challenge:

# config/deploy.yml
...
traefik:
  options:
    publish:
      - "443:443"
    volume:
      - "/letsencrypt/acme.json:/letsencrypt/acme.json"
    network: "private"
  args:
    entryPoints.web.address: ":80"
    entryPoints.websecure.address: ":443"
    certificatesResolvers.letsencrypt.acme.email: "support@imagebinary.com"
    certificatesResolvers.letsencrypt.acme.storage: "/letsencrypt/acme.json"
    certificatesResolvers.letsencrypt.acme.httpchallenge: true
    certificatesResolvers.letsencrypt.acme.httpchallenge.entrypoint: web

servers:
  ...

That’s however not the entire Traefik story as we’ll need the routers host rules. We’ll add them next when specifying our servers configuration.

Servers

The main configuration of Kamal happens inside servers directive. In our case we’ll have only one host hosting both the Rails application server as the web container as well as Sidekiq as the job container. We’ll need to specify our public IP address among the hosts array of servers, provide domain name labels for the web container, and specify custom cmd to run the background jobs:

# config/deploy.yml
...

servers:
  web:
    hosts:
      - 165.227.160.215
    labels:
      traefik.http.routers.imagebinary.rule: Host(`imagebinary.com`)
      traefik.http.routers.imagebinary_secure.entrypoints: websecure
      traefik.http.routers.imagebinary_secure.rule: Host(`imagebinary.com`)
      traefik.http.routers.imagebinary_secure.tls: true
      traefik.http.routers.imagebinary_secure.tls.certresolver: letsencrypt
    options:
      network: "private"
  job:
    hosts:
      - 165.227.160.215
    # Or something like bin/job
    cmd: bundle exec sidekiq
    # Limit workers resources
    #
    # options:
    #   cap-add: true
    #   cpu-count: 2
    options:
      network: "private"

We’ll again use options to specify our new private network. If we want we can limit server resources with cap-add. This can be helpful if we don’t want our workers to take down the entire application.

PostgreSQL

PostgreSQL is our primary accessory which is the Kamal terminology for auxiliary services.

A lot of people will opt-in for a managed service, but if you want to manage MySQL or PostgreSQL yourself, it’s nothing you cannot do. Kamal doesn’t know about databases per se, but will happily deploy any additional services on hosts you specify.

To deploy PostgreSQL, we’ll use the official Docker image, specify the host, port, environment variables, custom entrypoint, and location for the raw database data. We’ll need to provide the database user, name and a password:

# config/deploy.yml
...
accessories:
  db:
    image: postgres:15
    host: 165.227.160.215
    port: 5432
    env:
      clear:
        POSTGRES_USER: "myapp"
        POSTGRES_DB: 'myapp_production'
      secret:
        - POSTGRES_PASSWORD
    files:
      - config/init.sql:/docker-entrypoint-initdb.d/setup.sql
    directories:
      - data:/var/lib/postgresql/data

The files directive lets us provide our own entrypoint. The config/init.sql needs to create the database as expected from the config/database.yml configuration:

# config/init.sql
CREATE DATABASE myapp_production;

And on the Rails side:

# config/database.yml
production:
  <<: *default
  username: myapp
  password: <%= ENV["POSTGRES_PASSWORD"] %>
  database: myapp_production
  host: <%= ENV["DB_HOST"] %>

Put your POSTGRES_PASSWORD inside .env. Notice that we are not using the DATABASE_URL string so make sure it’s not set.

In my deployment book I suggest using the peer authentication for better security (no passwords) and direct UNIX sockets for better performance on a single host, but this setup will become handy later when scaling up to multiple hosts.

Also remember that you’ll now have to make your own backups of PostgreSQL databases located on /myapp-db/data which will be automatically created for you if specified as a Kamal directory.

Redis

Redis is a bit tricky at first. The default image accepts connections from anywhere so if you would just expose the port without providing a specific bind directive, you would let anyone to connect. Some people might start with using just a password, but I recommend to keep things more secure if you can.

We’ll use Kamal roles to make Redis available for the application, pass the --requirepass to require a password and depend on the private Docker network from before. Since we’ll pass the password directly as a custom image command, we’ll take the advantage of Kamal processing the file as ERB template:

# config/deploy.yml
...
accessories:
  db:
  ...
  redis:
    image: redis:latest
    roles:
      - web
      - job
    cmd: "redis-server --requirepass <%= File.read('/path/to/redis/password') %>"
    volumes:
      - /var/lib/redis:/data
    options:
      network: "private"

If the password is secret, your Redis connection URL will become redis://:secret@myapp-redis:6379/1. Save it as a secret REDIS_URL. The port stays 6379 in this case as it’s specified in the image.

They are more ways to provide custom configuration, either with a custom Dockerfile, or with envs and files as we did with PostgreSQL. However, if we want to stick with the official image, the easiest way to not expose the port to public is by leveraging the private network and setting the password with --requirepass.

On the Rails side make sure you are setting up REDIS_URL everywhere where it’s required. In case of Sidekiq, you likely need to set it both for client and server:

# app/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.redis = {
    url: ENV.fetch("REDIS_URL", "redis://localhost:6379/1")
  }
end

Sidekiq.configure_client do |config|
  config.redis = {
    url: ENV.fetch("REDIS_URL", "redis://localhost:6379/1")
  }
end

Assets

Since we don’t have a dedicated file server in this setup, we’ll let Rails handle assets too. If you used the official Rails Dockerfile, assets should be part of your image already and the only remaining thing is to recheck the Rails configuration:

# Disable serving static files from the `/public` folder by default since
# Apache or NGINX already handles this.
config.public_file_server.enabled = ENV["RAILS_SERVE_STATIC_FILES"].present?

Make sure that the public_file_server configuration is enabled. In the default setup we’ll just have to set RAILS_SERVE_STATIC_FILES environment variable in the Kamal envs section and things will start to work.

Active Storage

A lot of people will opt-in for an object storage, but starting out with a disk service will save you some setup time and costs. Active Storage local storage only really need an access to a directory on a server. Assuming you set up the /storage directory beforehand, you only need to mount this path in Kamal:

# config/deploy.yml
image: ...

volumes:
  - "/storage:/rails/storage"

servers:
  ...

Remember that the Rails root is dependening on the application location inside the Docker image. In the official Rails Dockerfile that’s /rails but chances are you have there /app or something similar.

The ./storage path can be configured in config/storage.yml:

test:
  service: Disk
  root: <%= Rails.root.join("tmp/storage") %>

local:
  service: Disk
  root: <%= Rails.root.join("storage") %>

Also recheck your Rails configuration:

# Store uploaded files on the local file system (see config/storage.yml for options).
config.active_storage.service = :local

Envs

At this point you should have all the envs you need. In the env section, put the public configuration under clear and private one under secret:

env:
  clear:
    HOSTNAME: imagebinary.com
    DB_HOST: 10.135.0.3
    RAILS_SERVE_STATIC_FILES: true
    RAILS_LOG_TO_STDOUT: true
  secret:
    - KAMAL_REGISTRY_PASSWORD
    - RAILS_MASTER_KEY
    - POSTGRES_PASSWORD
    - REDIS_URL

Secret values will be automatically populated from an .env file so make sure all the value are there.

I am also setting RAILS_LOG_TO_STDOUT to true since we want to access logs from Docker and kamal app logs.

Hooks

Kamal let’s us extend the default deployment flow with the following hooks:

$ ls .kamal/hooks
post-deploy.sample  pre-build.sample  pre-connect.sample  pre-deploy.sample

The unfortunate thing is that we need a one-time hook running on the server after Docker is installed.

The closest thing to use is the pre-deploy hook, but we can also do these manual steps after Kamal installs Docker on the servers:

$ ssh root@165.227.160.215
root# mkdir -p /letsencrypt && touch /letsencrypt/acme.json && chmod 600 /letsencrypt/acme.json
root# mkdir /storage -p && chmod a+rwx /storage
root# docker network create -d bridge private

This sets up the directory for Let’s Encrypt, an example /storage disk space and the Docker private network called private.

Deploy

Once you went through the manual steps from the beginning and wrote your config/deploy.yml file, you are ready to deploy with:

$ kamal setup

Remember that if you haven’t created the Let’s Encrypt directory or the Docker network, things will break. But even if they do you can ssh into the machine, run whatever you need and continue with kamal deploy.

Later you can update your environments with:

$ # won't restart services
$ kamal env push

And redeploy with:

$ kamal deploy

If there is something wrong, check the logs:

$ kamal traefik logs
$ kamal app logs

Auxiliary services must be deleted and rebooted:

$ kamal accessory delete redis
$ kamal accessory boot redis

Remember it’s just Docker so ssh into the machine and run your usual Docker commands.

Notes

Please take this post as an inspiration rather than a hardened example:

  • You might not want Rails to serve static assets. Create an NGINX container or introduce a CDN as Traefik is pure load balancer and won’t serve your files.

  • You might not want to expose your database to the outside world. I used public setup for the database and private for Redis as it shows the two different approaches.

  • Your mount volumes should be ideally accessed by web and job containers only.

  • You don’t want to use root user to access your VM. I always give sudo privileges to a different user and disable logging as root.

I welcome your feedback and will update the post accordingly.

ruby rails linux kamal deployment

Work with me

I have some availability for contract work. I can be your fractional CTO, a Ruby on Rails engineer, or consultant. Write me at strzibny@strzibny.name.

RSS