Jun 10 2017

Deploying Phoenix Apps

Liam

Blog post image

Many facets of Rails development are transferrable to Phoenix. Ecto migrations, for example, are extremely similar to ActiveRecord migrations. Deployment is not one of those things that Elixir and Ruby have in common. This article help you navigate the differences and leverage the power of the BEAM!


In Ruby, generally one would use Capistrano and its Rails integration to deploy a release to production, which looks something like this:

  • Make sure that configuration files are in the right place
  • Pull the latest version of master into a local copy of the repository
  • Create a “release” directory
  • Copy all of the files from the repository into the release directory
  • Digest assets
  • Swap a symlink that points from the new release directory to the “current” directory
  • Run all the database migrations that haven’t been run yet
  • Restart the application server (if necessary) so that it loads the new files in the “current” directory

One of the biggest value propositions for the BEAM VM, Erlang, Elixir, and thus Phoenix, is the concept of a “release.” These releases can be hot upgraded and downgraded. This is a pretty far-out idea; it essentially means that instead of bringing down your service momentarily and bringing up a new version, you modify the code in memory as it’s running. Naturally, there are a lot of caveats here, so this post describes my experience.

Methods I’ve used

Edeliver

Edeliver is probably the closest thing you’ll find to Capistrano, though it fills a different niche (e.g., it won’t run migrations for you out of the box). It is being actively developed and has a few bugs which I filed five months ago which are still open.

Although I found Edeliver’s philosophy (“it’s just bash scripts”) refreshingly simple (and its CLI very appealing), the issues I ran into above were sufficiently mysterious to dissuade me from going down this road any further.

Gatling

Gatling is a great little tool. It’s basically a “roll your own Heroku” utility, in that the servers are Git remotes and you use a simple git push production to deploy your code. It features an elegant DSL for executing callsbacks at various phases of deployment (like Capistrano) and is probably the tool featuring the greatest return on investment. I had a great time with it, save for a couple of design issues:

  • When the ERTS version changes on your development machine (which happens regularly with sudo apt upgrade), your application will remain up, but the next upgrade will fail. Not Gatling’s fault, but a caveat when using hot upgrades. Once you get into this state, though, how to recover is undocumented and seems basically impossible without manual intervention. It would be nice to be able to build a new release, push it, and do an old-fashioned “cold upgrade” (just restart the service!). This is a consequence of using include_erts: true in the build configuration, which is required for hot upgrades – but where is ERTS pulled from? Your local machine (which builds the release).
  • Gatling tries to configure nginx for you, but that configuration isn’t configurable - meaning that you’d have to provision the server with Gatling (i.e., use the tool to put the nginx configuration in place at first), then hand-edit the files on production to install things like SSL certificates. However, because Gatling doesn’t overwrite the production configs you have in place, if you copy these first, it’s easy to work around this limitation.
  • It puts an upstart script in /etc/init that ensures that the service comes up if the server reboots. This is great (and a serious limitation of other tools), but I prefer native systemd scripts which have been the standard as of the Ubuntu LTS release 16.04. This gives you integration with journald, which makes it easy to see, search and rotate all your logs in one place (among other benefits).

Distillery + A small shell script

It’s a convention at Tenex to write a short, self-documenting shell script in the root of the project’s directory that can deploy the code to different environments. Often this is a one liner: cap ${1:-staging} deploy. This, in my opinion, is a pretty solid choice. It’s very “no magic” and there’s not a whole lot that can go wrong.

#!/bin/bash
PROD_HOST='awesome-service.tenex.tech'
PROD_USER='awesome-service'
BUILD_PATH='_build/prod/rel/awesome_service'

# set bash "safe mode"
set -euo pipefail
endpoint="$PROD_USER@$PROD_HOST"
# change directory to base of project
pushd "$(dirname $0)"
# install the frontend packages
yarn install
# build the frontend
./node_modules/brunch/bin/brunch build --production
# digest the assets and build a release
# the "release" at the end is a Distillery Mix task
MIX_ENV=prod mix do phoenix.digest, release
# transfer the release to production
rsync -avz "${BUILD_PATH}/" "${endpoint}:app"
# bounce the service
ssh "$endpoint" \
  sudo systemctl restart awesome-service
# check to make sure it worked
ssh "$endpoint" \
  systemctl status awesome-service

As you can see by the systemctl commands, this uses systemd. Distillery has great documentation, including its own guide on setting up your unit file. I recommend setting it up to run in the foreground so that logging can be better integrated with journald. Of course this approach comes with a major limitation: no hot upgrades! But the advantages of BEAM, Elixir and Phoenix go far, far beyond hot upgrades, and although they look great at first, I think the additional effort (and unpleasant surprises at the worst possible time) offset the advantage of eliminating a second or two of downtime.