Continuous Integration with Jenkins, Docker-Compose, and Elixir

There’s currently no plugins for Jenkins to tie in nicely with any sort of Elixir testing framework. This lack of plug-n-play makes for another good case to leverage more Docker magic.

So for this test, I wanted to leverage Docker-Compose so that I could future proof our unit tests. Even though I know unit tests should not rely on any external backend services, sometimes its unavoidable — so I wanted to make sure we would have room to grow in case I needed to support unit tests that needed a Docker container running SQL or ElasticSearch or something like that.

The first problem we ran into was getting tests out of Elixir into a format Jenkins could parse. JUnit is the obvious choice for Jenkins-Junkies, so we’re using junit_formatter to output our tests results in XML. To get this working edit your deps in mix.exs to include junit_formatter. Below is a sample from what we’re using for our testing dependencies:

#mix.exs 
  defp deps do
    [
      #Other deps go here....
      # dev/test deps
      {:exvcr, "~> 0.10", only: :test},
      {:stream_data, "~> 0.4.0", only: [:dev, :test]},
      {:credo, "~> 0.9.0-rc1", only: [:dev, :test], runtime: false},
      {:distillery, "~> 1.5", runtime: false},
      {:excoveralls, "~> 0.8", only: :test},
      {:dialyxir, "~> 0.5.0", only: [:dev], runtime: false},
      {:junit_formatter, "~> 2.2", only: [:test]}
    ]
  end

Then to have our tests use the junit_formatter, we have a helper test file that is configured to use JUnitFormatter

#test_helper.exs
ExUnit.configure(formatters: [JUnitFormatter, ExUnit.CLIFormatter])
ExUnit.start()

We’re only launching a single container running our application, so our compose file is pretty simple. Below is our docker-compose-test.yml. A very important part of this is the volumes section — this makes sure that our docker container /opt/app directory is mounted to our $PWD when Jenkins runs the job. This makes it a lot easier to grab test results and publish them out of the ./_build directory after the test runs.

#docker-compose-test.yml
version: '3'
services:
  web:
    build:
      context: ./
      dockerfile: Dockerfile.test
    command: ./infra/build/test/unittest.sh
    volumes:
      - .:/opt/app

Our Dockerfile.test is very simple as well (I’d like to move to the Alpine Elixir image, but we ran into issues using it, so we’re using the big fat Elixir image instead)

#Dockerfile.test
FROM elixir:1.6.5
ENV REFRESHED_AT=2018-04-06 \
    # Set this so that CTRL+G works properly
    TERM=xterm
RUN apt-get update \
  && update-ca-certificates --fresh \
  && rm -rf /var/lib/apt/lists/* \
  && mix local.hex --force \
  && mix local.rebar --force \
  && mix archive.install hex sobelow --force
WORKDIR /opt/app

unittest.sh is also a straight forward script to run our Elixir tests and linters:

#!/bin/bash
set -e
echo "## get deps"
mix do deps.get, deps.compile, dialyzer --plt
echo "## check for warnings"
mix do clean, compile --force --warning-as-errors
echo "## checking format"
mix format --check-formatted
echo "## run credo for linting"
mix credo --strict
echo "## running tests"
mix test --trace
echo "## run coverage"
mix coveralls.html
echo "## run dialyzer"
mix dialyzer
echo "## run sobelow security check"
mix sobelow --ignore Config.HTTPS,Config.CSRF,Config.Headers --private --verbose --exit

To run our tests, we do some magic to test if docker-compose-test.yml has any changes since the last build and force a rebuild if it has been changed, otherwise we just run normally. Here’s our docker_wrapper.sh script that runs it all:

#docker_wrapper.sh
if $DOCKERCOMPOSE_CHANGED
then
  echo "## rebuilding docker-compose"
  _cmd="docker-compose --file ./docker-compose-test.yml up --force-recreate --abort-on-container-exit --build web"
else
  echo "## skipping docker-compose rebuild"
  _cmd="docker-compose --file ./docker-compose-test.yml up --abort-on-container-exit"
fi
echo "## running tests in a docker container"
echo "> ${_cmd}"
/bin/bash -l -c "${_cmd}"

Tying it all together on Jenkins is now a breeze. Whenever a PR or a check-in comes in to master, we trigger tests.

The Build step is simple — we just call our docker_wrapper.sh script and off we go —

Jenkins Build Step

And to publish test results — I use the xUnit plugin so that I can fine tune control of how many test are allowed to fail or be skipped. As you can see from the screenshot below, there are no steps to copy the file from the docker container because we used the volumes directive in our docker-compose-test.yml file to make sure it’s sitting locally in the Jenkins build directory. As a final step, we publish the HTML report with our code coverage.

Jenkins Publishing Results

The end result is a unit test job just like any other in Jenkins, which is exactly what we wanted! The test results are parsed natively by Jenkins because we’re using JUnit. No dependencies, except for Docker and Docker-Compose, need to be installed on any of the Jenkins Slave nodes. My favorite aspect is how dead simple the job configuration ends up being — call a script and publish the results!

Our Pull Requests Test Page

Shout out to Khaja Minhajuddin for helping to develop this testing strategy!