Warnings as errors and tests

• Wojtek Mach
• November 24th, 2020
• compiler, tests, warnings

Suppose we have this Elixir code:

# lib/foo.ex
defmodule Foo do
  def foo() do
    a = 42
  end
end

when we compile it, we’ll see this helpful warning:

$ mix compile
Compiling 1 file (.ex)
warning: variable "a" is unused (if the variable is not meant to be used, prefix it with an underscore)
  lib/foo.ex:3: Foo.foo/0

$ echo $?
0

where $? in a Unix shell contains the exit status of the last executed command in the shell, 0 is success, non-zero code is a failure.

To make sure we don’t accidentally commit code that has warnings, we can pass the --warnings-as-errors option:

$ mix compile --warnings-as-errors
Compiling 1 file (.ex)
warning: variable "a" is unused (if the variable is not meant to be used, prefix it with an underscore)
  lib/foo.ex:3: Foo.foo/0

Compilation failed due to warnings while using the --warnings-as-errors option

$ echo $?
1

Notice our shell reports the failure, exit status 1. This is very helpful because many CI systems will automatically fail the build when encountering non-zero exit code from a command.

Warnings as errors during compilation for CIs

Let’s see how to enable warnings as errors on CIs. Here’s a typical GitHub Actions setup for an Elixir project:

# .github/workflows/ci.yml
jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
      - name: Install OTP and Elixir
        uses: actions/setup-elixir@v1
        with:
          otp-version: 23.1.1
          elixir-version: 1.11.1

      - run: mix deps.get
      - run: mix test

mix test will compile the code by calling mix compile and then run tests. To enable warnings as errors, all we need to do is to call mix compile --warnings-as-errors explicitly: (remember to use the right MIX_ENV!)

# (...)
- run: mix deps.get
- run: MIX_ENV=test mix compile --warnings-as-errors
- run: mix test

we could even combine it using mix do task:

# (...)
- run: mix deps.get
- run: MIX_ENV=test mix do compile --warnings-as-errors, test

Warnings as errors during tests for CIs

We have enabled warnings as errors for compiled code but suppose we have warnings in our tests:

defmodule FooTest do
  use ExUnit.Case

  test "foo" do
    a = 42
  end
end

We run our CI command again:

$ MIX_ENV=test mix do compile --warnings-as-errors, test
Compiling 1 file (.ex)
Generated foo app
warning: variable "a" is unused (if the variable is not meant to be used, prefix it with an underscore)
  test/foo_test.exs:5: FooTest."test foo"/1

.

Finished in 0.01 seconds
1 test, 0 failures

Randomized with seed 834982

$ echo $?
0

However, our command was successful, why?

In short, mix compile by default wouldn’t see files in test/. While the test files are of course compiled too, the compilation happens inside mix test and starts with the default compilation options.

We can alleviate it by setting compiler options in test/test_helper.exs, the first file that is loaded before we load any tests:

# test/test_helper.exs
Code.put_compiler_option(:warnings_as_errors, true)

ExUnit.start()

See Code.put_compiler_option/2 for the list of all available options.

Now, if we re-run the command it will fail:

$ MIX_ENV=test mix do compile --warnings-as-errors, test
warning: variable "a" is unused (if the variable is not meant to be used, prefix it with an underscore)
  test/foo_test.exs:5: FooTest."test foo"/1

Compilation failed due to warnings while using the --warnings-as-errors option

$ echo $?
1

Finally, on Elixir v1.12+ instead of changing test_helper.exs, we can simply do mix test --warnings-as-errors. Note we still need to pass --warnings-as-errors to mix compile, see docs!

Summary

We are big fans of keeping projects free of warnings and we usually configure our CIs to ensure that. Here’s an excerpt from GitHub Actions configuration:

# .github/workflows/ci.yml
jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
      - name: Install OTP and Elixir
        uses: actions/setup-elixir@v1
        with:
          otp-version: 23.1.1
          elixir-version: 1.11.1

      - run: mix deps.get
      - run: MIX_ENV=test mix do compile --warnings-as-errors, test

And from Elixir v1.12+, you can do:

- run: MIX_ENV=test mix do compile --warnings-as-errors, test --warnings-as-errors

On large projects there’s usually a lot of compilation output in which case breaking it up might be helpful to be able to inspect each step’s output separately:

- run: MIX_ENV=test mix deps.compile
- run: MIX_ENV=test mix compile --warnings-as-errors
- run: mix test --warnings-as-errors

Finally, we also like to add mix format --check-formatted and mix deps.unlock --check-unused to our CI pipeline to catch even more things before code gets committed.

Happy hacking!