Speeding up re-compilation of Elixir projects

• Wojtek Mach
• March 23rd, 2020
• compiler, elixir, phoenix

UPDATE #1: We have updated this article to mirror Elixir v1.11+’s best practices.

Recently, one of our Elixir Development Subscription clients noticed their development feedback cycle felt a bit sluggish, they sometimes had to wait seconds, or tens of seconds, for a code change to take effect. Today we will talk about how to understand and diagnose those issues.

Before we get started, it is worth making a distinction between initial compilation and re-compilation: the initial compilation is a one time cost that in the long-term doesn’t matter that much. On the other hand, everytime we make a change to our Elixir source code, part of our project needs to be recompiled and that may take time if Elixir believes it has to recompile a large part of our project. If the compilation is slow then it quickly can become a source of frustration. Let’s fix that!

Types of dependencies

Whenever you change one or more files in your project, Elixir will re-compile all “stale” files as well as everything the “stale” files depend on. Understanding how Elixir tracks dependencies between files is essential to understand how Elixir recompiles our projects.

Say you have a module A in a.ex and module B in b.ex, when we change a.ex the module A is understandably re-compiled but B might need to be re-compiled too - why?

From Elixir v1.11+, the Elixir compiler tracks 3 types of dependencies between modules:

• runtime dependencies - if module A calls some function from module B and B changes, A does not have to be re-compiled, that’s good!

• compile-time dependencies - if A uses any functionality from B in its module body (instead of inside its functions) and B changes, A needs to be re-compiled

• export dependencies - If A uses imports B or uses a struct from B, such as %B{}, A needs to be re-compiled whenever B adds or remove a function or changes its struct definition

Additionally, if A has compile-time dependency on B, and B has runtime dependency on C, if C changes, B doesn’t have to be re-compiled but A has to! In our experience this is by far the biggest source of re-compilations.

Diagnosing

Generally speaking, we notice compilation problems when working on our own projects. For example, you change a single file, then you run mix test, and you suddenly see:

$ mix test
Compiling 27 files (.ex)

Now that we understand that this may be caused by compile-time dependencies, how can we identify and solve those dependencies?

The Elixir team has given us tools to do just that, namely mix xref graph. For example, if you changed lib/foo.ex and that caused a large recompilation, you can run:

$ mix xref graph --sink lib/foo.ex --only-nodes

That will list all files that depend on lib/foo.ex and which kind of dependency. If some file has a compile time dependency on lib/foo.ex, say lib/bar.ex, then you can do the same and see all dependencies on lib/bar.ex:

$ mix xref graph --sink lib/bar.ex --only-nodes

Alternatively, you can remove the --only-nodes flag and see a tree of dependencies on lib/foo.ex, albeit it is often quite deep for large projects:

$ mix xref graph --sink lib/foo.ex

From Elixir v1.11, you will be able to filter this tree to all transitive compile time dependencies:

$ mix xref graph --sink lib/foo.ex --label compile

Finally, if you are not sure where to get started, you can use mix xref graph --format stats to get general information about the project. For Hex.pm, here is what it would look like:

$ mix xref graph --format stats
Tracked files: 165 (nodes)
Compile dependencies: 402 (edges)
Structs dependencies: 73 (edges)
Runtime dependencies: 429 (edges)

Top 10 files with most outgoing dependencies:
  * lib/hexpm_web/router.ex (42)
  * lib/hexpm/factory.ex (20)
  * lib/hexpm_web/controllers/dashboard/organization_controller.ex (16)
  * lib/hexpm/repository/releases.ex (14)
  * lib/hexpm/repository/package.ex (14)
  * lib/hexpm/accounts/user.ex (14)
  * lib/hexpm/accounts/audit_log.ex (14)
  * lib/hexpm_web/controllers/package_controller.ex (12)
  * lib/hexpm/repository/release.ex (12)
  * lib/hexpm/accounts/users.ex (12)

Top 10 files with most incoming dependencies:
  * lib/hexpm/shared.ex (109)
  * lib/hexpm_web/web.ex (75)
  * lib/hexpm_web/router.ex (43)
  * lib/hexpm_web/views/icons.ex (40)
  * lib/hexpm_web/controllers/controller_helpers.ex (38)
  * lib/hexpm_web/controllers/auth_helpers.ex (37)
  * lib/hexpm_web/endpoint.ex (32)
  * lib/hexpm/accounts/user.ex (31)
  * lib/hexpm/repo.ex (25)
  * lib/hexpm/schema.ex (24)

Once you learn from where the compile-time dependencies come from, the goal is to refactor the code in order to remove said dependencies. Let’s see some examples from the Phoenix team.

Tips for Phoenix projects

Luckily, the Phoenix team is also well aware of the issues behind over-relying on compile-time dependencies. For this reason, Phoenix v1.4 eliminated two common sources of re-compilations in new Phoenix apps: router helper imports and plugs. However, if you started your Phoenix application before v1.4, your code may not be up to date on the latest practices. So let’s take a look at them.

The first change done by the Phoenix team was to rewrite router imports to aliases, like this:

# web.ex
-  import HexpmWeb.Router.Helpers
+  alias HexpmWeb.Router.Helpers, as: Routes

# lib/hexpm_web/controllers/dashboard_controller.ex
-    redirect(conn, to: dashboard_path(conn, :profile))
+    redirect(conn, to: Routes.dashboard_path(conn, :profile))

In Elixir v1.10 and earlier, imports were considered compile-time dependencies, so this change yielded large improvements in recompilation times. Elixir v1.11 improved its compiler so imports are now tagged as export dependencies, therefore this change is no longer strictly required. Still, moving from imports to aliases converts them from an export to a runtime dependency. Furthermore, many developers prefer aliases over imports as it makes the code clearer.

The second change was related to plugs. First just a tiny bit of background. Here’s a sample plug:

defmodule MyPlug do
  def init(opts), do: opts

  def call(conn, opts) do
    # ...
  end
end

The init/1 function, as an optimization, is called at compile-time. This way, any heavy work is only done once, as the project is being compiled, as opposed to on every HTTP request, as is the case with the call/2 function. The consequence of this is that any module that invokes plug MyPlug now has a compile time dependency on MyPlug and a transitive compile-time dependency on any module invoked by MyPlug, even at runtime. Fortunately, since Phoenix v1.4 we can configure Plug’s behaviour around init, setting:

# config/dev.exs
config :phoenix, :plug_init_mode, :runtime

will ensure init/1 is only called at runtime and so we’ll remove yet another source of possible re-compilations. Remember this is only appropriate for development, don’t set it in production!

Summary

In this article we talked about common source of re-compilations and how we can fix them with simple refactorings, such as changing imports to aliases and by avoiding compile-time dependencies.