Speeding up re-compilation of Elixir projects
UPDATE #1: We have updated this article to mirror the latest improvements coming in Elixir v1.11+.
Recently, one of our Elixir Development Subscription clients noticed their development feedback cycle felt a bit sluggish, they sometimes had to wait seconds, or tens of seconds, for a code change to take effect. Today we will talk about how to understand and diagnose those issues.
Before we get started, it is worth making a distinction between initial compilation and re-compilation: the initial compilation is a one time cost that in the long-term doesn’t matter that much. On the other hand, everytime we make a change to our Elixir source code, part of our project needs to be recompiled and that may take time if Elixir believes it has to recompile a large part of our project. If the compilation is slow then it quickly can become a source of frustration. Let’s fix that!
Types of dependencies
Whenever you change one or more files in your project, Elixir will re-compile all “stale” files as well as everything the “stale” files depend on. Understanding how Elixir tracks dependencies between files is essential to understand how Elixir recompiles our projects.
Say you have a module
a.ex and module B in
b.ex, when we change
a.ex the module
is understandably re-compiled but
B might need to be re-compiled too - why?
From Elixir v1.11+, the Elixir compiler tracks 3 types of dependencies between modules:
runtime dependencies - if module
Acalls some function from module
Adoes not have to be re-compiled, that’s good!
compile-time dependencies - if
Arequires (or imports from, implements behaviour, defdelegates, etc) module
Aneeds to be re-compiled
export dependencies - If
Bor uses a struct from
B, such as
Aneeds to be re-compiled whenever
Badds or remove a function or changes its struct definition
A has compile-time dependency on
B has runtime dependency on
B doesn’t have to be re-compiled but
A has to! In our experience this is by far
the biggest source of re-compilations.
Generally speaking, we first notice compilation problems when working on our own projects.
For example, you change a single file, then you run
mix test, and you suddenly see:
$ mix test Compiling 27 files (.ex)
Now that we understand that this may be caused by compile-time dependencies, how can we identify and solve those dependencies?
Fortunately, the Elixir team has given us tools to do just that, namely
mix xref graph. For
example, if you changed
lib/foo.ex and that caused a large recompilation, you can run:
$ mix xref graph --sink lib/foo.ex --only-nodes
That will list all files that depend on
lib/foo.ex and which kind of dependency. If some
file has a compile time dependency on
lib/bar.ex, then you can do the
same and see all dependencies on
$ mix xref graph --sink lib/bar.ex --only-nodes
From Elixir v1.11, you will be able to filter this tree to all transitive compile time dependencies:
$ mix xref graph --sink lib/foo.ex --label compile
Alternatively, you can remove the
–only-nodes flag and see a tree of dependencies on
lib/foo.ex, albeit it is often quite deep for large projects:
$ mix xref graph --sink lib/foo.ex
Finally, if you are not sure where to get started, you can use
mix xref graph –format stats
to get general information about the project. For Hex.pm, here is what it would look like:
$ mix xref graph --format stats Tracked files: 165 (nodes) Compile dependencies: 402 (edges) Structs dependencies: 73 (edges) Runtime dependencies: 429 (edges) Top 10 files with most outgoing dependencies: * lib/hexpm_web/router.ex (42) * lib/hexpm/factory.ex (20) * lib/hexpm_web/controllers/dashboard/organization_controller.ex (16) * lib/hexpm/repository/releases.ex (14) * lib/hexpm/repository/package.ex (14) * lib/hexpm/accounts/user.ex (14) * lib/hexpm/accounts/audit_log.ex (14) * lib/hexpm_web/controllers/package_controller.ex (12) * lib/hexpm/repository/release.ex (12) * lib/hexpm/accounts/users.ex (12) Top 10 files with most incoming dependencies: * lib/hexpm/shared.ex (109) * lib/hexpm_web/web.ex (75) * lib/hexpm_web/router.ex (43) * lib/hexpm_web/views/icons.ex (40) * lib/hexpm_web/controllers/controller_helpers.ex (38) * lib/hexpm_web/controllers/auth_helpers.ex (37) * lib/hexpm_web/endpoint.ex (32) * lib/hexpm/accounts/user.ex (31) * lib/hexpm/repo.ex (25) * lib/hexpm/schema.ex (24)
Once you learn from where the compile-time dependencies come from, the goal is to refactor the code in order to remove said dependencies. Let’s see some examples from the Phoenix team.
Tips for Phoenix projects
Luckily, the Phoenix team is also well aware of the issues behind over-relying on compile-time dependencies. For this reason, Phoenix v1.4 eliminated two common sources of re-compilations in new Phoenix apps: router helper imports and plugs. However, if you started your Phoenix application before v1.4, your code may not be up to date on the latest practices. So let’s take a look at them.
The first change done by the Phoenix team was to rewrite router imports to aliases, like this:
# web.ex - import HexpmWeb.Router.Helpers + alias HexpmWeb.Router.Helpers, as: Routes # lib/hexpm_web/controllers/dashboard_controller.ex - redirect(conn, to: dashboard_path(conn, :profile)) + redirect(conn, to: Routes.dashboard_path(conn, :profile))
In Elixir v1.10 and earlier, imports were considered compile-time dependencies, so this change yielded large improvements to recompilation times. Elixir v1.11 improved its compiler so imports are now tagged as exports, no longer making this change strictly required. Still, moving from imports to aliases converts them from an export to a runtime dependency. Furthermore, many developers prefer aliases over imports as it makes the code clearer.
The second change was related to plugs. First just a tiny bit of background. Here’s a sample plug:
defmodule MyPlug do def init(opts), do: opts def call(conn, opts) do # ... end end
init/1 function, as an optimization, is called at compile-time. This way, any heavy work is
only done once, as the project is being compiled, as opposed to on every HTTP request, as is the
case with the
call/2 function. The consequence of this is that any module that invokes
plug MyPlug now has a compile time dependency on
MyPlug and a transitive compile-time
dependency on any module invoked by
MyPlug, even at runtime. Fortunately, since Phoenix v1.4
we can configure Plug’s behaviour around init, setting:
# config/dev.exs config :phoenix, :plug_init_mode, :runtime
init/1 is only called at runtime and so we’ll remove yet another source of possible
re-compilations. Remember this is only appropriate for development, don’t set it in production!
In this article we talked about common source of re-compilations and how we can fix them with simple refactorings, such as changing imports to aliases and by avoiding compile-time dependencies.