Speeding up re-compilation of Elixir projects
Recently, one of our Elixir Development Subscription clients noticed their development feedback cycle felt a bit sluggish, they sometimes had to wait seconds, or tens of seconds, for a code change to take effect. Today we will talk about how to understand and diagnose those issues.
Before we get started, it is worth making a distinction between initial compilation and re-compilation: the initial compilation is a one time cost that in the long-term doesn’t matter that much. On the other hand, everytime we make a change to our Elixir source code, part of our project needs to be recompiled and that may take time if Elixir believes it has to recompile a large part of our project. If the compilation is slow then it quickly can become a source of frustration. Let’s fix that!
Types of dependencies
Whenever you change one or more files in your project, Elixir will re-compile all “stale” files as well as everything the “stale” files depend on. Understanding how Elixir tracks dependencies between files is essential to understand how Elixir recompiles our projects.
Say you have a module
a.ex and module B in
b.ex, when we change
a.ex the module
is understandably re-compiled but
B might need to be re-compiled too - why?
Elixir compiler tracks 3 types of dependencies between modules:
runtime dependencies - if module
Acalls some function from module
Adoes not have to be re-compiled, that’s good!
compile-time dependencies - if
Arequires (or imports from, implements behaviour, defdelegates, etc) module
Aneeds to be re-compiled
struct dependencies - If
Bstruct definition changes,
Aneeds to be re-compiled
A has compile-time dependency on
B has runtime dependency on
B doesn’t have to be re-compiled but
A has to! In my experience this was by far
the biggest source of re-compilations.
In your typical Phoenix app you likely have code similar to one below, here’s an excerpt from Hex.pm:
# lib/hexpm_web/views/dashboard_view.ex defmodule HexpmWeb.DashboardView do use HexpmWeb, :view end
# lib/hexpm_web.ex defmodule HexpmWeb do def view() do quote do use Phoenix.View, root: "lib/hexpm_web/templates", namespace: HexpmWeb import HexpmWeb.ViewHelpers # ... end end # ... end
HexpmWeb.ViewHelpers is a collection of utility functions we need in our templates. Importing
that module is convenient, but there’s a cost. Turns out it references one of the Ecto schemas
(which then references other Ecto schemas), which typically would be a runtime dependency,
but because we’re importing
ViewHelpers, our runtime dependency becomes a compile-time
dependency. Anytime one of the schema files changes all of the views (and controllers, etc) needs
to be re-compiled, this can be really problematic!
Turns out that eliminating compile-time dependencies between our views and schemas was just a matter of changing imported calls to regular or aliased calls, we chose the latter:
- import HexpmWeb.ViewHelpers + alias HexpmWeb.ViewHelpers
- <%= pretty_date(last_use.used_at) %> ... + <%= ViewHelpers.pretty_date(last_use.used_at) %> ...
We performed this refactoring on the Hex.pm codebase and that alone removed many compile-time dependencies, reducing the total time it took to recompile the project by 20-30%! We’ve actually wrote a script that automatically performs the import to alias conversion. We’ll talk more about the script in a future blog post, so stay tuned! Meanwhile, you can check out the Hex.pm PR.
However, you may be wondering, should you convert all imports to aliases? That would certainly be a bit too much and we don’t believe that to be necessary. But how can we identify what are the problematic compile-dependencies in our application? Let’s talk about diagnosing.
Generally speaking, we first notice compilation problems when working on our own projects.
For example, you change a single file, then you run
mix test, and you suddenly see:
$ mix test Compiling 27 files (.ex)
Now that we understand that this may be caused by compile-time dependencies, how can we identify and solve those dependencies?
Fortunately, the Elixir team has given us tools to do just that, namely
mix xref graph. For
example, if you changed
lib/foo.ex and that caused a large recompilation, you can run:
$ mix xref graph --sink lib/foo.ex --only-nodes
That will list all files that depend on
lib/foo.ex and which kind of dependency. If some
file has a compile time dependency on
lib/bar.ex, then you can do the
same and see all dependencies on
$ mix xref graph --sink lib/bar.ex --only-nodes
Alternatively, you can remove the
--only-nodes flag and see a tree of dependencies on
foo, albeit it is often quite large for large projects:
$ mix xref graph --sink lib/foo.ex
Finally, if you are not sure where to get started, you can use
mix xref graph --format stats
to get general information about the project. For Hex.pm, here is what it would look like:
$ mix xref graph --format stats Tracked files: 165 (nodes) Compile dependencies: 402 (edges) Structs dependencies: 73 (edges) Runtime dependencies: 429 (edges) Top 10 files with most outgoing dependencies: * lib/hexpm_web/router.ex (42) * lib/hexpm/factory.ex (20) * lib/hexpm_web/controllers/dashboard/organization_controller.ex (16) * lib/hexpm/repository/releases.ex (14) * lib/hexpm/repository/package.ex (14) * lib/hexpm/accounts/user.ex (14) * lib/hexpm/accounts/audit_log.ex (14) * lib/hexpm_web/controllers/package_controller.ex (12) * lib/hexpm/repository/release.ex (12) * lib/hexpm/accounts/users.ex (12) Top 10 files with most incoming dependencies: * lib/hexpm/shared.ex (109) * lib/hexpm_web/web.ex (75) * lib/hexpm_web/router.ex (43) * lib/hexpm_web/views/icons.ex (40) * lib/hexpm_web/controllers/controller_helpers.ex (38) * lib/hexpm_web/controllers/auth_helpers.ex (37) * lib/hexpm_web/endpoint.ex (32) * lib/hexpm/accounts/user.ex (31) * lib/hexpm/repo.ex (25) * lib/hexpm/schema.ex (24)
You can also pass
--label compile so it considers only compile-time dependencies
when computing the stats.
Tips for Phoenix projects
Luckily, the Phoenix team is also well aware of the issues behind over-relying on compile-time dependencies. For this reason, Phoenix v1.4 eliminated two other sources of re-compilations in new Phoenix apps: router helper imports and plugs. However, if you started your Phoenix application before v1.4, your code may not be up to date on the latest practices. So let’s take a look at them.
Regarding router helper imports, this is the same situation as with our view helpers! All we need to do is perform a similar refactoring:
# web.ex - import HexpmWeb.Router.Helpers + alias HexpmWeb.Router.Helpers, as: Routes # lib/hexpm_web/controllers/dashboard_controller.ex - redirect(conn, to: dashboard_path(conn, :profile)) + redirect(conn, to: Routes.dashboard_path(conn, :profile))
Regarding plugs, first just a tiny bit of background. Here’s a sample plug:
defmodule MyPlug do def init(opts), do: opts def call(conn, opts) do # ... end end
init/1 function, as an optimization, is called at compile-time. This way, any heavy work is
only done once, as the project is being compiled, as opposed to on every HTTP request, as is the
case with the
call/2 function. The consequence of this is very similar to imports vs aliases
mentioned above: any module that invokes
plug MyPlug now has a compile time dependency on
MyPlug and a transitive compile-time dependency on any module invoked by
MyPlug, even at
runtime. Fortunately, since Phoenix v1.4 we can configure Plug’s behaviour around init, setting:
# config/dev.exs config :phoenix, :plug_init_mode, :runtime
init/1 is only called at runtime and so we’ll remove yet another source of possible
re-compilations. Remember this is only appropriate for development, don’t set it in production!
In this article we talked about common source of re-compilations and how we can fix them with simple refactorings, notably by changing imports to aliases. We also teased at a tool that can perform the conversion automatically and we’ll talk more about that in the next post. Stay tuned!