Speeding up re-compilation of Elixir projects

Recently, one of our Elixir Development Subscription clients noticed their development feedback cycle felt a bit sluggish, they sometimes had to wait seconds, or tens of seconds, for a code change to take effect. Today we will talk about how to understand and diagnose those issues.

Before we get started, it is worth making a distinction between initial compilation and re-compilation: the initial compilation is a one time cost that in the long-term doesn’t matter that much. On the other hand, everytime we make a change to our Elixir source code, part of our project needs to be recompiled and that may take time if Elixir believes it has to recompile a large part of our project. If the compilation is slow then it quickly can become a source of frustration. Let’s fix that!

Types of dependencies

Whenever you change one or more files in your project, Elixir will re-compile all “stale” files as well as everything the “stale” files depend on. Understanding how Elixir tracks dependencies between files is essential to understand how Elixir recompiles our projects.

Say you have a module A in a.ex and module B in b.ex, when we change a.ex the module A is understandably re-compiled but B might need to be re-compiled too - why?

Elixir compiler tracks 3 types of dependencies between modules:

  • runtime dependencies - if module A calls some function from module B and B changes, A does not have to be re-compiled, that’s good!

  • compile-time dependencies - if A requires (or imports from, implements behaviour, defdelegates, etc) module B and module B changes, A needs to be re-compiled

  • struct dependencies - If A uses %B{} and B struct definition changes, A needs to be re-compiled

Additionally, if A has compile-time dependency on B, and B has runtime dependency on C, if C changes, B doesn’t have to be re-compiled but A has to! In my experience this was by far the biggest source of re-compilations.

Example

In your typical Phoenix app you likely have code similar to one below, here’s an excerpt from Hex.pm:

# lib/hexpm_web/views/dashboard_view.ex
defmodule HexpmWeb.DashboardView do
  use HexpmWeb, :view
end
# lib/hexpm_web.ex
defmodule HexpmWeb do
  def view() do
    quote do
      use Phoenix.View,
        root: "lib/hexpm_web/templates",
        namespace: HexpmWeb

      import HexpmWeb.ViewHelpers

      # ...
    end
  end

  # ...
end

HexpmWeb.ViewHelpers is a collection of utility functions we need in our templates. Importing that module is convenient, but there’s a cost. Turns out it references one of the Ecto schemas (which then references other Ecto schemas), which typically would be a runtime dependency, but because we’re importing ViewHelpers, our runtime dependency becomes a compile-time dependency. Anytime one of the schema files changes all of the views (and controllers, etc) needs to be re-compiled, this can be really problematic!

Turns out that eliminating compile-time dependencies between our views and schemas was just a matter of changing imported calls to regular or aliased calls, we chose the latter:

-  import HexpmWeb.ViewHelpers
+  alias HexpmWeb.ViewHelpers
-  <%= pretty_date(last_use.used_at) %> ...
+  <%= ViewHelpers.pretty_date(last_use.used_at) %> ...

etc.

We performed this refactoring on the Hex.pm codebase and that alone removed many compile-time dependencies, reducing the total time it took to recompile the project by 20-30%! We’ve actually wrote a script that automatically performs the import to alias conversion. We’ll talk more about the script in a future blog post, so stay tuned! Meanwhile, you can check out the Hex.pm PR.

However, you may be wondering, should you convert all imports to aliases? That would certainly be a bit too much and we don’t believe that to be necessary. But how can we identify what are the problematic compile-dependencies in our application? Let’s talk about diagnosing.

Diagnosing

Generally speaking, we first notice compilation problems when working on our own projects. For example, you change a single file, then you run mix test, and you suddenly see:

$ mix test
Compiling 27 files (.ex)

Now that we understand that this may be caused by compile-time dependencies, how can we identify and solve those dependencies?

Fortunately, the Elixir team has given us tools to do just that, namely mix xref graph. For example, if you changed lib/foo.ex and that caused a large recompilation, you can run:

$ mix xref graph --sink lib/foo.ex --only-nodes

That will list all files that depend on lib/foo.ex and which kind of dependency. If some file has a compile time dependency on lib/foo.ex, say lib/bar.ex, then you can do the same and see all dependencies on lib/bar.ex:

$ mix xref graph --sink lib/bar.ex --only-nodes

Alternatively, you can remove the --only-nodes flag and see a tree of dependencies on foo, albeit it is often quite large for large projects:

$ mix xref graph --sink lib/foo.ex

Finally, if you are not sure where to get started, you can use mix xref graph --format stats to get general information about the project. For Hex.pm, here is what it would look like:

$ mix xref graph --format stats
Tracked files: 165 (nodes)
Compile dependencies: 402 (edges)
Structs dependencies: 73 (edges)
Runtime dependencies: 429 (edges)

Top 10 files with most outgoing dependencies:
  * lib/hexpm_web/router.ex (42)
  * lib/hexpm/factory.ex (20)
  * lib/hexpm_web/controllers/dashboard/organization_controller.ex (16)
  * lib/hexpm/repository/releases.ex (14)
  * lib/hexpm/repository/package.ex (14)
  * lib/hexpm/accounts/user.ex (14)
  * lib/hexpm/accounts/audit_log.ex (14)
  * lib/hexpm_web/controllers/package_controller.ex (12)
  * lib/hexpm/repository/release.ex (12)
  * lib/hexpm/accounts/users.ex (12)

Top 10 files with most incoming dependencies:
  * lib/hexpm/shared.ex (109)
  * lib/hexpm_web/web.ex (75)
  * lib/hexpm_web/router.ex (43)
  * lib/hexpm_web/views/icons.ex (40)
  * lib/hexpm_web/controllers/controller_helpers.ex (38)
  * lib/hexpm_web/controllers/auth_helpers.ex (37)
  * lib/hexpm_web/endpoint.ex (32)
  * lib/hexpm/accounts/user.ex (31)
  * lib/hexpm/repo.ex (25)
  * lib/hexpm/schema.ex (24)

You can also pass --label compile so it considers only compile-time dependencies when computing the stats.

Tips for Phoenix projects

Luckily, the Phoenix team is also well aware of the issues behind over-relying on compile-time dependencies. For this reason, Phoenix v1.4 eliminated two other sources of re-compilations in new Phoenix apps: router helper imports and plugs. However, if you started your Phoenix application before v1.4, your code may not be up to date on the latest practices. So let’s take a look at them.

Regarding router helper imports, this is the same situation as with our view helpers! All we need to do is perform a similar refactoring:

# web.ex
-  import HexpmWeb.Router.Helpers
+  alias HexpmWeb.Router.Helpers, as: Routes

# lib/hexpm_web/controllers/dashboard_controller.ex
-    redirect(conn, to: dashboard_path(conn, :profile))
+    redirect(conn, to: Routes.dashboard_path(conn, :profile))

Regarding plugs, first just a tiny bit of background. Here’s a sample plug:

defmodule MyPlug do
  def init(opts), do: opts

  def call(conn, opts) do
    # ...
  end
end

The init/1 function, as an optimization, is called at compile-time. This way, any heavy work is only done once, as the project is being compiled, as opposed to on every HTTP request, as is the case with the call/2 function. The consequence of this is very similar to imports vs aliases mentioned above: any module that invokes plug MyPlug now has a compile time dependency on MyPlug and a transitive compile-time dependency on any module invoked by MyPlug, even at runtime. Fortunately, since Phoenix v1.4 we can configure Plug’s behaviour around init, setting:

# config/dev.exs
config :phoenix, :plug_init_mode, :runtime

will ensure init/1 is only called at runtime and so we’ll remove yet another source of possible re-compilations. Remember this is only appropriate for development, don’t set it in production!

Summary

In this article we talked about common source of re-compilations and how we can fix them with simple refactorings, notably by changing imports to aliases. We also teased at a tool that can perform the conversion automatically and we’ll talk more about that in the next post. Stay tuned!