Rewriting imports to aliases with compilation tracers

• Wojtek Mach
• April 7th, 2020
• compiler, eex, elixir

This article will explore how we used compilation tracers to implement a tool that automatically rewrote imports to aliases on the Hex.pm codebase, called import2alias.

For example, to replace HexpmWeb.ViewHelpers imported calls with ViewHelpers, we used the script like this:

cd /path/to/hexpm
mkdir -p lib/mix/tasks

curl https://gist.githubusercontent.com/wojtekmach/4e04cbda82ba88af3f84c44ec746b7ca/raw/import2alias.ex > lib/mix/tasks/import2alias.ex

curl https://gist.githubusercontent.com/wojtekmach/4e04cbda82ba88af3f84c44ec746b7ca/raw/lib_import2alias.ex > lib_import2alias.ex

elixir -r lib_import2alias.ex -S mix import2alias HexpmWeb.ViewHelpers ViewHelpers

As you can see, the script is actually quite tiny! In this blog post we’ll look under the hood and discuss some other improvements we’ve recently made. Let’s get started.

UPDATE #1: Thanks to feedback from @kleinernik, we’ve changed the script to a Mix task to avoid warnings on protocol consolidation.

UPDATE #2: Elixir v1.11+ will no longer consider imports as compile-time dependencies. Therefore converting imports to aliases is no longer strictly necessary for improving recompilation times. This article, however, can still be useful for those interested in converting imports to aliases for code readability reasons or for those willing to learn more about compilation tracers.

Compilation tracers

import2alias is built on top of compilation tracers, a feature introduced in Elixir v1.10. Per Elixir Code documentation:

A tracer is a module that implements the trace/2 function. The function receives the event name as first argument and Macro.Env as second and it must return :ok.

And here are some example events:

• {:import, meta, module, opts} - traced whenever module is imported. meta is the import AST metadata and opts are the import options.

• {:imported_function, meta, module, name, arity} and {:imported_macro, meta, module, name, arity} - traced whenever an imported function or macro is invoked. (…)

• {:local_function, meta, name, arity} and {:local_macro, meta, name, arity} - traced whenever a local function or macro is referenced. (…)

etc.

Here’s the tracer we wrote for our import2alias script:

defmodule Import2Alias.CallerTracer do
  def trace({:imported_function, meta, module, name, arity}, env) do
    Import2Alias.Server.record(env.file, meta[:line], meta[:column], module, name, arity)
    :ok
  end

  def trace(_event, _env) do
    :ok
  end
end

We are only interested in :imported_function events, we record file/line/column and module/name/arity for further processing and ignore remaining events.

We could do the processing in the trace/2 function directly but the recommendation is to do there as least work as possible because it slows down the compilation. Thus, we save the work for further processing. Import2Alias.Server is an Agent that filters imported calls and groups them by source filenames. This way we’d rewrite any given source file just once:

for {file, entries} <- entries do
  lines = File.read!(file) |> String.split("\n")

  lines =
    Enum.reduce(entries, lines, fn entry, acc ->
      {line, column, module, name, arity} = entry

      List.update_at(acc, line - 1, fn string ->
        # ...
      end)
    end)

  File.write!(file, Enum.join(lines, "\n"))
end

If we have the column information, we rewrite the line becasue we know exactly where the imported call started and we rewrite it to be an aliased call.

if column do
  pre = String.slice(string, 0, column - 1)
  offset = column - 1 + String.length("#{name}")
  post = String.slice(string, offset, String.length(string))
  pre <> "#{inspect(alias)}.#{name}" <> post
else
  # print warning
end

and this results in e.g.:

-  <%= pretty_date(last_use.used_at) %> ...
+  <%= ViewHelpers.pretty_date(last_use.used_at) %> ...

and that’s it!

However, you may have noticed that we explicitly checked if the column information is available. Why wouldn’t we have the column information? This brings us to…

Other compiler improvements

To get precise information where a function is called, not only at which line but also at which column, we’ve set this compile option:

Code.put_compile_option(:parser_options, [columns: true])

This worked fine in .ex files but not in .eex files, the EEx engine uses its own compiler. We’ve changed EEx.Compiler to properly track column information and use that in error messages.

EEx templates can also be directly embedded in Elixir modules, such as using Phoenix ~E or Phoenix LiveView ~L sigils:

defmodule AppWeb.ThermostatLive do
  use Phoenix.LiveView

  def render(assigns) do
    ~L"""
    Current temperature: <%= pretty_temperature @temperature %>
    """
  end
end

To handle that, we’ve changed Elixir compiler to track indentation of heredoc blocks and used that in EEx, Phoenix.HTML ~E, and Phoenix.LiveView ~L.

To take advantage of these improvements you need to wait for Elixir v1.11 or use a version manager, such as asdf install elixir master, to get the latest.

These compiler changes, besides making import2alias more useful, should give more capabilities to existing and future tooling and allow more accurate stacktraces, editor integrations, and more. Perhaps that is the biggest win from all of this recent work after all!

Summary

In this article we’ve looked at the import2alias script, how it was built on top of compilation tracers, and about some of our recent compiler changes that made that more reliable. We are looking forward to hearing what you’ve built with compilation tracers, happy hacking!