Introducing Rustler Precompiled

• Philip Sampaio
• March 15th, 2022
• rustler, rust, precompilation, explorer, nerves

Rust is becoming widely used for systems programming. Because of its safety guarantees and performance, it is a good language for writing NIFs to the BEAM ecosystem, and it’s fairly easy to write them using the awesome Rustler project.

Rustler was created a few years ago by Hans Elias J., and it’s a project that aims to be a bridge between Rust and Elixir/Erlang. It makes really easy to develop packages - there are around 90 of them using Rustler on Hex.pm while I’m writting this - but there are some challenges with actually using them. First of all, Rustler-based packages require Rust toolchain to be installed to compile native code. Secondly, we need to actually compile the native code which for some projects can be really time and resources consuming.

This is where Rustler Precompiled comes in.

Rustler Precompiled is a project that enables the usage of precompiled NIFs. Precompiled NIFs are then downloaded from the internet and validated using checksums. This way we can precompile our Rustler projects in the CI and download them in the user machine securely. This can bring a huge benefit in compilation time to several projects. Since no Rust code is compiled, the only requirement is an internet connection for downloading and compiling your dependencies. And if you’d rather always build from scratch, due to security concerns, you can always bypass RustlerPrecompiled and force a local build.

For example, the html5ever package takes 22 seconds to compile, but only 2 seconds to download if precompiled. The difference is bigger for larger projects like the Explorer: it takes 2 min and 29 seconds to compile the project from scratch and only 3.3 seconds when using a precompiled version. The tests were made using my Dell XPS with an Intel® Core™ i7-1065G7 CPU (4 cores and 8 threads) and the time command with dependencies already compiled (mix deps.compile before).

Where the most of the work happens

Almost the entire work happens in the CI server, where the NIF project in your library will be compiled to several targets. A “target” is combination of NIF version, operating system, CPU architecture, the vendor or manufacturer, and sometimes the ABI - usually describing the tool used to compile that software.

To build your NIF you will need a special tool named cross. This is a great tool that reduces the setup needed for “cross-compiling” to different targets. In the background cross will try to use the default cross-compilation abilities from Rust, and when that is not possible, it will run a Docker container compiling for that given target.

In the end the build matrix of your project will look like this:

matrix:
  job:
    # NIF version 2.16
    - { target: arm-unknown-linux-gnueabihf , os: ubuntu-20.04 , nif: "2.16", use-cross: true }
    - { target: aarch64-unknown-linux-gnu   , os: ubuntu-20.04 , nif: "2.16", use-cross: true }
    - { target: aarch64-apple-darwin        , os: macos-10.15  , nif: "2.16" }
    - { target: x86_64-apple-darwin         , os: macos-10.15  , nif: "2.16" }
    - { target: x86_64-unknown-linux-gnu    , os: ubuntu-20.04 , nif: "2.16" }
    - { target: x86_64-unknown-linux-musl   , os: ubuntu-20.04 , nif: "2.16", use-cross: true }
    - { target: x86_64-pc-windows-gnu       , os: windows-2019 , nif: "2.16" }
    - { target: x86_64-pc-windows-msvc      , os: windows-2019 , nif: "2.16" }
    # NIF version 2.15
    - { target: arm-unknown-linux-gnueabihf , os: ubuntu-20.04 , nif: "2.15", use-cross: true }
    - { target: aarch64-unknown-linux-gnu   , os: ubuntu-20.04 , nif: "2.15", use-cross: true }
    - { target: aarch64-apple-darwin        , os: macos-10.15  , nif: "2.15" }
    - { target: x86_64-apple-darwin         , os: macos-10.15  , nif: "2.15" }
    - { target: x86_64-unknown-linux-gnu    , os: ubuntu-20.04 , nif: "2.15" }
    - { target: x86_64-unknown-linux-musl   , os: ubuntu-20.04 , nif: "2.15", use-cross: true }
    - { target: x86_64-pc-windows-gnu       , os: windows-2019 , nif: "2.15" }
    - { target: x86_64-pc-windows-msvc      , os: windows-2019 , nif: "2.15" }

In the example, I’m assuming you are using GitHub Actions. Notice that we have 8 targets for each NIF version, which will produce 16 lib binaries.

The workflow

“When this is going to build”, you may be asking. It is going to build on each tag pushed, just right before the publishing of your package to Hex.pm. This is really important, since we need to have the files ready to generate the checksum file. This file is meant to guarantee that no one replaces a published NIF with a malicious version.

In order to check for the integrity of the NIF, the checksum file will be published with your package. It is not necessary to do the versioning of this file, though.

Rustler Precompiled has a specific audience: package developers. So I wrote a detailed guide for publishing packages in the Rustler Precompiled docs.

The NIF module

Like with Rustler, the NIF module is usually simple. Taking our example app, it looks like this:

defmodule RustlerPrecompilationExample.Native do
  version = Mix.Project.config()[:version]

  use RustlerPrecompiled,
    otp_app: :rustler_precompilation_example,
    crate: "example",
    base_url:
      "https://github.com/philss/rustler_precompilation_example/releases/download/v#{version}",
    force_build: System.get_env("RUSTLER_PRECOMPILATION_EXAMPLE_BUILD") in ["1", "true"],
    version: version

  # When your NIF is loaded, it will replace this function.
  def add(_a, _b), do: :erlang.nif_error(:nif_not_loaded)
end

This is similar to Rustler API, but with addition of three important options:

• :base_url - the place where the NIF will be downloaded from. In this example it uses the GitHub releases schema. If using the version 0.3.0 it is going to download the files from the release list.

• :force_build - a way to force the build by falling back to Rustler. In this case we are reading the environment variable RUSTLER_PRECOMPILATION_EXAMPLE_BUILD. There is also an application environment that can be set, so you don’t need to configure in the module:

  config :rustler_precompiled, :force_build, your_otp_app: true

  Another important thing to mention is that development pre release versions are always forced to compile. If you have a 0.1.0-dev version the project will always fallback to Rustler.

• :version - finally we need to specify the version of the package in use. This version is needed both for the file name resolution and to define if it’s a pre-release.

Conclusion

Now we can safely use precompiled NIFs written with Rustler in our packages. This can increase the adoption of tools like the Explorer that uses Polars underneath.

It makes the publishing of packages harder, but the usage of those packages is much easier since it won’t require the dependencies needed for Rust code compilation.

There is also a bonus: Rustler Precompiled is prepared for Nerves projects! It was tested in Raspberry Pi machines thanks to Frank Hunleth of the Nerves core team.

Finally, I want to thank Hans Elias J., Magnus and Benedikt Reinartz of the Rustler core team for the support and code reviews. And also, thank José Valim for the guidance and code reviews.

Happy coding!