Building custom Hex repositories

• Wojtek Mach
• January 19th, 2021
• elixir, hex

Elixir developers have a few options when it comes to using private packages. What you’ll end up choosing usually comes down to whether you want (and/or need!) to fully own the infrastructure. If you don’t, Hex.pm has a hosted private packages offering and it’s a great option for many organizations. However, if you need full control, you may look into community projects like MiniRepo (full disclosure: I’m the author and as of today it’s deprecated and you’ll soon learn why) or the more recent urepo. Finally, while working on Bytepack, we’ve implemented another custom repository that you can check out. As a matter of fact, it was exactly while working on Bytepack we realized that a large number of users can benefit from a much simpler solution. We think it strikes a nice balance of covering common use cases while also having an implementation that is simple and can be easily customized for your exact needs. We are very glad to have contributed it to Hex itself and that’s exactly what we’ll cover today.

As of Hex v0.21, we can create a local registry with mix hex.registry build, let’s see how we can use it.

A quick aside: we’ve used the words “repository” and “registry”, what do we mean by them? In a nutshell, a Hex registry is a collection of resources that describe the packages and their relationships and allows for efficient dependency resolution. A Hex repository is basically a Hex registry + actual package tarballs hosting.

The mix hex.registry build task requires three things:

• the name of the registry
• a directory to hold public files
• a private key used to sign the registry

Let’s create an “acme” directory for our repository, generate a random private key, a public directory, and finally let’s build the registry resources:

$ mkdir acme
$ cd acme
$ openssl genrsa -out private_key.pem
$ mkdir public
$ mix hex.registry build public --name=acme --private-key=private_key.pem
* creating public/public_key
* creating public/tarballs
* creating public/names
* creating public/versions

and that’s it! Now, all we need to do is start a HTTP server that exposes the public directory and we can point Hex clients to it. However, let’s add a package to our repository first.

To publish a package you need to copy the tarball to public/tarballs and re-build the registry. You can build your own package (using mix hex.build) or simply use an existing one. Let’s do the latter, we can easily fetch a package with the mix hex.package fetch task:

$ mix hex.package fetch decimal 2.0.0
decimal v2.0.0 downloaded to decimal-2.0.0.tar
$ cp decimal-2.0.0.tar public/tarballs/
$ mix hex.registry build public --name=acme --private-key=private_key.pem
* creating public/packages/decimal
* updating public/names
* updating public/versions

Now let’s test our repository, all we need to do is expose our public/ directory via http:

$ python3 -m http.server 8000 --directory=public/
Serving HTTP on :: port 8000 (http://[::]:8000/) ...

And let’s now add the repository and try fetching the package that we just published:

$ mix hex.repo add acme http://localhost:8000 --public-key=public/public_key
$ mix hex.package fetch decimal 2.0.0 --repo=acme
decimal v2.0.0 downloaded to decimal-2.0.0.tar

it worked!

Here’s how you’d use the package from your custom repository in your project, add this to mix.exs:

defp deps() do
  {:decimal, "~> 2.0", repo: "acme"}
end

and run mix deps.get.

Let’s briefly talk about deploying your custom repository solution to production.

Deploying to S3

Deploying to Amazon S3 (or similar cloud services) is probably the easiest way to have a reliable Hex repository.

If you already have an S3 bucket, you can use AWS CLI to sync the contents of the public/ directory like this:

$ aws s3 sync public s3://my-bucket

Warning: Remember to sync only the public directory and not private_key.pem! And if you do want to sync your private key, remember to set appropriate bucket policy so it isn’t accidentally exposed.

Your repository should now be available under an URL like: https://<bucket>.s3.<region>.amazonaws.com or however you configured your bucket.

See “Deploying to S3” on the new Hex.pm self-hosting guide for more information.

Deploying with Plug.Cowboy

If you need any customizations to your Hex server, you may consider creating a proper Elixir project. Since we’re basically just hosting static files, Plug & Plug.Cowboy is more than enough:

Step 1: Create a new project with $ mix new my_app --sup

Step 2: Add dependencies

defp deps do
  [
    {:plug, "~> 1.11"},
    {:plug_cowboy, "~> 2.4"}
  ]
end

Step 3: Update your supervision tree to start Cowboy

# lib/my_app/application.ex

def start(_type, _args) do
  port = 4000
  
  children = [
    {Plug.Cowboy, scheme: :http, plug: MyApp.Plug, options: [port: port]}
  ]
  
  opts = [strategy: :one_for_one, name: MyApp.Supervisor]
  Supervisor.start_link(children, opts)
end

Step 4: Add MyApp.Plug

# lib/my_app/plug.ex

defmodule MyApp.Plug do
  use Plug.Builder

  plug Plug.Logger
  plug Plug.Static, at: "/", from: "/path/to/repo/public"
  plug :not_found

  defp not_found(conn, _opts) do
    send_resp(conn, 404, "not found")
  end
end

And that should be it!

See “Deploying with Plug.Cowboy & Docker” on the new Hex.pm self-hosting guide for more information. In particular, you’ll learn how to add HTTP Basic authentication, use Elixir releases, configure your application with environment variables, and prepare for Docker deployment.

Conclusion

In this article we’ve introduced the mix hex.registry build task that allows you quickly building a local registry. We’ve also touched on deploying your custom solution to Amazon S3 or rolling your own with Plug. Definitely check out Hex.pm self-hosting guide for a more comprehensive reference.

Happy hacking!