10 years(-ish) of Elixir
Edit Sep/2021: The list of companies using Elixir in production has been updated to add recent successes and new cases.
This past weekend, on January 9th, we celebrated 10 years since the first commit to the Elixir repository. While I personally don’t consider Elixir to be 10 years old yet - the language that became what Elixir is today surfaced only 14 months later - a decade is a mark to celebrate!
The goal of this post is to focus on the current state of some projects in the ecosystem and then briefly highlight a few of the exciting efforts coming over the next months.
Recap: The language goals
When I started working on Elixir, I personally had the ambition of using it for building scalable and robust web applications. However, I didn’t want Elixir to be tied to the web. My goal was to design an extensible language with a diverse ecosystem. Elixir aims to be a general purpose language and allows developers to extend it to new domains.
Given Elixir is built on top of Erlang and Erlang is used for networking and distributed systems, Elixir would naturally be a good fit in those domains too, as long as I didn’t screw things up. The Erlang VM is essential to everything we do in Elixir, which is why compatibility has become a language goal too.
I also wanted the language to be productive, especially by focusing on the tooling. Learning a functional programming language is a new endeavor for most developers. Consequently their first experiences getting started with the language, setting up a new project, searching for documentation, and debugging should go as smoothly as possible.
Extensibility, compatibility, and productivity are the goals we built the language upon.
Recap: Elixir in production
Last year we started a series of articles on companies using Elixir in production on the official website. As of today, we have 7 production cases listed with more coming this year! Overall it is very exciting to see many different companies using a variety of business models and industries running Elixir in production.
Companies like BlockFi, Discord (case), Divvy, Podium, Remote, SalesLoft, and Stord have reached “unicorn status” and rely heavily on Elixir. Startups like Boulevard (podcast), Community (case), Duffel (case), Ockam, Mux (podcast), Ramp, and V7 (case) also use Elixir and have received funding in the last year or two. Elixir is also used within known brands and enterprises such as Bleacher Report, Change.org (case), Heroku (case), Mozilla (case), PagerDuty, PepsiCo (case), StoneCo, and TheRealReal.
There is also a special category of startups that run Elixir alonside an open source model, such as Plausible Analytics, Supabase, Logflare (podcast), and Hex.pm (podcast) itself. Still on the open source front, you will find projects like Mozilla’s Hubs, Pleroma, and Changelog (podcast). There also many small scale and hobby projects that use Elixir for a productive and joyful development experience.
Recap: Diverse ecosystem
Today, Elixir has a diverse ecosystem that works on a wide range of domains and industries. Let’s take a look at some examples.
Web
Most developers are familiar with using Elixir for web development thanks to the Phoenix web framework. Phoenix gained traction in the ecosystem because it was the first to fully leverage the language and the platform for building real-time applications besides the usual MVC (Model-View-Controller) offering.
It all started with Phoenix Channels, as a bi-directional communication between clients and servers, and Phoenix PubSub, which uses Erlang’s distributed compatibilities to broadcast messages across nodes. As far as I know, Phoenix was the first major web framework to provide a multi-node web real-time solution completely out-of-the-box. Regardless if you are using one node or ten nodes, everything just works, with minimal configuration and dependencies.
Phoenix has matured a lot since its first stable release. Phoenix v1.2 included Phoenix Presence, that allows developers to track which users, IoT devices, etc are connected to your cluster right now. No databases or external dependencies required! This is one of the problems that look deceptively simple at first, but once you outline all scalability, performance, and fault-tolerance requirements, it becomes quite complex. Luckily, Phoenix is running on a platform that excels at these problems, and I am not aware of any other framework that provides such a lean and elegant solution as part of its default stack.
Most recently, Phoenix LiveView was released and brought new ways to build rich, real-time user experiences with server-rendered HTML, inspiring developers to attempt similar solutions for other languages and frameworks. You can read the original announcement or learn how to build a real-time Twitter clone in 15 minutes. As part of the Live family, we have also announced Phoenix LiveDashboard, making monitoring and instrumentation a first-class citizen for Phoenix applications.
Embedded and IoT
While I always expected Elixir to shine for building web applications, I was taken by surprise when I heard about the Nerves platform for creating high-end embedded applications. However, once I learned their premise, it all made sense: writing embedded systems is complicated. Reasoning about failures is hard. So what if we could leverage the decades of lessons learnt by Erlang/OTP to design embedded applications? What if a fault on the Wi-Fi driver could be fixed by having a supervisor simply restart it? After all, the first major use of Erlang/OTP was in an embedded system, the Ericsson AXD301 ATM switch.
Nerves brings the Elixir ecosystem and the battle-tested Erlang VM to edge computing, providing a rich developer experience using proven technology. Nerves started as a one step process for turning an Elixir project into a complete software image for common hardware devices. Today, Nerves is being used in production in industrial automation, machine learning, consumer electronics and more, with Farmbot (case) and Rose Point Navigation being two notable examples.
The Nerves team also created NervesHub, a fully open-source device management system. Combining all these technologies makes Elixir a comprehensive language for building end-to-end IoT platforms.
Data ingestion and pipelines
Shortly after Elixir v1.0 was released, the Elixir Core Team and I started looking into abstractions for tackling data ingestions and data pipelines in Elixir. We ran through a couple designs until we eventually landed on GenStage: a behaviour for exchanging data with back-pressure between Elixir processes and external systems. For an introduction, make sure to check out my keynote introducing both GenStage and Flow.
Today, almost 5 years later, GenStage has been used by many industries and has become one of the factors driving Elixir adoption. For example, you can read how both Discord and Change.org have built systems on Elixir and GenStage that handle spikes and run at massive scale.
However, GenStage was just the beginning. In 2019, we announced Broadway, which is a higher-level abstraction on top of GenStage that makes building data ingestion pipelines a breeze. We originally released with Amazon SQS support. Nowadays, RabbitMQ, Google Cloud PubSub, Apache Kafka, and other sources (known as producers in Broadway terms) are also available.
Audio/Video streaming
Since the Erlang VM was designed for scalable network processing, one can expect to also be an excellent platform for audio and video streaming. However, if you also wanted to process and transform those streams on the fly, the situation becomes much more complicated as you likely have to integrate with native code.
Luckily, the tables have turned when Erlang/OTP 20 was released a couple years ago with the so-called Dirty NIFs. The Erlang VM always had the ability to invoke native code, but this native code could not run for long, as to not interfere with the preemptive features of the Erlang runtime. Dirty NIFs allow developers to tag native code either as IO or CPU bound, which runs on specific threads. Between ports (I/O based), NIFs, Dirty NIFs, and remote nodes, developers now have many options to interface with native code with different performance and reliability guarantees. That’s exactly the foundation the Membrane Framework builds on top of.
Membrane was extracted from RadioKit, a startup aiming at disrupting the radio broadcasting industry. Originally it focused on processing and mixing audio. Later, Software Mansion acquired the framework and provided stable funding and a solid team to help it grow into a full-scale framework. Currently, it allows developers to process, transmit, broadcast, and transform audio and videos streams on the fly. Whether you are building a Twitch clone, a VOD application or a video conferencing system, Membrane provides a growing set of high-level abstractions and pre-made modules so you don’t have to dive into idiosyncrasies of particular codecs, protocols, and formats.
Looking ahead: what is coming in 2021
The year of 2021 looks very exciting for the Erlang Ecosystem and the Elixir community. In this section, we are going to mention some of the things we expect to see in 2021.
Erlang/OTP 24 with JIT
In September 2020, Lukas Larsson and the Erlang/OTP team announced a JIT compiler for the Erlang VM called BeamAsm. How faster the JIT will be in practice depends on your application but the results posted in the announcement are promising. To quote Lukas:
If we run the JSON benchmarks found in the Poison or Jason, BeamAsm achieves anything from 30% to 130% increase (average at about 70%) in the number of iterations per second for all Erlang/Elixir implementations. For some benchmarks, BeamAsm is even faster than the pure C implementation jiffy.
More complex applications tend to see a more moderate performance increase, for instance, RabbitMQ is able to handle 30% to 50% more messages per second depending on the scenario.
I have been running Erlang/OTP master since the JIT pull request has been merged. I am also interested in the benefits the JIT brings to the developer experience and I must say the improvements are clear: code compilation and test suites run distinctively faster (around 30% to 50% in my case) and that’s quite promising!
My understanding is that there is more to explore when it comes to JIT but the benefits so far are already substantial beyond micro-benchmarks, bringing measurable benefits to end-users.
Web
On the web front, we should soon see the release of Phoenix v1.6, where one of the major features is the addition of the mix phx.gen.auth code generator that sketches out an authentication solution with registration, confirmation, password recovery, and more. These improvements to the getting started workflow alongside the metrics and dashboards added in v1.5 put Phoenix in a unique position to provide a great and complete developer experience from development to production, with a scalable runtime to back it up.
We will most likely see Phoenix LiveView get the 1.0 stamp this year too, with a refined template syntax and exciting component features. While many teams and companies have adopted and leveraged LiveView to build great user experiences, it is understandable that some are waiting for a stable release to jump in with both feet. Stability also means more learning resources, books, courses, etc. All of those will lead to more growth.
Phoenix LiveView will also lead the ecosystem to more visual tools. We have already talked about the Phoenix LiveDashboard but I expect to see more tools in this area soon, such as Surface, Oban Pro, and the soon to be released Broadway dashboard showcased by our own Marlus Saraiva at ElixirConf.
Data and multimedia: WebRTC and more
One of the major features the Membrane team is working on is WebRTC support. Until now the framework was capable of processing streams delivered to it over numerous protocols but not from the web browser. The combination of Membrane and Phoenix can become a powerful addition to the ecosystem, allowing developers to add a multimedia component to their real-time applications, all directly from Elixir.
The Dashbit team also hopes to release Broadway v1.0 this year. The biggest feature we are working on is support for network based producers, allowing developers to create HTTP endpoints or implement custom TCP/UDP protocols, such as Fluentd or Logstash formats, which feed directly into their Broadway pipeline.
Get involved!
If you want to participate, you should definitely consider getting involved with the many projects and efforts happening in the community. Note the list above is not comprehensive and there is more exciting work happening in different areas.
If you are just learning or want to learn Elixir, the website is a good starting point, check out the guides for a fast-paced introduction or our learning resources page with many resources for different levels of your learning curve.
Finally, we at Dashbit continue exploring new domains and areas to bring Elixir into. Last month we announced a research Master of Science project sponsored by Dashbit into eBPF by the Compilers Lab in the Federal University of Minas Gerais, Brazil.
We have also been really hard at work over the last 2 months or so on a project called Nx and a set of auxiliary tools that have the potential to bring Elixir to a whole new domain and open up the language to areas that were not explored in depth before! I have shared some early benchmarks and I will be officially presenting these projects this February on Lambda Days 2021. Come join us and stay tuned! Edit: Nx is now publicly available.