SDKs with Req: Stripe

• Wojtek Mach
• June 25th, 2024
• req, http client

In this blog post, we discuss Software Development Kits and propose an alternative interpretation using Stripe as an example.

Software Development Kits

SDKs, or software development kits, are a bit hard to describe. There is a couple of different definitions I found. The way I think about them is they are a set of APIs and tools to interact with a, usually large, platform. Here are some examples and popular Elixir projects targeting them:

• GitHub: Octokit, tentacat

• Stripe: Stripe SDKs, stripity_stripe

• AWS: AWS Tools, ex_aws*, aws

• Google: Google API Client Libraries, google_api*

These projects tend to be pretty huge. If all APIs are in a single package it can have many 10s or 100s of modules (aws, stripity_stripe). Sometimes the APIs are split into multiple packages (ex_aws*, google_api*).

The APIs are either hand-crafted (ex_aws*, stripity_stripe prior to v3.0) or automatically generated off of shared API description. The sheer size and, perhaps more importantly, rate of change of these APIs means that automatically generating them (“codegen”) is the only sensible way to go about them since otherwise you’d always play catch up.

Btw, what do I mean by APIs exactly? In Elixir these would be modules and functions. Let’s take a look at an example function from stripity_stripe v3.1.1, a function to create a checkout session:

@doc "<p>Creates a Session object.</p>\n\n#### Details\n\n * Method: `post`\n * Path: `/v1/checkout/sessions`\n"
@spec create(
        params :: %{
          # ...,
          optional(:billing_address_collection) => :auto | :required,
          # ...,
          optional(:customer_email) => binary,
          # ...,
          optional(:line_items) => list(line_items),
          # ...,
          optional(:mode) => :payment | :setup | :subscription,
          # ...,
          optional(:success_url) => binary,
          # ...,
          optional(:tax_id_collection) => tax_id_collection,
          # ...
          ...
        },
        opts :: Keyword.t()
      ) ::
        {:ok, Stripe.Checkout.Session.t()}
        | {:error, Stripe.ApiErrors.t()}
        | {:error, term()}
def create(params \\ %{}, opts \\ [])

(See Stripe.Checkout.Session.create/2.)

This is really cool. We automatically have a module/function for each object/method, some basic docs, and even typespecs speciyfing allowed values. The latter can be leveraged by LSPs to provide helpful suggestions and similar static analysis tools to catch mistakes. More generally we can use features like IEx/ExDoc autocomplete to discover things. In some SDKs the auto-generated functions even validate the inputs, catching mistakes even sooner, even before making the network request. That’s one of the problems with these SDKs though, they are as good as the generated code. If we want to use a bleeding edge functionality just added to the platform, well, that might be unavailable until the SDK is up-to-date with the platform!

I think auto-generated docs are a nice touch but I found myself reaching to online API documentation anyway as it would often contain extra context, guides, examples, executable shells etc. The auto-generated docs may also be generic, because they are meant to work across several programming languages.

Another problem is bloat. Often times we need just a handful of functions and so it is pretty wasteful to compile and load into the memory all of that code that we’re never going to use. (This is of course true of using a subset of any dependencies but I believe due to the sheer size of these SDKs it’s a distinct problem.)

Finally, and to me this is by far the biggest problem, what do we do when things go wrong. Due to their nature these SDKs tend to have pretty abstract internals because they need to, you know, support every little API the platform exposes including all of the legacy baggage. And so in my experience they tend to be pretty hard to debug. And even if we fix the issue ourselves and send the patches upstream, what if the maintainers are unresponsive? Do we want to be stuck with maintaining such a large fork?

That being said, there’s one other benefit of SDKs worth mentioning and that is authentication. Depending on the platform, we may need just a simple bearer token (Stripe, GitHub when using Personal Access Token) or something way more complicated. SDKs tend to have great built-in support for these.

I believe SDKs have unquestionable benefits like IDE support, discoverability of (vast) platform features and authentication. I’m glad they exist and grateful to the authors and contributors. As anything non-trivial they have tradeoffs though and I’d like to propose a different set. Instead of SDKs as described above, I’d like to propose “Small Development Kits”.

Small Development Kits: Stripe

Small Development Kits are, well, small. Modest. They don’t try to support every possible feature but just what you need. Here’s a Stripe SDK that we use at Dashbit:

# lib/teams/stripe.ex
defmodule Teams.Stripe do
  def new(options \\ []) when is_list(options) do
    Req.new(
      base_url: "https://api.stripe.com/v1",
      auth: {:bearer, Application.fetch_env!(:teams, :stripe_api_key)}
    )
    |> Req.Request.append_request_steps(
      post: fn req ->
        with %{method: :get, body: <<_::binary>>} <- req do
          %{req | method: :post}
        end
      end
    )
    |> Req.merge(options)
  end

  def request(url, options \\ []), do: Req.request(new(url: parse_url(url)), options)

  def request!(url, options \\ []), do: Req.request!(new(url: parse_url(url)), options)

  defp parse_url("product_" <> _ = id), do: "/products/#{id}"
  defp parse_url("price_" <> _ = id), do: "/prices/#{id}"
  defp parse_url("sub_" <> _ = id), do: "/subscriptions/#{id}"
  defp parse_url("cus_" <> _ = id), do: "/customers/#{id}"
  defp parse_url("cs_" <> _ = id), do: "/checkout/sessions/#{id}"
  defp parse_url("inv_" <> _ = id), do: "/invoices/#{id}"
  defp parse_url("evt_" <> _ = id), do: "/events/#{id}"
  defp parse_url(url) when is_binary(url), do: url
end

That’s pretty much it! Here is an example request to retrieve a customer:

iex> Teams.Stripe.request!("/customers/cus_PbmIREVnJJoq8p").body["name"]
"Alice"

Or simply (note the defp parse_url above)

iex> Teams.Stripe.request!("cus_PbmIREVnJJoq8p").body["name"]
"Alice"

(It’s really convenient that all Stripe object IDs have distinct prefixes!)

Updating a customer object is easy too. Our custom request step automatically sets HTTP method to POST when there is request body:

iex> Teams.Stripe.request!("cus_PbmIREVnJJoq8p", form: [name: "Bob"])

The Teams.Stripe module is tiny and yet very flexible. It sets some default options (:base_url and :auth) and a custom step. Thanks to Req design you can override any option anytime. For example, want to try some things with different API key? And, say, different retry policy? No problem, just pass a different :auth and :retry values at the call site:

Teams.Stripe.request!(url, auth: {:bearer, other_token}, retry: false)

When it comes to testing, since you have full control, you can choose any strategy that is the most appropriate for the problem at hand. In “Req API Client Testing” I list a few different options. Testing may not be otherwise as straightfoward with “Big SDKs”, where you depend on implementation details or testing affordances they may or may not provide.

Bonus Points: Stripe Webhook Listener

A critical part of integrating with Stripe is processing webhooks (and veryfing them!) Stripe CLI can listen to them, forward them, and even trigger them:

$ stripe listen --forward-to http://localhost:4000/webhooks/stripe

# and in another session:
$ stripe trigger customer.created

This is really helpful.

Instead of having our developers remember to run the listener, we simply start it in our application supervision tree in development:

@impl true
def start(_type, _args) do
  children =
    [
      # ...
      TeamsWeb.Endpoint
    ] ++ stripe_webhook_listener_specs()

  Supervisor.start_link(children, strategy: :one_for_one, name: Teams.Supervisor)
end

defp stripe_webhook_listener_specs do
  if Application.get_env(:teams, :dev_routes, false) and
       Phoenix.Endpoint.server?(:teams, TeamsWeb.Endpoint) do
    [{Teams.Stripe.WebhookListener, ...}]
  else
    []
  end
end

The listener itself is a simple GenServer managing a port:

defmodule Teams.Stripe.WebhookListener do
  use GenServer
  require Logger

  def start_link(options) do
    {stripe_cli, options} = Keyword.pop(options, :stripe_cli, System.find_executable("stripe"))
    {forward_to, options} = Keyword.pop!(options, :forward_to)
    options = Keyword.validate!(options, [:name, :timeout, :debug, :spawn_opt, :hibernate_after])
    GenServer.start_link(__MODULE__, %{stripe_cli: stripe_cli, forward_to: forward_to}, options)
  end

  @impl true
  def init(%{stripe_cli: nil}) do
    Logger.warning("""
    Stripe CLI not found

    Run:
        brew install stripe/stripe-cli/stripe
    """)

    :ignore
  end

  def init(%{stripe: stripe_cli, forward_to: forward_to}) do
    args = [
      "listen",
      "--skip-update",
      "--color",
      "--forward-to",
      forward_to
    ]

    port =
      Port.open(
        {:spawn_executable, stripe_cli},
        [
          :binary,
          :stderr_to_stdout,
          line: 2048,
          args: args
        ]
      )

    {:ok, port}
  end

  @impl true
  def handle_info({port, {:data, {:eol, line}}}, port) do
    IO.puts(["stripe: ", line])
    {:noreply, port}
  end
end

Here’s to SDKs being APIs and tools!

Conclusion

In this article I presented an alternative to SDKs. Instead of having tens, hundreds if not thousands of modules, we implement ourselves just what we need. Such leaner codebase tends to be easier to maintain. We don’t depend on someone else’s code and their release schedule, we have full control. In future articles I’ll show some other examples of Small Development Kits. Stay tuned!