What libring actually does

libring is a consistent hashing library. Given a set of nodes and a key, it answers exactly one question: which node is responsible for this key?

ring =
  HashRing.new()
  |> HashRing.add_nodes([:node1, :node2, :node3])

HashRing.key_to_node(ring, "USD/EUR")
# => :node2

That's the entire interface. libring doesn't store anything. It doesn't talk to the network. It doesn't know or care whether the node it names is alive, whether that node has any data for the key, or whether every other node in the cluster would give you the same answer right now. It's a pure function. Same ring, same key, same answer, every time, computed locally with zero coordination.

The reason this matters more than a plain hash(key) rem length(nodes) is what happens when the node list changes. With modulo hashing, adding a single node reshuffles almost every key's owner. A routine deploy becomes a mass, unjustified reshuffle. Consistent hashing hashes both nodes and keys onto the same ring; adding a node only takes over the keys between it and its neighbour on the ring. Everyone else's ownership is untouched.

What delta_crdt actually does

delta_crdt solves a completely different problem: how do multiple copies of the same data agree on a value, without a coordinator, without a leader, and without blocking on a network round trip.

Every node runs its own local delta_crdt process holding a full copy of the state. Writes happen locally, immediately, with no need to ask permission from anyone else. Periodically, each node gossips its recent changes to its configured neighbours, and the CRDT's merge function combines them, so every node converges on the same final state regardless of the order updates arrive in.

{:ok, crdt} = DeltaCrdt.start_link(DeltaCrdt.AWLWWMap, on_diffs: &handle_diff/1)

DeltaCrdt.set_neighbours(crdt, [neighbour_pid_1, neighbour_pid_2])

DeltaCrdt.put(crdt, "USD/EUR", 1.08)

on_diffs fires locally whenever new deltas arrive from a neighbour. This is how you'd mirror the CRDT's state into something faster to read, like ETS.

The important property: delta_crdt doesn't need a single owner per key. Any node can write any key at any time, and the merge guarantees convergence. That's what "conflict-free" means in the name, commutative, associative, idempotent merge. So no matter how updates arrive, the result is the same everywhere.

Why you need both

If delta_crdt already handles writes from anywhere without a designated owner, it's fair to ask why libring is involved at all. Two reasons, both about scale rather than correctness.

Avoiding "everyone stores everything." A single shared CRDT map, gossiped to every node, means every node holds a full copy of every key. Fine for a small exchange-rate cache. Not fine for a large keyspace across a big cluster. libring lets you shard. Run multiple CRDT instances, each covering a subset of keys, with the ring deciding which node hosts which shard.

Fewer concurrent writers means fewer conflicts to resolve. Even within a single CRDT, if every node writes every key at any time, you get constant concurrent writes to the same keys. For a last-write-wins map, that means constant silent overwrites. Routing writes through the ring's designated owner most of the time means conflicts become the rare exception, only during partitions or membership drift, rather than the normal case.

So the honest division: libring decides who is allowed to write. delta_crdt decides what survives if that decision was ever ambiguous, and makes every node able to read the result locally regardless of who wrote it.

A working example

A ring-routed, distributed key-value store, storing the ring in :persistent_term for fast, lock-free reads on every request.

defmodule CrdtDemo.Topology do
  @hash_ring :ring

  def owner(key) do
    :persistent_term.get(@hash_ring)
    |> HashRing.key_to_node(key)
  end

  def add_nodes_to_ring(nodes) do
    ring =
      HashRing.new()
      |> HashRing.add_nodes([node() | nodes])

    :persistent_term.put(@hash_ring, ring)
  end
end

:persistent_term is the right fit here specifically because writes are rare (only on membership change) and reads happen on every request. Writes are expensive because they trigger a scan across every process in the VM. But that cost is paid only when the cluster topology actually changes, not on every lookup.

Membership changes are watched with :net_kernel.monitor_nodes/1, and both the ring and the CRDT's gossip neighbours are kept current together:

defmodule CrdtDemo.Ring do
  use GenServer

  def start_link(_) do
    GenServer.start_link(__MODULE__, [], name: __MODULE__)
  end

  def init(_) do
    :net_kernel.monitor_nodes(true)
    sync()
    {:ok, []}
  end

  def handle_info({:nodeup, _}, state) do
    sync()
    {:noreply, state}
  end

  def handle_info({:nodedown, _}, state) do
    sync()
    {:noreply, state}
  end

  defp sync() do
    nodes = Node.list()
    CrdtDemo.Topology.add_nodes_to_ring(nodes)
    CrdtDemo.Store.set_neighbours(nodes)
  end
end

The store wraps a delta_crdt instance, mirrors its state into ETS for fast local reads, and routes writes to whichever node the ring names as owner:

defmodule CrdtDemo.Store do
  use GenServer

  @table :store_mirror
  @crdt_pid_key :store_crdt_pid

  def start_link(_), do: GenServer.start_link(__MODULE__, [], name: __MODULE__)

  def get_crdt_pid, do: :persistent_term.get(@crdt_pid_key)

  def set_neighbours(nodes) do
    results = :erpc.multicall(nodes, __MODULE__, :get_crdt_pid, [])
    crdt_pids =
        Enum.reduce(results, [], fn
          {:ok, pid}, acc -> [pid | acc]
          _error, acc -> acc
        end)

    DeltaCrdt.set_neighbours(get_crdt_pid(), crdt_pids)
  end

  def put(key, value) do
    case CrdtDemo.Topology.owner(key) do
      node when node == node() -> update_crdt(key, value)
      other_node -> :erpc.call(other_node, __MODULE__, :put, [key, value])
    end
  end

  def update_crdt(key, value) do
    get_crdt_pid() |> DeltaCrdt.put(key, value)
  end

  def get(key) do
    case :ets.lookup(@table, key) do
      [{^key, value}] -> value
      [] -> nil
    end
  end

  def init(_) do
    {:ok, crdt} = DeltaCrdt.start_link(DeltaCrdt.AWLWWMap, on_diffs: &handle_diff/1)
    :persistent_term.put(@crdt_pid_key, crdt)
    :ets.new(@table, [:named_table, :public, read_concurrency: true])
    {:ok, %{crdt: crdt}}
  end

  defp handle_diff(diffs) do
    Enum.each(diffs, fn
      {:add, key, value} -> :ets.insert(@table, {key, value})
      {:remove, key} -> :ets.delete(@table, key)
    end)
  end
end

Notice get/1 never consults the ring at all. It reads whatever's in this node's own ETS table. That works because handle_diff/1 runs on every node as gossip arrives, continuously mirroring the CRDT's state into local ETS. By the time you call get/1 anywhere, the value is very likely already sitting in local memory. The ring decides where writes are accepted. It has nothing to do with where reads are served.

Trying it yourself

Both Ring and Store are added as children in application.ex, so they start automatically on boot:

def start(_type, _args) do
  children = [
    CrdtDemo.Store,
    CrdtDemo.Ring
  ]

  Supervisor.start_link(children, strategy: :one_for_one, name: CrdtDemo.Supervisor)
end

Store before Ring matters. Ring.init/1 calls sync/0 immediately, which calls Store.set_neighbours/1. Store needs to exist first.

Start three named nodes locally:

iex --sname node1 --cookie demo -S mix
iex --sname node2 --cookie demo -S mix
iex --sname node3 --cookie demo -S mix

Connect them from any one node.
The example below is from node 1.

Node.connect(:"node2@yourhostname")
Node.connect(:"node3@yourhostname")

Each node's Ring process gets a :nodeup message as the others join, rebuilds the ring, and re-establishes CRDT neighbours. Write a value from one node:

CrdtDemo.Store.put("USD/EUR", 1.08)

Read it back from a different node entirely:

CrdtDemo.Store.get("USD/EUR")
# => 1.08

Nothing was asked of the writing node to get that answer. The value arrived via gossip and is sitting in this node's own ETS table.

Kill one node (Ctrl+C twice, or System.stop()) and the remaining two get :nodedown, rebuild their ring without it, and keep working. Bring the node back up and reconnect it. Its Ring.init/1 runs again, sync/0 fires, and it picks up the current ring and re-establishes neighbours. Read the same key from the node that just rejoined:

CrdtDemo.Store.get("USD/EUR")
# => 1.08

It has the value again, without you doing anything to restore it. Gossip caught it up once neighbours were re-established. That's the whole point made concrete: the node's own local state was gone the moment it went down, and it came back purely by rejoining the mesh.

The one thing worth remembering

libring's ownership answer is computed from each node's own local view of cluster membership Node.list() at the moment sync/0 runs. That view isn't guaranteed to be identical across every node at every instant, especially during a deploy or a brief network hiccup. libring doesn't coordinate that away, and it isn't supposed to.

That's exactly the gap delta_crdt exists to close. If two nodes briefly hold different rings and both accept a write for the same key, both writes survive as deltas and merge deterministically once gossip catches up. The ring is the fast, common-case routing decision. The CRDT is what makes it safe for that decision to occasionally be wrong.

What File Descriptors Are

The OS gives every process a budget of numbered tokens called file descriptors, and every connection costs one. TCP connections, open files, pipes, Unix sockets, all consume from the same budget. When the budget runs out, new connections are silently turned away. Existing connections are unaffected. Everything else looks normal.
The default budget on most Linux systems is 1,024. Check yours:

ulimit -n

At 1,024 your BEAM server stops accepting new connections. Ranch logs the error {:error, :emfile}, but nothing else looks wrong. CPU normal, memory normal, scheduler utilization normal, existing connections unaffected. The error is easy to miss unless you have alerting on it. New users can't get in. Current users notice nothing.

The First Hidden Limit: +Q

ulimit -n is the OS-level limit. How many file descriptors the process is allowed to hold. Raise it and the OS will allow more.

But the BEAM has its own internal limit: +Q. It controls how many ports the VM will open, regardless of what the OS allows.

effective limit = min(ulimit -n, +Q)

Raise only ulimit -n and +Q stays as the ceiling. The system still runs out at the BEAM's limit. Raise only +Q and ulimit -n stays as the ceiling. Same result.

Both must move together:

# Check the OS limit
ulimit -n

# Check the system-wide ceiling on Linux, ulimit -n cannot exceed this
cat /proc/sys/fs/file-max

# Check what the BEAM is actually using
# From inside a running node:
:erlang.system_info(:port_limit)   # the +Q ceiling
:erlang.system_info(:port_count)   # current usage

65,536 is the standard production value. Both the OS and the BEAM handle it comfortably on any modern server, no additional system configuration needed.

In vm.args:

+Q 65536

Set it slightly below ulimit -n, the BEAM uses a small number of ports for internal purposes and +Q needs headroom above your application's usage.

The Second Hidden Limit: Containers

Fix both limits locally, deploy to Docker or ECS, and the problem comes back.

Containers do not inherit the host's ulimit. Each container gets its own resource limits, and the default is usually whatever the base image ships with.

The host can have ulimit -n 65536 and every container running on it will still cap out at 1,024 unless you set the container's limit explicitly.

Docker:

# docker-compose.yml
services:
  app:
    ulimits:
      nofile:
        soft: 65536
        hard: 65536

ECS task definition:

"ulimits": [
  {
    "name": "nofile",
    "softLimit": 65536,
    "hardLimit": 65536
  }
]

Three Layers, All Must Move

Host OS ulimit       ← set in /etc/security/limits.conf or systemd LimitNOFILE
      ↓
Container ulimit     ← set in docker-compose or ECS task definition
      ↓
BEAM +Q              ← set in vm.args

The effective limit is always the lowest of the three. Fix two and leave one at the default and the system hits the same wall.

Verification after any change:

# Find the BEAM process ID
pgrep beam.smp

# Then check its limits
cat /proc/<pid>/limits | grep "open files"

# From inside the running BEAM node
:erlang.system_info(:port_limit)
:erlang.system_info(:port_count)

These two numbers should tell the same story. If port_limit is lower than your ulimit, +Q wasn't set. If port_limit is right but connections are still refused, the container's ulimit wasn't set.

The Production Configuration

# /etc/security/limits.conf (or systemd: LimitNOFILE=65536)
* soft nofile 65536
* hard nofile 65536

# vm.args
+Q 65536

// ECS task definition
"ulimits": [{ "name": "nofile", "softLimit": 65536, "hardLimit": 65536 }]

65536 is the standard production value. It's high enough for most workloads and well within OS and BEAM defaults.

The One Thing To Remember

When connections are being refused and everything else looks normal, check all three limits.

The Problem: GenServers process one message at a time

A GenServer handles messages one at a time, in order. While it's handling one message, every other message waits in the mailbox.

This is fine until a message handler does something slow. The ExchangeMonitor fetches exchange rates from an external API on a timer:

def handle_info(:refresh, state) do
  # HTTP call - blocks for 300-500ms
  rate = ExchangeAPI.fetch(state.currency_pair)
  schedule_next_refresh()
  {:noreply, %{state | rate: rate}}
end

While that HTTP call is running, the GenServer is frozen. A payment request that calls GenServer.call(monitor, :get_rate) has to wait in the mailbox for the entire duration of the fetch.

At low traffic this is invisible. Under load, with many concurrent callers and a slow API, the mailbox grows. Callers hit their default 5-second timeout. The system looks broken.

The root cause: slow work inside a GenServer callback.

What Task.Supervisor.async_nolink Does

Task.Supervisor.async_nolink moves the slow work into a separate process. The GenServer spawns the task and returns immediately. It keeps processing messages while the fetch runs concurrently in the background.

The nolink part matters. A regular Task.async creates a linked task. If the task crashes, the exit signal propagates to the GenServer and crashes it too. async_nolink removes that link. The task can fail without affecting the GenServer.

When the task completes, it sends a message back to the GenServer(success or failure). The GenServer handles that message like any other.

defmodule MyApp.ExchangeMonitor do
  use GenServer

  @refresh_interval :timer.seconds(30)

  def init(currency_pair) do
    {:ok, %{currency_pair: currency_pair, rate: nil, pending_ref: nil},
     {:continue, :initial_fetch}}
  end

  # handle_continue runs before any messages are processed
  def handle_continue(:initial_fetch, state) do
    {:noreply, spawn_fetch(state)}
  end

  def handle_info(:refresh, state) do
    schedule_refresh()

    if state.pending_ref do
      # Previous fetch still in flight — skip this tick
      {:noreply, state}
    else
      {:noreply, spawn_fetch(state)}
    end
  end

  # Task completed successfully
  def handle_info({ref, {:ok, rate}}, state) when state.pending_ref == ref do
    Process.demonitor(ref, [:flush])
    {:noreply, %{state | rate: rate, pending_ref: nil}}
  end

  # Task crashed
  def handle_info({:DOWN, ref, :process, _pid, reason}, state)
      when state.pending_ref == ref do
    Logger.warning("Exchange rate fetch failed: #{inspect(reason)}")
    {:noreply, %{state | pending_ref: nil}}
  end

  defp spawn_fetch(state) do
    task =
      Task.Supervisor.async_nolink(MyApp.TaskSupervisor, fn ->
        ExchangeAPI.fetch(state.currency_pair)
      end)

    %{state | pending_ref: task.ref}
  end

  defp schedule_refresh do
    Process.send_after(self(), :refresh, @refresh_interval)
  end
end

A few things worth noting in this pattern:

pending_ref prevents overlapping fetches. If the timer fires while a fetch is still in flight, the GenServer skips that tick. Without this, you can accumulate multiple concurrent fetches for the same currency pair.

Process.demonitor(ref, [:flush]) clears the monitor on success and flushes any pending DOWN message from the queue. Without the flush, a DOWN message can arrive after you've already processed the result and confuse the state.

handle_continue for the initial fetch. This guarantees the first fetch happens before any other message is processed. Without it, a caller could ask for the rate before it's ever been fetched.

When to Use Task.Supervisor.start_child

async_nolink is the right tool when you need the result back in the GenServer.

Task.Supervisor.start_child is the right tool when you don't. The work happens, you never hear from it:

Task.Supervisor.start_child(MyApp.TaskSupervisor, fn ->
  Notifications.send_transfer_confirmation(transfer)
end)

No reference stored, no message handled, no state updated. If the task fails, the work is lost and the caller never knows. Use it for side effects where that's acceptable, i.e: sending notifications, warming a cache, writing to an audit log, ...

The rule: if you need the result or need to know it failed, use async_nolink. If you genuinely don't care about the outcome, use start_child.

Extracting 10 bytes from a 10MB binary. The 10MB never left memory.

Here's what's actually happening, and how to find and fix it in production.

How Binaries Work in Elixir

Not all binaries live in the same place. The BEAM splits them into two categories based on size.

Small binaries (≤ 64 bytes) live directly on the process heap. They behave like any other data. They are copied when sent between processes, garbage collected when the process GCs.

Large binaries (> 64 bytes) live on a shared binary heap outside any process. The process doesn't hold the binary itself, it holds a ProcBin: a small reference-counted pointer to the shared heap.

Passing a large binary between processes is cheap because only the pointer moves, not the data.

The Discovery: Sub-Binaries Don't Copy

Here's what surprised me.

When you slice a large binary, you don't get an independent copy. You get a sub-binary(a small data structure on the process heap containing a pointer to the original, an offset, and a size).

The original large binary stays alive on the shared heap as long as any sub-binary references it.

This applies to every common slicing operation:

large = File.read!("huge_file.json")  # 10MB on shared heap

# All of these create sub-binaries — not copies
<<header::binary-size(10), _rest::binary>> = large
binary_part(large, 0, 10)
String.slice(large, 0, 10)
String.split(large, ",")
String.trim(large)
:binary.split(large, ",")

The rule: any function that returns a portion of a large binary without explicitly copying creates a sub-binary. This is not a bug. It is an intentional optimisation. Copies are expensive. But it means the original binary anchors in memory longer than you expect.

Why It's Hard to Notice

The trap is in what you see versus what's actually happening.

Your process heap looks small. The ProcBin is a few bytes. :observer shows low process memory. Everything looks fine.

Meanwhile, the shared binary heap is accumulating every large binary that has an active reference somewhere. No warning. No error. Just silent memory pressure building over time.

A GenServer that parses large payloads and stores a slice in state is the classic case:

def handle_info({:http_response, body}, state) do
  # body is a 500KB binary
  # This extracts 20 bytes — but keeps the 500KB referenced
  <<header::binary-size(20), _::binary>> = body
  {:noreply, %{state | last_header: header}}
end

Every response keeps its full body alive on the shared heap until the process GCs and the ProcBin for last_header is swept. Under load, these accumulate.

Step-by-Step Diagnosis

Step 1. Add :recon

:recon is a production introspection library for the BEAM. Add it to your dependencies:

# mix.exs
defp deps do
  [
    {:recon, "~> 2.5"}
  ]
end

Step 2. Find the offending processes

Find which processes are holding the most binary memory:

:recon.proc_count(:binary_memory, 10)

This returns the top 10 processes ranked by binary memory. Output looks like:

[{#PID<0.234.0>, 4823042, [
  current_function: {MyApp.Worker, :handle_info, 2},
  initial_call: {MyApp.Worker, :init, 1},
  ...
]}, ...]

The first number is binary memory in bytes. If a process shows several megabytes here, it's your suspect.

Step 3. Distinguish GC lag from a real leak

Memory that looks like a leak might just be data that hasn't been GC'd yet. Force GC on the suspect process and check if memory drops:

# Force GC
:erlang.garbage_collect(pid)

# Check memory again
:recon.proc_count(:binary_memory, 10)

Two outcomes:

• Memory drops : GC lag. The data was already dead, just unswept. No design problem, the GC will handle it naturally under load.

• Memory stays : Real retention. Something in the process state is still holding a reference to a large binary. This is a design problem.

Step 4. Find all leaking processes in one shot

:recon.bin_leak(10)

Returns 10 processes currently retaining the most binary memory

Step 5. Find what's holding the reference

Once you have the PID, inspect its state.

:sys.get_state(pid)

Look for fields that contain large binaries or sub-binaries. Common culprits:

• State fields storing slices of large payloads

• ETS tables holding sub-binary values

• Accumulated log buffers

Resolution

Once you've found the reference, the fix is :binary.copy/1. It creates a fully independent binary without connection to the original:

# Before — sub-binary anchors the full body
<<header::binary-size(20), _::binary>> = body
%{state | last_header: header}

# After — independent copy, original can be GC'd
<<header::binary-size(20), _::binary>> = body
%{state | last_header: :binary.copy(header)}

The same fix applies regardless of how the sub-binary was created:

# Any of these:
slice = binary_part(large, 0, 10)
slice = String.slice(large, 0, 10)
[first | _] = String.split(large, ",")

# Wrap with :binary.copy/1 before storing
:binary.copy(slice)

The Rule

Any function returning a portion of a large binary without explicitly copying creates a sub-binary that anchors the original in memory.

When you need to store a slice of a large binary (in process state, in ETS, in a message), copy it first. Pay the copy cost once at the point of extraction. Everything downstream gets a clean, independent value with no hidden reference.