Portunus

portunus is a small, generic lock server for the Erlang ecosystem, built on Ra, RabbitMQ's Raft implementation. It provides cluster-wide mutual exclusion, TTL leases with renewal, leader election, and a succession queue with pluggable placement affinity. It is a CP (consistency over availability) service in the tradition of etcd and ZooKeeper, but embedded as a library rather than run as an external service.

Portunus implements a Ra state machine. It does not depend on or use Khepri. It is named after the Roman god of keys, doors, and gates.

Project Maturity

This project is very young, therefore breaking changes are not only possible but likely.

Why

Because the standard library's global module has its limits (no clear partition handling semantics, no fencing token support, major changes and rapid iteration are out of the question due to its maturity), and hand-rolling a distributed locking library (compared to to ZooKeeper, etcd, Consul in terms of core features) is hard and error-prone.

At the same time, the field of Raft-based distributed locking services is mature and well understood. And Team RabbitMQ happens to have a mature and solid Raft implementation.

Core Ideas

portunus is an embedded distributed lock server with the following key concepts and design ideas:

• Safety and liveness are separated. At most one owner per key, and monotonically increasing fencing tokens, are guaranteed by Raft, independent of clocks. Lease expiry (TTL) is liveness: approximate and clock-dependent. A client must treat a renewal failure as "lease possibly lost" and stop acting
• Fencing tokens. Every grant returns a token (the Raft index). The guarded resource (usually another component) records the highest token it has seen and rejects stale ones. This is what makes a lock safe across a paused or partitioned holder
• Leases. A lock is held under a TTL lease and stays held for as long as the lease is renewed. Many exclusive keys can share one lease (a session)
• Succession and affinity. A held key keeps a queue of succession candidates; release, revocation, and expiry promote the best-ranked one. Ranking is FIFO by default; an affinity strategy (pinned, preferred, hash, metric, random, or a custom portunus_affinity module) biases which node wins. Affinity is a hint, never a correctness requirement

Supported Erlang/OTP Versions

Portunus requires Erlang/OTP 27 and should work equally well on Erlang 28 and 29, including mixed-version clusters during upgrades.

Supported ra Versions

Portunus targets ra 3.1.8 or later versions.

Getting Started

Portunus requires Erlang/OTP 27 or later. Every node starts a Ra system, which keeps its Raft state under the given directory, and the nodes then form a named cluster:

%% on every node, once
ok = portunus:start_system(portunus, "/var/lib/portunus"),

%% on the first node: form the cluster
{ok, _Started, _Failed} = portunus:start_cluster(portunus, my_locks, [node()]),

%% on the other nodes: join the existing members (idempotent)
ok = portunus:join_or_form(portunus, my_locks, ['first@host']).

For nodes that boot independently and retry until the cluster is up, call join_or_form/3 with the same full member list on every node from a retry loop: every node picks the same seed, so two nodes can never each form their own cluster.

Locks and Leases

The core API grants a lease, then acquires keys under it. Every grant returns a fencing token:

{ok, Lease} = portunus:grant_lease(my_locks, 30000),
{ok, Token} = portunus:acquire(my_locks, {resource, 42}, Lease, self()),
%% carry Token into every external write made under this lock
ok = portunus:release(my_locks, {resource, 42}, Token).

A lease granted this way must be renewed by the caller (renew_leases/2), or it expires after its TTL and its locks are released. With auto_renew, a renewer process linked to the caller keeps the lease alive for as long as the caller lives:

{ok, Lease} = portunus:grant_lease(my_locks, 30000, #{auto_renew => true}).

The one-shot conveniences bundle the lease, its renewal, and the acquire into a single auto-renewing handle:

{ok, Handle} = portunus:lock(my_locks, {resource, 42}, 30000),
ok = portunus:unlock(Handle),

%% or scoped to a function, released on return or exception
Result = portunus:with_lock(my_locks, {resource, 42}, 30000,
                            fun() -> do_exclusive_work() end).

Waiting for a Held Key

acquire/4 never queues: if the key is held, it returns {error, {held_by, Owner}}. To wait for the key instead, join its succession queue; the caller is promoted when the current owner releases, is revoked, or expires:

case portunus:acquire_or_join_succession_queue(my_locks, Key, Lease, self()) of
    {ok, Token} ->
        %% the key was free: we own it now
        owned(Token);
    {queued, _Depth} ->
        %% promoted later: the lease holder receives this message
        receive
            {portunus, granted, Key, Token, Lease} -> owned(Token)
        end
end.

Watches

watch/2 subscribes the calling process to a key's acquire and release events, and owner/2 reads the current owner directly:

{ok, Ref} = portunus:watch(my_locks, Key),
receive
    {portunus, watch, Ref, {acquired, Owner}} -> track(Owner);
    {portunus, watch, Ref, released} -> untrack()
end,
ok = portunus:unwatch(my_locks, Ref).

Watches are best-effort notifications; a decision that must be safe should be fenced with the token, not made from a watch event.

Sessions

A session is one lease with many exclusive keys claimed under it: renewal cost stays per session, not per key, and the session process is the lease holder, so its death releases all of its keys at once. On lease loss the session exits with reason lease_lost, taking a linked, non-trapping opener with it (the fail-stop default, since its claims are gone):

{ok, Session} = portunus_session:open(my_locks, #{ttl_ms => 30000}),
{ok, _Token1} = portunus_session:claim(Session, {vhost, <<"a">>}),
{ok, _Token2} = portunus_session:claim(Session, {vhost, <<"b">>}),
ok = portunus_session:release(Session, {vhost, <<"a">>}),
ok = portunus_session:close(Session).

Health and Introspection

status/1 returns the leader, members, quorum, and machine-derived counts; has_quorum/1 is a quorum-confirming read; is_member/1 answers from the local replica, so it holds during an election and is the right check for a bootstrap retry loop. Operational metrics are exposed as seshat counters and gauges per node (see portunus_counters).

The sections below are the "batteries": higher-level components built on top of the lock and lease core.

Leader Election

portunus_election keeps exactly one instance of a component running in the cluster. A participant runs on every node; the elected one has elected/1 called (with the fencing token in its context), and stepped_down/1 when the lease is lost, at which point ownership moves to another node:

-module(my_scheduler).
-behaviour(portunus_election).
-export([elected/1, stepped_down/1]).

elected(#{token := Token}) ->
    {ok, Pid} = my_scheduler_worker:start_link(Token),
    {ok, Pid}.

stepped_down(Pid) ->
    my_scheduler_worker:stop(Pid).

{ok, _Pid} = portunus_election:start_link(my_locks, scheduler_key,
                                          my_scheduler, [],
                                          #{ttl_ms => 30000}).

A Managed Set of Keys

portunus_service runs one election per key from a fixed key set, with start/3 and stop/2 invoked per key. Every node runs the same service; each key ends up with exactly one owner in the cluster:

-module(my_partition_owner).
-behaviour(portunus_service).
-export([keys/1, start/3, stop/2]).

keys(NumPartitions) ->
    lists:seq(1, NumPartitions).

start(Partition, Token, _Args) ->
    my_partition_worker:start_link(Partition, Token).

stop(_Partition, Pid) ->
    my_partition_worker:stop(Pid).

{ok, _Pid} = portunus_service:start_link(my_locks, my_partition_owner, 8,
                                         #{ttl_ms => 30000}).

Declarative Supervision

portunus_supervisor looks like an Erlang/OTP supervisor, except only one instance of each child spec returned by init/1 can exist in the cluster at any given time. The elected owner runs it under a local supervisor, and Portunus drives the cross-node ownership transfer:

-module(example_supervisor_mod).
-behaviour(portunus_supervisor).
-export([start_link/0, init/1]).

start_link() ->
    portunus_supervisor:start_link(my_locks, ?MODULE, []).

init([]) ->
    SupFlags = #{strategy => one_for_one, intensity => 10, period => 10},
    {ok, {SupFlags,
          [#{id => stats_collector,
             start => {my_stats_collector, start_link, []},
             restart => permanent}]}}.

Child specs may carry the extended {permanent, Delay} / {transient, Delay} restart type (as supervisor2 accepts), rewritten by portunus_delayed_restart into a rate-limited standard spec.

The Dynamic Registry

portunus_registry is the same idea with children added and removed at runtime instead of being returned by init/1. This is the mirrored_supervisor replacement for runtime-managed children such as dynamic shovels and federation links: every node registers the same child specs (driven by replicated parameter or policy events), Portunus runs one election per key, and the elected node runs the child under a local supervisor:

{ok, Reg} = portunus_registry:start_link(my_locks, #{}),
ok = portunus_registry:add(Reg, #{id => shovel_a,
                                  start => {my_shovel, start_link, [a]},
                                  restart => permanent}).

remove/2 stops contending on this node; removing a key on its current owner moves the child to another node, and the key is gone cluster-wide once every node that added it removes it. owned_keys/1 and which_children/1 report what this node currently runs.

Placement Affinity

Elections, services, and registries accept #{affinity => Spec} to bias which node wins a key. A spec names a built-in strategy with its argument, a custom portunus_affinity behaviour module, or a scoring fun:

%% prefer these nodes, earliest first, over any others
#{affinity => {preferred, ['a@host', 'b@host']}}

%% spread keys evenly across members (rendezvous hashing)
#{affinity => {hash, undefined}}

%% the least-loaded node wins, by a locally read metric
#{affinity => {metric, fun() -> spare_capacity() end}}

Layout

• portunus_machine: the Ra state machine (leases, locks, fencing tokens, score-ordered succession, tick-based expiry, periodic log snapshots)
• portunus: the public client API
• portunus_session: one process's lease with many keys claimed under it
• portunus_election: keeps one elected instance of a component per key, cluster-wide
• portunus_service: a managed set of keys driven by a callback module
• portunus_supervisor: a declarative, supervisor-shaped layer built on top of the registry
• portunus_registry: a dynamic cluster-wide supervisor with children added and removed at runtime
• portunus_affinity: placement strategies for succession, built-in and custom
• portunus_counters: seshat metrics

Building and Testing

Both rebar3 (primary) and erlang.mk are supported, like Ra.

rebar3 compile
rebar3 ct
rebar3 dialyzer
rebar3 ex_doc      # API documentation under doc/
gmake              # the erlang.mk equivalent

License

Dual-licensed under the Apache License 2.0 and the Mozilla Public License 2.0, the same as Ra. See LICENSE, LICENSE-APACHE2, and LICENSE-MPL-RabbitMQ.

© Team RabbitMQ <teamrabbitmq@gmail.com>.