EntroQ

EntroQ is a fault-tolerant, competing-consumer task queue with strongly transactional semantics: a claimed task cannot be dropped on worker crash, and results cannot be committed more than once. Workers scale horizontally without coordination: just add more.

As of v1.0.0, EntroQ also ships a Kubernetes service mesh that makes ordinary HTTP microservices asynchronous without any queue code in the services themselves. A sidecar intercepts outbound HTTP calls and routes them through queues transparently.

Go, Python, TypeScript, and Elixir clients. PostgreSQL, Redis, and in-memory backends. Kubernetes deployment via Helm.

---

Pronounced "Entro-Q" ("Entro-Queue"), as in the letter that comes after "Entro-P", the next step in managing systems complexity.

It's the right way to outsource reliability and consistency guarantees when working in a competing-consumer environment where exactly-once semantics are needed and work should never be lost. It enables infinite composability, and the microservice mesh built on top of it showcases just one of the many powerful ways it can be used.

Background: Asynchronous Thinking for Microservice System Design Go docs: pkg.go.dev/github.com/shiblon/entroq | CHANGELOG

Core Concepts

EntroQ simplifies distributed task management by narrowing the entire mutating interface down to two atomic operations: Claiming an available task and Modifying (inserting, deleting, or changing) a set of tasks. Every one of these operations is wrapped in a version-locked transaction. The moment a task is claimed or modified, its version increments, which ensures that if one worker succeeds in a mutation, all other workers holding an older version will naturally fail. This "Commit Once" semantic eliminates the risk of work getting dropped after delivery or, more dangerously, being committed to a downstream system more than once.

Progress in EntroQ is counterintuitive in its simplicity. While many systems favor strict FIFO ordering, EntroQ selects available tasks randomly. This prevents "poison pill" tasks from rising to the head of the line and starving the entire cluster; if a task causes a worker to crash, it eventually times out and is returned to the pool where it is likely to be picked up by another worker while others continue making progress on "good" work. This design allows you to scale your processing power simply by adding more workers. Once a task is in a queue, no further communication between nodes is required.

Workers can set configuration options that automatically quarantine tasks if they have failed too many times, where they can be inspected manually, fixed, and reintroduced to their work queue as needed.

Getting Started

The fastest way to see these concepts in action is using the provided Docker sandbox.

Sandbox (Docker Compose)

The sandbox provides a complete environment: PostgreSQL, the EntroQ service, Prometheus, and Grafana for monitoring.

docker compose up --build

The service will be available at:

• gRPC: localhost:37706
• HTTP/JSON API: localhost:9100
• Prometheus Metrics: localhost:9100/metrics
• Grafana Dashboards: localhost:3000

Command Line

You can poke at the running service using the Go-based command line client:

go install github.com/shiblon/entroq/cmd/eqc@latest
eqc --help

The Python client is on PyPI:

python3 -m pip install entroq            # client + async worker
python3 -m pip install "entroq[pg]"      # + experimental direct-PostgreSQL client (entroq.experimental.pg)

To install an unreleased version straight from the clients/py subdirectory:

python3 -m pip install "git+https://github.com/shiblon/entroq#subdirectory=clients/py"

The JavaScript/TypeScript client is on npm:

npm install entroq

Shell Workers

eqc work is a small worker protocol for shell scripts and other local commands. It is meant for leaf work and simple work-then-respond flows: one input task is claimed, its JSON value is written to the command's stdin with a trailing newline, and each non-empty stdout line is parsed as one JSON output task.

In one terminal, start a worker:

eqc work -q /demo/in -Q /demo/out -c 'printf "{\"seen\":%s}\n" "$(cat)"'

In another terminal, insert work and read the response:

eqc ins -q /demo/in -v '{"name":"Ada"}'
eqc ts -q /demo/out

Use -- COMMAND [ARG...] instead of -c when you do not need a shell. This worker copies each input task value to /demo/out:

eqc work -q /demo/in -Q /demo/out -- cat

There is no implicit copy mode; cat is the explicit command in this example.

Important details:

• -q, --queue can be repeated; the worker claims one task at a time from any listed input queue.
• -Q, --out-queue is the single queue for all stdout JSONL output tasks.
• --in delays output tasks, matching eqc ins --in.
• On success, the input task is deleted and all stdout tasks are inserted in the same Modify.
• On command failure, the current task is modified in place: attempt and err are updated, and At is moved by --retry-in. The default --max-attempts 0 means unlimited retries.
• Invalid JSONL output, too much stdout, or exhausted attempts move the input task to --error-queue, defaulting to <input>/err.
• --recur-in is the cron-like mode: after a successful run, the claimed task is deleted and a fresh copy of its input value is inserted back into the same input queue with that relative delay.

Scripts should write logs to stderr. They may call eqc for ordinary client operations, including document claims, but those operations are not part of the worker's final atomic Modify; claimed docs also need to be used within their original lease because eqc work does not renew them for the script.

Language Clients

EntroQ defines a simple protocol: claim -> work -> modify. All clients follow this transactional loop.

Go

The Go client is the reference implementation and supports automatic background task renewal.

svc, _ := entroq.New(ctx, eqgrpc.Opener("localhost:37706", eqgrpc.WithInsecure()))
defer svc.Close()

w := worker.New(svc,
    worker.WithDoModify(func(ctx context.Context, task *entroq.Task, val json.RawMessage, docs []*entroq.Doc) (*worker.Result, error) {
        log.Printf("Processing: %s", string(task.Value))
        return worker.Modify(task.Delete()), nil // finish by deleting
    }),
)
w.Run(ctx, worker.Watching("/my/queue"))

Python

The Python client is async (asyncio). A worker claims tasks, renews the claim in the background while your handler runs, and atomically applies the Modification your handler returns.

import asyncio
from entroq.json import EntroQJSON
from entroq.types import Modification
from entroq.worker import EntroQWorker

async def main():
    client = EntroQJSON("http://localhost:9100")

    @EntroQWorker.handler
    async def process(task, docs):
        print(f"Processing: {task.value}")
        return Modification(Modification.deleting(task))  # finish by deleting

    worker = EntroQWorker(client, "/my/queue")
    await worker.run(process)

asyncio.run(main())

TypeScript (Node.js)

The TypeScript client provides a worker abstraction for modern async/await environments. See clients/js/README.md for installation and full API docs.

import { EntroQClient, EntroQWorker } from "entroq";

const client = new EntroQClient({ baseUrl: "http://localhost:9100" });
const worker = new EntroQWorker(client);

await worker.run(["/my/queue"], async (task) => {
    console.log("Processing:", task.value);
    // Return a modification to apply atomically; here we delete to finish.
    return { deletes: [{ id: task.id, version: task.version, queue: task.queue }] };
});

Elixir

The Elixir client provides an HTTP/JSON client and a worker abstraction that renews task/doc claims while perform/2 runs, then applies returned modifications after renewal stops and final versions are stable. See clients/elixir/README.md for details.

defmodule MyWorker do
  use EntroQ.Worker

  @impl true
  def perform(task, _docs) do
    {:modify, EntroQ.Modification.delete(task)}
  end
end

client = EntroQ.new("http://localhost:9100")
EntroQ.Worker.run(client, ["/my/queue"], MyWorker)

Any language (work gateway)

A worker can be written in any language without importing EntroQ or gRPC: eqlink work runs the hard part of the worker loop (claim, renew, commit) on its behalf and speaks a small newline-delimited JSON protocol over a stdio pipe or a WebSocket. See the wire contract and the client recipes in docs/workgateway-protocol.md, and a minimal Python example in examples/workgateway/. This surface is new and currently experimental (see the changelog).

Document Store

In addition to tasks, EntroQ provides a key-value document store that shares the same atomic transaction space. A doc has a namespace, an ID, a primary key, an optional secondary key, a JSON content field, and an optional expiration time.

Unlike tasks, docs are not work items. They are durable shared state: configuration, counters, reduce output, or any data that multiple workers need to read or coordinate around.

Docs can opt into the same built-in garbage collection as task queues by placing /gc=<timestamp> in their primary key. Once active, GC removes the entire (namespace, primary key) group atomically, and only while every member is unclaimed. Malformed gc= values are reported and never collected.

Docs and tasks share the same Modify call, so you can atomically insert a task and its initial doc state, or delete a task and update a doc, in one round trip with no possibility of partial failure.

ClaimDocs acquires an exclusive lease on all docs sharing a primary key in one atomic operation, making the primary key the natural unit of coordinated ownership. Results are returned ordered by (primary key, secondary key), which makes reduce-shaped strategies straightforward.

Reusable Workers

pkg/workers provides ready-made worker implementations for common patterns, so you don't have to write the claim/renew/modify loop yourself:

Package	What it does
batchworker	Aggregates tasks and processes them in configurable batches
mapworker	Generic map: transforms a task's value and routes the result to an outbox queue
httpworker	Executes HTTP requests described by task values; writes the response back as a task
procworker	Runs a subprocess described by the task value; captures stdout/stderr as an output task
fileworker	Writes task values to files
appendworker	Appends task values to a stuffedio WAL journal

Kubernetes Service Mesh

EntroQ v1.0.0 ships a Kubernetes operator that turns the queue into a transparent async service mesh. Ordinary HTTP microservices communicate through queues with no queue-awareness in their code. Each pod gets an eqlink sidecar that intercepts outbound HTTP calls, converts them to queue insertions, and converts inbound queue tasks back to HTTP calls on the local service.

[you] ──HTTP──▶ svc-a ──[queue]──▶ svc-b ──[queue]──▶ svc-c
               frontend             greeter            compliment

A service calls its neighbor with a plain HTTP request to http://svc-b.localhost:8080/path. The pod's hostAliases resolve the address to 127.0.0.1, so the request hits the local eqlink sender. eqlink strips the domain suffix, maps the hostname to a queue name, and inserts a task. On the other side, the receiver's eqlink claims the task, calls the local service over loopback, and routes the response back through a per-request reply queue. Neither service knows any of this happened.

Authorization is declared with two CRDs:

• EntroQQueue: declares which queues a service exposes and which callers may enqueue to them, using label-set predicates with AND/OR semantics.
• EntroQIdentity: maps a Kubernetes service account to a set of mesh label claims.

The operator watches these CRDs cluster-wide and maintains an OPA authorization document that the EntroQ service enforces on every queue operation.

See examples/greetings-demo for a working end-to-end example with three Python services, deployment manifests, and a step-by-step walkthrough.

CRD reference (field-by-field, worked examples, policy verification): docs/mesh-policy.md

Cross-Instance Task Handoff

eqlink handoff links two EntroQ instances directly, moving tasks from a queue on one instance into an inbox on another exactly once in effect. Direction is fixed by the endpoints — it claims from --from and delivers to --to — and you can run it next to either side:

[--from EQ] ──claim──▶ [eqlink handoff] ──insert──▶ [--to EQ inbox]

eqlink handoff --from <source-addr> --from-queue <q> --to <dest-addr> --to-queue <inbox>

Each delivery atomically inserts the inbox task and a value-stripped dedup tombstone keyed by a deterministic transfer ID, then deletes the source task. If a crash re-delivers, the insert collides on the tombstone and no duplicate is produced; the happy path cleans up its own tombstone, and the destination backend's built-in GC sweeps crash orphans once their TTL elapses. Inbox consumers need no special handling; they see ordinary tasks.

Production Deployment (Helm)

For Kubernetes environments, a Helm v3 chart is available in charts/entroq. It supports backend toggling and secure secret management. See charts/entroq/README.md for full options. Postgres is shown below. In-memory (journaled) and Redis-backed are also available.

make helm-sync
helm install my-queue ./charts/entroq --set entroq.backend.type=postgres

---

Technical Protocol Details

Tasks

A task is defined by a Queue Name, a Globally Unique ID, a Version, an Arrival Time, and a Value. The version increments every time the task is mutated, providing the foundation for transactional safety.

A task's identity for modification is the tuple (id, version, queue): a delete, change, or dependency must name the queue the task currently occupies. A mismatch is rejected (DependencyError) and the server never substitutes the stored queue — this is what makes queue-based authorization enforceable, since you cannot reach a task by misdeclaring where it lives.

Queues

Queues are not first-class entities; they spring into existence only when tasks are assigned to them and vanish when empty, allowing for highly dynamic, ad-hoc workflows.

Claims and Safe Work

When a worker issues a Claim, it atomically increments the task's version and pushes its arrival time into the future, "locking" it for a specific duration. For long-running work, the best practice is to keep these initial claim times low and rely on background renewal to maintain the lock.

Any finalization or downstream update is handled by the Modify call, which can include any combination of insertions, deletions, or value changes. If any single part of the modification fails (perhaps because a task's version has drifted or a dependency was not met), the entire operation rolls back. This requires a shift in how you think about "Safe Work": you should aim for idempotence, perhaps by writing results to unique, timestamped files before committing the final task deletion. The rule is simple: only mutate tasks you have successfully claimed, and always assume your work might be retried.

Backends

EntroQ is backend-agnostic. The Go library supports:

• In-memory: Perfect for testing or light-duty singleton services (includes a WAL journal).
• PostgreSQL: Production-grade persistence using SKIP LOCKED for high performance.
• Redis: Persistence backed by Redis (server binary eqredis).
• gRPC: A client that talks to a remote eqpg, eqmem, or eqredis service instance.

Authorization

EntroQ integrates with Open Policy Agent (OPA) for queue-level access control. Enable it with --authz=opahttp on any service binary. On every operation the server packages the request (queues + actions + Authorization header) into a JSON document, POSTs it to OPA, and allows or denies based on the response.

Two provider sets ship with the repo:

Deployment	Provider	Identity source
Standalone / OIDC	pkg/authz/opadata/conf/providers/entroq/	JWT from any standard OIDC IDP
Kubernetes mesh	pkg/authz/opadata/conf/providers/k8s/	Pod service account tokens + EntroQQueue/EntroQIdentity CRDs

For the Kubernetes mesh path, the eqk8s operator maintains the OPA authorization document automatically from CRDs, with no manual data wiring needed when using the Helm chart.

Full configuration guide, IDP-specific settings, and examples: pkg/authz/opadata/OPA_AUTHZ.md

Runnable sandbox: examples/authz/

Versioning

EntroQ follows Semantic Versioning in spirit, but while the user base is small it does not promise a major-version bump for every breaking change: a breaking change may ship in a minor release, always called out in the changelog. Pin an exact version and read the changelog before upgrading. This policy will tighten as adoption grows.

The PostgreSQL backend (eqpg) carries its own schema version, which only changes when the schema does and never exceeds the module version. After upgrading across a release whose changelog notes a schema change, run eqpg schema upgrade before starting the service — it refuses to open on a version mismatch rather than migrating a live database silently. The in-memory and Redis backends need no migration.