Introduce the Control Plane role: topology, orchestration, authorization, audit

[em3s]

Background

Actionbase was originally designed for a single-tenant, single-cluster deployment model.

Today production environments are more complex:

• Multiple tenants
• Active/Standby deployments
• Dedicated read clusters
• Environment-specific policies (dev/test/prod)
• User-level authorization
• Audit requirements
• Operational workflows such as cleanup

In these environments, operational actions such as DDL, cleanup, and cluster management must be coordinated across multiple Actionbase clusters.

Every operational client — Console, CLI, scripts — carries its own table of which hostname is which, and each one reimplements the same orchestration: the order of steps, the preconditions that make a step safe, the active/standby fanout, the queue that retries what failed. None of it is reusable, and it couples clients to deployment topology.

That logic belongs in one place: the Control Plane.

Architecture

Revised in #480: one binary, two roles — not a separate module.

This issue originally called for a new control-server module. The dependency argument for splitting turned out to be about image size rather than correctness, so a config property decides what an instance is instead:

actionbase.role=DATA      Query, DML and DDL against this instance's own cluster. The default.
actionbase.role=CONTROL   Operational APIs spanning clusters: topology, orchestration, audit.

One artifact, one image, one release. The two HTTP surfaces are exclusive — a DATA instance refuses /control, a CONTROL instance refuses /graph/v2, /graph/v3 and /queue/v1 — so the runtime isolation is what two modules would have given, without a second release to maintain.

Console / CLI
        ↓
  role=CONTROL      topology, authorization, sequencing, audit
        ↓
  role=DATA         query, DML, DDL, per-cluster invariants
        ↓
      HBase

An existing deployment that sets no role keeps behaving exactly as before.

Control Plane vs Data Plane

The terms are used from an architectural perspective.

• Data Plane — data access, cluster-local request execution, and the invariants of a single cluster.
• Control Plane — topology, authorization, operational policy, orchestration across clusters, and auditing.

DDL is inherently a control-plane concern, but DDL execution stays in the data plane. The control plane decides whether, in what order, and against which clusters.

Where the boundary is

One rule decides most of the design questions that follow:

A single cluster's invariants are enforced by the data plane. The control plane owns sequencing, direction and execution — and only predicts what the data plane will refuse.

The data plane already refuses teardown while a reference is live (#477) and already answers what still references a table (#478). The control plane must not reimplement those checks. Duplicating them guarantees that one day the two will disagree, and the copy will be the one operators trust. It composes them instead: which steps, in which order, on which side.

What the data plane already provides

Every primitive the first workflow needs already exists, per cluster:

Operation	Endpoint
replication off (scope 1 → 0)	POST /graph/v3/datastore/hbase/tables/{table}/replication/disable
replication on (scope 0 → 1)	POST /graph/v3/datastore/hbase/tables/{table}/replication/enable
enable / disable a table	POST /graph/v3/datastore/hbase/tables/{table}/enable | /disable
drop a table	DELETE /graph/v3/datastore/hbase/tables/{table}
deactivate metadata	PUT /graph/v3/databases/{database}/tables/{table} (active=false), then PUT /graph/v2/storage/{storage} for an indirect reference
delete metadata	DELETE /graph/v3/databases/{database}/tables/{table}, then DELETE /graph/v2/admin/storage/{storage}
what still references a table	GET /graph/v3/datastore/hbase/tables/{table}/references (#478)

Nothing above needs to be built again. What is missing is the layer over it.

The first workflow to move: datastore table cleanup

Teardown order — local → unlink → disable → drop → purge. Setup order — enable → global.

Action	Meaning
local	replication scope 1 → 0
global	replication scope 0 → 1
unlink	deactivate the graph metadata bound to the table (active=false)
disable	disable the table
enable	enable the table
drop	drop the table
purge	delete the metadata left behind after the table is gone

Preconditions, all of which a client currently has to know:

• disable requires scope 0 first — disabling while replication is on breaks standby sync
• global requires the table enabled first
• drop requires all three of local, unlink, disable, and only applies where the table still exists
• purge applies only where the table is already gone — complementary to drop
• fanout is partial: only the sides not already at the target state are acted on

The control plane answers these instead of every client deriving them.

Roadmap

• Support tenant and cluster topology configuration — Let an instance be deployed as the control plane #480
• Expose what references a datastore table — Expose what references a datastore table #478
• Reject datastore table teardown while a live reference remains — Reject datastore table teardown while a URI-bound label references it #477
• Cluster fanout client — per-side calls driven by the resolved topology, with each side's failure surfaced independently rather than one masking the other
• Cleanup state projection — per-table state for every side, keyed by cluster, carrying the reference chain. Because the control plane sees every tenant, it can finally answer which other tenant is holding a table, which no single-cluster client can
• Planner + job resource, dry run — ordered steps, or a refusal naming the precondition that failed. One planner and one job model, with a dryRun flag that stops before any side effect, so plan and execution can never drift apart
• Authorization — OIDC delegation with JWT verification; permissions over env / tenant / resource / operation / side, with ddl ⊃ dml ⊃ query
• Execution — the same job resource with dryRun=false: 202 and a job id, batch targets, per-side partial results, idempotent skip where a side is already at the target
• Audit logging for operational actions

Destructive execution deliberately lands after authorization. Until then the control plane stays read-only plus dry run, so a cross-cluster drop never becomes reachable before there is something to check who asked for it.

Design decisions

• Sides are symmetric. A table's state is one entry per side, so a table that exists only on the standby is simply an absent active entry rather than a special case.
• Jobs take a list of targets. Clients build serial queues because the API is per-table; batch input removes the reason to build one, in every client at once.
• Plan and execution are one endpoint. Two code paths computing the same plan is the duplication this issue exists to remove — repeating it inside the control plane would be the same mistake at a smaller scale.
• Reads are keyed by cluster. One table can be referenced from several tenants, so cleanup is a cluster-level activity even though topology maps tenants to URLs.
• Actions stay imperative. Inferring steps from a desired-state diff reads well until drop and purge, which cannot be undone and whose audit record should carry what the operator actually asked for.

Done When

• A CONTROL instance is the operational entry point for Console and CLI.
• Topology is managed centrally and clients no longer carry hostname tables.
• Active/Standby orchestration, including partial fanout, is handled by the control plane.
• Authorization is enforced centrally.
• Audit logs are recorded for operational actions.
• Existing server APIs remain compatible without modification.

Notes

Storage

The control plane keeps operational metadata — permissions, audit logs, job state — in Actionbase itself. Which cluster holds it is a deployment decision to settle when execution lands, since a CONTROL instance does not serve its own data plane.