AppControl Architecture

Network Topology

                        +------------------+
                        |    Browser /     |
                        |    appctl CLI    |
                        +--------+---------+
                                 |
                                 | HTTPS :443 (nginx)
                                 | or HTTP :3000 (direct API)
                                 v
                   +----------------------------+
                   |     nginx (TLS termination)|
                   |        :80/:443            |
                   |                            |
                   |  /api/*  --> backend:3000  |
                   |  /ws     --> backend:3000  |
                   |  /*      --> frontend:8080 |
                   +-------------+--------------+
                                 |
                    HTTP/WS (internal network)
                                 |
         +-----------------------+-----------------------+
         |                                               |
         v                                               v
+------------------+                          +-------------------+
|    Backend API   |                          |    PostgreSQL 16  |
|    :3000         |                          |    :5432          |
|                  +------- sqlx pool ------->+                   |
|  REST API        |                          +-------------------+
|  WebSocket /ws   |
|  (JWT auth)      |
+--------+---------+
         |
         | WebSocket /ws/gateway
         | (internal, gateway connects here)
         |
+--------+---------+       +------------------+
|    Gateway PRD   |       |    Gateway DR    |
|    :4443         |       |    :4443         |
|                  |       |                  |
| Directly exposed |       | Directly exposed |
| to agents (mTLS) |       | to agents (mTLS) |
+---+-----------+--+       +---+-----------+--+
    |           |              |           |
    | WSS :4443 | WSS :4443    | WSS :4443 | WSS :4443
    |  (mTLS)   |  (mTLS)      |  (mTLS)   |  (mTLS)
    v           v              v           v
+--------+ +--------+      +--------+ +--------+
| Agent  | | Agent  |      | Agent  | | Agent  |
| srv-01 | | srv-02 |      | dr-01  | | dr-02  |
+--------+ +--------+      +--------+ +--------+

Note: Agents connect DIRECTLY to gateways, NOT through nginx.
      Each gateway is exposed on its own IP/DNS.

Component Roles

Component	Role	Inbound Port	Config Source
nginx	TLS termination for web UI	`:80/:443`	`nginx.conf`
Frontend	React SPA	`:8080` (internal)	Build-time env
Backend	REST API + WebSocket hub + FSM engine	`:3000` (internal)	Env vars only
Gateway	Agent connection multiplexer (one per zone)	`:4443` (exposed directly)	YAML + env var overrides
Agent	Local process manager + health checker	None (client only)	YAML + env var overrides
CLI	CLI client for automation/schedulers	None (client only)	Env vars + CLI args
PostgreSQL	Persistent storage	`:5432`	Standard

Important: Gateways are exposed directly on their own IP/port (e.g., gateway-prd.company.com:4443). They do NOT go through nginx. Each zone (PRD, DR, DMZ) has its own gateway.

Communication Protocols

Browser ---HTTPS---> nginx:443 ---HTTP---> backend:3000
                                     (proxy_pass /api/* and /ws)
                               ---HTTP---> frontend:8080
                                     (proxy_pass /* for SPA)

appctl  ---HTTP(S)---> backend:3000
                       (direct API calls, no nginx needed)

Gateway ---WebSocket---> backend:3000/ws/gateway
          (internal, persistent connection, reconnect on failure)

Agent ---WSS (mTLS)---> gateway:4443/ws
         (DIRECT connection, not through nginx)
         (outbound only, reconnect with exponential backoff 1s..60s)

Connection Details

Agent --> Gateway (WSS, mTLS)

Agent initiates outbound WebSocket connection to gateway
No inbound port needed on the agent (firewall-friendly)
mTLS: agent presents its certificate, gateway verifies it
WSS on port 443/4443 passes through most corporate firewalls (indistinguishable from HTTPS traffic)
Automatic reconnect with exponential backoff (1s, 2s, 4s... max 60s)
Offline buffer: agent stores messages locally if gateway is unreachable

Gateway --> Backend (WebSocket)

Gateway connects to backend WebSocket endpoint
Routes messages bidirectionally: agents <--> backend
Reconnect on disconnect with configurable interval
In-memory message buffer during backend unavailability

Frontend --> Backend (via nginx)

nginx proxies /api/* to backend:3000/api/
nginx proxies /ws to backend:3000/ws (WebSocket upgrade)
All other routes serve the React SPA (client-side routing)

CLI --> Backend (HTTP)

Direct HTTP calls to backend REST API
No nginx/frontend needed
Configured via APPCONTROL_URL env var or --url CLI arg
Exit codes designed for scheduler integration (0=OK, 1=Fail, 2=Timeout)

Environment Variables

Backend

Variable	Default	Description
`PORT`	`3000`	HTTP listen port
`DATABASE_URL`	`postgresql://appcontrol:appcontrol@localhost:5432/appcontrol`	PostgreSQL connection string
`JWT_SECRET`	`dev-secret-change-in-production`	JWT signing secret (required in production)
`JWT_ISSUER`	`appcontrol`	JWT issuer claim
`APP_ENV`	`development`	Environment: `development`, `staging`, `production`
`CORS_ORIGINS`	(none)	Comma-separated allowed origins
`LOG_FORMAT`	`text`	`text` or `json` (structured logging)
`DB_POOL_SIZE`	`20`	Database connection pool max connections
`DB_IDLE_TIMEOUT_SECS`	`600`	Idle connection timeout (seconds)
`DB_CONNECT_TIMEOUT_SECS`	`30`	Connection acquisition timeout (seconds)
`SHUTDOWN_TIMEOUT_SECS`	`30`	Graceful shutdown timeout (seconds)
`RATE_LIMIT_AUTH`	`10`	Auth endpoint rate limit (per IP per minute)
`RATE_LIMIT_OPERATIONS`	`5`	Operation endpoint rate limit (per user per minute)
`RATE_LIMIT_READS`	`200`	Read endpoint rate limit (per user per minute)
`RETENTION_ACTION_LOG_DAYS`	`0`	Days to keep action_log entries (0 = unlimited)
`RETENTION_CHECK_EVENTS_DAYS`	`0`	Days to keep check_events partitions (0 = unlimited)

Gateway

Variable	Default	Description
`LISTEN_ADDR`	`0.0.0.0`	Bind address
`LISTEN_PORT`	`4443`	Bind port for agent connections
`BACKEND_URL`	`ws://localhost:3000/ws`	Backend WebSocket URL
`GATEWAY_ID`	`gateway-01`	Gateway identifier
`GATEWAY_SITE_ID`	(none)	UUID of the site this gateway belongs to. Optional: gateways without a site appear as "Unassigned" in the UI and can be assigned later.
`GATEWAY_ZONE`	(deprecated)	Legacy zone label. Use `GATEWAY_SITE_ID` instead.
`BACKEND_RECONNECT_SECS`	`5`	Reconnect interval to backend

Agent

Variable	Default	Description
`GATEWAY_URL`	`ws://localhost:4443/ws`	Gateway WebSocket URL
`AGENT_ID`	`auto` (hostname-based UUID)	Agent identifier
`GATEWAY_RECONNECT_SECS`	`10`	Reconnect interval
`TLS_ENABLED`	`false`	Enable mTLS
`TLS_CERT_FILE`	-	Agent certificate path
`TLS_KEY_FILE`	-	Agent private key path
`TLS_CA_FILE`	-	CA certificate path

CLI

Variable	Default	Description
`APPCONTROL_URL`	`http://localhost:3000`	Backend API URL
`APPCONTROL_API_KEY`	-	API key for authentication

Docker Compose Ports

Service	Internal Port	External Port	Purpose
postgres	5432	5432	Database
backend	3000	3000	REST API + WebSocket
frontend	8080	8080	Web UI
gateway	4443	4443	Agent connections

Multi-Site (DR) Topology

                    +-----------------------+
                    |    Central Backend    |
                    |    + PostgreSQL       |
                    +-----------+-----------+
                                |
                +---------------+---------------+
                |                               |
    +-----------+-----------+       +-----------+-----------+
    |   Gateway PRD         |       |   Gateway DR          |
    |   zone: PRD           |       |   zone: DR            |
    |   :4443               |       |   :4443               |
    +---+-------+-------+--+       +---+-------+-------+---+
        |       |       |              |       |       |
     +--+--+ +--+--+ +--+--+       +--+--+ +--+--+ +--+--+
     |Agent| |Agent| |Agent|       |Agent| |Agent| |Agent|
     |db-p | |app-p| |web-p|       |db-d | |app-d| |web-d|
     +-----+ +-----+ +-----+       +-----+ +-----+ +-----+
       PRD site servers              DR site servers

Each site has its own gateway in its zone
Agents connect to the gateway in their zone
Both gateways connect to the same central backend
Switchover: stop PRD components → start DR components → commit
Backend tracks active_site_id per application

Data Flow: Health Check Cycle

1. Agent scheduler tick (every 30s)
   |
2. Execute check_cmd (shell command)
   |
3. Compare exit_code to cached value (deduplication)
   |
   +-- Same exit_code? → skip (no delta sent)
   |
   +-- Different exit_code? → send CheckResult message
       |
4. Agent --[WebSocket]--> Gateway --[WebSocket]--> Backend
       |
5. Backend: FSM transition (next_state_from_check)
   |
   +-- exit 0: STARTING→RUNNING, DEGRADED→RUNNING
   +-- exit 1: RUNNING→DEGRADED
   +-- exit ≥2: *→FAILED
   |
6. Backend: INSERT INTO state_transitions (append-only)
   |
7. Backend: INSERT INTO check_events (append-only)
   |
8. Backend: WebSocket push to subscribed frontends
   |
9. Frontend: React Query cache invalidation → UI update

Data Flow: Start Application (DAG Sequencing)

1. User clicks "Start" / POST /api/v1/apps/{id}/start
   |
2. Backend: INSERT INTO action_log (log before execute)
   |
3. Backend: Kahn's topological sort → execution plan
   |  Level 0: [Oracle-DB]
   |  Level 1: [Tomcat-App, RabbitMQ]
   |  Level 2: [Apache-Front, Batch-Processor]
   |
4. For each level (sequentially):
   |  For each component in level (in parallel):
   |    4a. Set state STARTING → INSERT INTO state_transitions
   |    4b. Send ExecuteCommand (start_cmd) → Gateway → Agent
   |    4c. Agent: double-fork + setsid → detached process
   |    4d. Wait for check_cmd to return exit 0 (RUNNING)
   |    4e. If timeout/fail → mark FAILED, suspend job
   |
5. All levels complete → all components RUNNING → job complete

Data Flow: DR Switchover (6 Phases)

Phase 1 - PREPARE:    Verify DR agents connected, run health checks
Phase 2 - FREEZE:     Block user operations on active site
Phase 3 - STOP_SOURCE: Stop all components on PRD (reverse DAG)
Phase 4 - START_TARGET: Start all components on DR (DAG order)
Phase 5 - VERIFY:     Run integrity checks on DR site
Phase 6 - COMMIT:     Update active_site_id (point of no return)

Rollback possible before COMMIT → restart source, cancel switchover
RTO measured: time from FREEZE to COMMIT

Data Flow: 3-Level Diagnostic + Rebuild

1. POST /api/v1/apps/{id}/diagnose
   |
2. For each component, run 3 check levels:
   |  Level 1 (Health):       check_cmd → exit code
   |  Level 2 (Integrity):    integrity_check_cmd → exit code
   |  Level 3 (Infrastructure): infra_check_cmd → exit code
   |
3. Recommendation matrix (8 combinations):
   |  H=OK, I=OK, F=OK  → HEALTHY (no action)
   |  H=OK, I=OK, F=KO  → HEALTHY (infra issue but app works)
   |  H=OK, I=KO, *     → HEALTHY (integrity issue, informational)
   |  H=KO, I=OK, F=OK  → RESTART (process died, data OK)
   |  H=KO, I=OK, F=KO  → INFRA_REBUILD (process + infra bad)
   |  H=KO, I=KO, F=OK  → APP_REBUILD (process + data bad)
   |  H=KO, I=KO, F=KO  → INFRA_REBUILD (everything bad)
   |
4. POST /api/v1/apps/{id}/rebuild
   |  Check rebuild_protected flag (409 if protected)
   |  Execute in DAG order (databases before appservers)
   |  For INFRA_REBUILD: use bastion agent (rebuild_agent_id)
   |  Measure RTR (Recovery Time for Rebuild)

WebSocket Considerations for Legacy Environments

WebSocket (RFC 6455) is used for all real-time communication. This is the right choice for AppControl because:

Agent connections are long-lived (hours/days) — HTTP polling would be wasteful
Bidirectional messaging — backend can push commands to agents without polling
Low latency — state changes are reflected in the UI within milliseconds

Potential Issues on Legacy Infrastructure

Scenario	Problem	Mitigation
Old HTTP/1.0 proxy	Cannot upgrade to WebSocket	Use WSS on port 443 (looks like HTTPS, proxy passes it through)
Corporate firewall	Blocks non-HTTP protocols	WSS on 443 is indistinguishable from HTTPS at the TLS level
Load balancer timeout	Closes idle connections	Agent heartbeat every 60s keeps connections alive
WAF inspection	Deep packet inspection rejects WS frames	Whitelist the gateway endpoint, or use mTLS (WAF cannot inspect)
Reverse proxy misconfigured	Missing `Upgrade`/`Connection` headers	nginx config includes proper WebSocket proxy headers (see `nginx.conf`)

Why Not HTTP Long Polling?

Agent-to-gateway requires bidirectional real-time messaging
Long polling would mean agents cannot receive commands instantly (only on next poll)
The 60s heartbeat interval already acts as a keep-alive mechanism
WebSocket overhead per message is 2-6 bytes vs. ~500 bytes for HTTP headers

Deployment Recommendation

For environments with strict proxy requirements:

Gateway: listen on port 443 (standard HTTPS port)
         with TLS termination + mTLS client verification
         → passes through all corporate firewalls
         → indistinguishable from HTTPS traffic

LISTEN_PORT=443  (in gateway config / env var)

The agent's outbound-only connection model means no firewall rules need to be opened for inbound traffic on monitored servers.

Message Protocol (Agent <-> Backend)

Agent → Backend

Message	Fields	Purpose
`Register`	agent_id, hostname, labels, version	First message on connect
`Heartbeat`	agent_id, cpu, memory, at	Every 60s, keeps connection alive
`CheckResult`	component_id, check_type, exit_code, stdout, duration_ms, at	Delta-only health/integrity/infra check result
`CommandResult`	request_id, exit_code, stdout, stderr, duration_ms	Response to ExecuteCommand

Backend → Agent

Message	Fields	Purpose
`ExecuteCommand`	request_id, component_id, command, timeout_seconds	Run a command on target host
`UpdateConfig`	components: Vec	Push new component configuration
`Ack`	request_id	Acknowledge receipt

FSM State Machine

                  +----------+
           +----->| UNKNOWN  |<-----+
           |      +----+-----+      |
           |           | first check|
           |           v            |
      +----+----+  +--------+  +---+-----+
      | STOPPED |  |RUNNING |  | FAILED  |
      +----+----+  +---+----+  +----+----+
           |           |            |
     start |    exit 1 |    retry   |
           v           v            v
      +----+-----+ +---+-----+ +---+-----+
      | STARTING | |DEGRADED | | STOPPING|
      +----+-----+ +----+----+ +----+----+
           |             |           |
     exit 0|       exit 0|     stop  |
           v             v     done  v
        RUNNING       RUNNING    STOPPED

  Any state --[heartbeat timeout]--> UNREACHABLE
  UNREACHABLE --[reconnect]--> previous state

Valid transitions:

UNKNOWN → RUNNING | STOPPED | FAILED (first check determines initial state)
STOPPED → STARTING (start command issued)
STARTING → RUNNING (check returns exit 0)
STARTING → FAILED (check returns exit ≥2 or timeout)
RUNNING → DEGRADED (check returns exit 1)
RUNNING → FAILED (check returns exit ≥2)
RUNNING → STOPPING (stop command issued)
DEGRADED → RUNNING (check returns exit 0)
DEGRADED → FAILED (check returns exit ≥2)
DEGRADED → STOPPING (stop command issued)
STOPPING → STOPPED (stop confirmed by check)
FAILED → STARTING (restart attempt)
FAILED → STOPPING (cleanup stop)
* → UNREACHABLE (heartbeat timeout, 3 missed cycles)
UNREACHABLE → * (agent reconnects, replays buffered state)

Permission Model

view < operate < edit < manage < owner

  view:    Read app status, components, reports
  operate: Start, stop, restart, execute commands
  edit:    Modify app config, add/remove components, dependencies
  manage:  Grant permissions, create share links, manage switchovers
  owner:   Delete app, transfer ownership

  Effective = MAX(direct_user_permission, team_permissions...)
  Org admin = implicit owner on ALL apps in the organization

Note: edit does NOT include operate. An editor can change config but cannot start/stop. This is intentional — separation of config and operations.

Process Detachment (Agent)

When the agent starts a process (start_cmd, rebuild_cmd), it uses double-fork + setsid to ensure the process survives agent crash:

Agent process
  |
  fork()
  |
  +-- Intermediate child
       |
       setsid()   ← new session leader, detached from agent's session
       |
       fork()
       |
       +-- Grandchild (detached process)
            |
            close(0,1,2)  ← close stdin/stdout/stderr
            /dev/null redirect
            execvp(command)
            |
            Now reparented to PID 1 (init)
            Survives agent crash/restart

  Intermediate child exits immediately
  Agent reaps intermediate child (waitpid)
  Grandchild PID returned via pipe for tracking

Navigation Menu

Uh oh!

AppControl Architecture

Network Topology

Component Roles

Communication Protocols

Connection Details

Agent --> Gateway (WSS, mTLS)

Gateway --> Backend (WebSocket)

Frontend --> Backend (via nginx)

CLI --> Backend (HTTP)

Environment Variables

Backend

Gateway

Agent

CLI

Docker Compose Ports

Multi-Site (DR) Topology

Data Flow: Health Check Cycle

Data Flow: Start Application (DAG Sequencing)

Data Flow: DR Switchover (6 Phases)

Data Flow: 3-Level Diagnostic + Rebuild

WebSocket Considerations for Legacy Environments

Potential Issues on Legacy Infrastructure

Why Not HTTP Long Polling?

Deployment Recommendation

Message Protocol (Agent <-> Backend)

Agent → Backend

Backend → Agent

FSM State Machine

Permission Model

Process Detachment (Agent)

Uh oh!

FilesExpand file tree

architecture.md

Latest commit

History

architecture.md

File metadata and controls

AppControl Architecture

Network Topology

Component Roles

Communication Protocols

Connection Details

Agent --> Gateway (WSS, mTLS)

Gateway --> Backend (WebSocket)

Frontend --> Backend (via nginx)

CLI --> Backend (HTTP)

Environment Variables

Backend

Gateway

Agent

CLI

Docker Compose Ports

Multi-Site (DR) Topology

Data Flow: Health Check Cycle

Data Flow: Start Application (DAG Sequencing)

Data Flow: DR Switchover (6 Phases)

Data Flow: 3-Level Diagnostic + Rebuild

WebSocket Considerations for Legacy Environments

Potential Issues on Legacy Infrastructure

Why Not HTTP Long Polling?

Deployment Recommendation

Message Protocol (Agent <-> Backend)

Agent → Backend

Backend → Agent

FSM State Machine

Permission Model

Process Detachment (Agent)