chore(db): drop the vestigial agent-DB port_mappings table [HOLD: phase 2] - #1011
Draft
odesenfans wants to merge 78 commits into
Draft
chore(db): drop the vestigial agent-DB port_mappings table [HOLD: phase 2]#1011odesenfans wants to merge 78 commits into
odesenfans wants to merge 78 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #1011 +/- ##
==========================================
+ Coverage 83.77% 83.83% +0.06%
==========================================
Files 208 211 +3
Lines 23083 23240 +157
Branches 1409 1415 +6
==========================================
+ Hits 19337 19483 +146
- Misses 3364 3375 +11
Partials 382 382 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
Author
|
Automated review (Claude, multi-agent find + verify pass). The two-phase gating logic in the PR description is sound, and the existence-guard + downgrade are more care than most drop migrations get. But as the branch stands (cut from dev, without #1010), the migration would not actually stick, and there are two coordination hazards worth handling now since this PR will sit open for a while. Must address before un-drafting
Worth considering (altitude)
Minor
One non-finding worth recording: the inline |
odesenfans
force-pushed
the
od/drop-vestigial-agent-port-mappings
branch
2 times, most recently
from
July 2, 2026 22:26
e30b590 to
a68cd2d
Compare
First step towards splitting the supervisor into an Aleph-agnostic hypervisor and an Aleph network agent. Adds a gRPC wire format for communication with the hypervisor + Python bindings.
The agnostic VM-management boundary introduced in 0.A was named "hypervisor", but QEMU and Firecracker are the actual hypervisors; this layer is a supervisor/orchestrator (historically aleph-vm-supervisor). Rename it before further Phase 0 work builds on it, so later diffs are not polluted by renames. - proto/hypervisor.proto -> proto/supervisor.proto (package aleph.hypervisor.v1 -> aleph.supervisor.v1, service Hypervisor -> Supervisor) - src/aleph/vm/hypervisor/ -> src/aleph/vm/supervisor/, with regenerated _pb bindings (supervisor_pb2, _grpc, .pyi) - tests/hypervisor/ -> tests/supervisor/ (merged into the existing dir) - scripts/generate_proto.py, scripts/check_proto_clean.sh, proto/README.md and pyproject.toml excludes updated to the new paths - the 0.A plan doc keeps its historical name and identifiers, with a note The pre-existing aleph.vm.hypervisors (plural) backend package is left untouched: those are the real hypervisor launchers.
…952) Introduces the Python `Supervisor` abstraction, the single agent-to-VM-management call path that a gRPC client and eventually a Rust supervisor drop into. New package `aleph.vm.supervisor`: - `types.py` - frozen-dataclass DTOs + enums mirroring `supervisor.proto` (no Aleph or protobuf types leak through). - `errors.py` - closed `SupervisorError` hierarchy (one per `ErrorCode`) + `translate_exception()` mapping the scattered internal backend exceptions + a `translating_errors()` context manager. This is the wire-error vocabulary, built in Python first. - `abc.py` - seven capability ABCs aggregated into `Supervisor` (25 async methods, 1:1 with the proto RPCs). - `inprocess.py` - `InProcessSupervisor` wrapping today's `VmPool` / `VmExecution`.
…translator (#953) Build a launchable QEMU instance configuration from a message-agnostic CreateVmSpec, and translate an Aleph message into a CreateVmSpec. The hypervisor side never touches ExecutableContent.
AlephQemuResources inherited from AlephFirecrackerResources purely to reuse host-resource mechanics (volumes, disk accounting, kernel path), even though QEMU is not a Firecracker VM. That coupling forces a single shared message_content declaration. Introduce a hypervisor-agnostic VmResources base (controllers/resources.py) holding the shared, message-free surface (namespace, volumes, rootfs/kernel paths, to_dict, download_kernel) plus the message-coupled mechanics as free functions (host_volumes_from_message, disk_usage_delta). Firecracker and QEMU specialise it independently: Firecracker keeps message_content required (clean dereferences in program/instance), QEMU owns its optional message_content (it may be built message-free). HostVolume moves to the agnostic module.
Make VmExecution constructible and operable from a message-free CreateVmSpec (resolved on-disk paths only), alongside the existing message path. - VmExecution.spec is a MessageSpec | CreateVmSpec union (exactly one source; 'neither' is unrepresentable). message/original/vm_spec are derived read-only properties; narrowing uses isinstance(self.spec, ...). - Widen AlephQemuResources.message_content to optional (QEMU may now be built message-free) with the matching download None-guards, and add from_spec, which builds a holder with no download. This is the capability the resource untangle made possible without touching Firecracker. - CreateVmSpec.rootfs / require_rootfs centralize the rootfs lookup (one-or-none / exactly-one, rejecting malformed multi-rootfs specs); both from_spec and build_qemu_configuration use them. - Design + implementation-plan docs under docs/plans/.
Wires the message-free create path into the pool and the supervisor boundary.
…spec (#956) Route the production creation path through the spec for eligible workloads.
Makes reboot-recovery message-free: the supervisor reattaches surviving VMs from on-disk controller configs + systemd, dropping the DB path entirely.
…Supervisor abstraction (#962) The agent's create path now goes through the message-agnostic Supervisor abstraction instead of reaching into VmPool directly. The supervisor creates VMs from a `CreateVmSpec` and never sees an Aleph message; the agent owns messages via a new agent-side `AgentVmRegistry` (which replaces the vestigial pool.message_cache).
…sor abstraction (#965)
…start_persistent_vm (#970) Lifts update-watching ("redeploy a VM when its Aleph message updates") off the VmExecution god-object into a new agent-owned UpdateWatcher, and makes start_persistent_vm execution-free.
…eps (#973) Replace time-based sleeps in the on_reaped reap/cancel tests with direct awaits on the timer task. on_reaped fires inside _expire/_watch's finally before the task completes, so awaiting the task deterministically observes the callback (or, for cancellation, drives the finally without firing it). Removes the 0.01s-timer-vs-0.05s-sleep margin that could flake on loaded CI.
…#971) Moves operator owner-authorization off the hypervisor pool object and onto the agent registry, so execution.message is read nowhere in orchestrator/views/operator.py.
…k fallback (#974) Replace example.org with one.one.one.one and dns.google.com (example.com has been flaky lately) and fix the CI to actually use the current version of the FastAPI example.
…ate_vm_execution readback (#972) Makes the agent persist its own DB record and migrates the last agent-side readers of pool-execution messages to the AgentVmRegistry, then kills the create_vm_execution readback.
… (off the raw pool) (#976)
…points Rebasing dev onto main collided main's confidential-VM listing feature with dev's message-free-supervisor rewrite of the same list/start paths. dev's rewrite reads VM status from the supervisor (VmInfo), which had no notion of 'awaiting confidential init', so taking it verbatim would silently drop a main feature: a confidential VM waiting for its owner would be filtered out of the executions list and re-allocated by the scheduler forever, and start_persistent_vm would try to wait/recreate it in a loop. Preserve the feature on dev's architecture: - VmInfo gains awaiting_confidential_init; InProcessSupervisor populates it from execution.is_awaiting_confidential_init in _to_vm_info. - The v1/v2 list endpoints expose it again; v1 keeps listing an awaiting-init VM once its tap network exists (and excludes it before then, per the schema). - start_persistent_vm leaves an awaiting-init VM untouched (no wait, no recreate); only the owner can start it via /confidential/initialize. main's agent-level batched-systemd optimization is obsoleted by dev commit #976 (the agent now reads status through supervisor.list_vms() rather than querying systemd itself), so its test is removed; the equivalent batching belongs in the supervisor's list_vms. Confidential-init tests from main are adapted to dev's supervisor-based code path.
Process-split between the Aleph agent and the VM supervisor. The VM supervisor is now its own service and serves agent requests via gRPC. Backported fixes from main related to confidential VMs and payment checks.
* docs: design for gRPC-only supervisor (full process split)
Two-process split as the only supported deployment: a supervisor daemon
owning the pool and serving the Supervisor interface over gRPC, and an
agent that reaches VMs exclusively through that interface.
Key decisions:
- rename InProcessSupervisor -> LocalSupervisor (the pool-backed engine)
- agent keeps a thin embedded seam for dev/tests only
- fold admission + GPU reservation into create_vm (atomic)
- HAProxy/L7 domain mapping moves fully to the agent (fed by list_vms),
out of the pool; the supervisor stays L3/L4
- interface-first, two phases: decouple agent from pool, then transport
and two-service packaging
* docs: Phase 1 implementation plan for gRPC-only supervisor
Decouple the agent from VmPool. P1.1 (rename + wiring) and P1.6a
(recreate_network) are ready to build; backup/restore, migration and
confidential measurement are blocked on three contract decisions noted
in the plan.
* Rename InProcessSupervisor to LocalSupervisor (module local.py)
* Update production references to LocalSupervisor
* Update supervisor tests to LocalSupervisor
* Extract build_supervisor factory for explicit agent wiring
* Remove throwaway rename smoke test
* Add recreate_network to the Supervisor interface
* Implement recreate_network in LocalSupervisor
* Route recreate_network endpoint through the supervisor
* feat(haproxy): add agent-side domain-mapping sync driven by supervisor
Introduce orchestrator/haproxy_sync.py with sync_domain_mappings(supervisor),
which derives each local VM's IP from VmInfo.ipv4.address and pushes the
domain->IP map to HAProxy. Add build_map_entries_from_vm_ips and
write_entries_to_backend to haproxy.py as the low-level building blocks.
* refactor(haproxy): route domain-mapping callers through agent sync
Replace pool.update_domain_mapping() calls in the orchestrator views and tasks
with sync_domain_mappings(supervisor): regenerate_proxy, notify_allocation,
operate_update, the periodic resync task, and the domains-aggregate handler.
The supervisor's list_vms() output now drives HAProxy, not the in-process pool.
* refactor(haproxy): remove pool HAProxy coupling, agent owns domain sync
Delete VmPool.update_domain_mapping and its load-time call; the agent now
seeds an initial forced sync in periodic_domain_resync at startup. Drop the
dead instances-based helpers from haproxy.py (fetch_list_and_update,
update_backends, _build_map_entries, _resolve_vm_ip), keeping fetch_list and
the low-level socket/map helpers. Add a guard test.
* docs: record resolved contract decisions (backup, migration, confidential)
* feat(supervisor): extend Measurement with SEV info and launch measure
Add a SevInfo dataclass mirroring the seven QEMU query-sev fields and
carry it (plus the base64 launch_measure) on Measurement, defaulted so
existing callers and the proto path keep working. This lets the
confidential measurement endpoint preserve its current response shape
once the logic moves into the engine.
* feat(supervisor): implement confidential ops in LocalSupervisor
Move the initialize/measurement/inject-secret logic out of the agent
endpoints and into the engine: write the SEV session certificates and
start the controller service, query the launch measurement plus the
seven query-sev fields, and inject the secret then resume the guest.
The confidential ops are no longer stubs, so drop them from the
conformance STUB_METHODS set.
* feat(agent): delegate confidential endpoints to the supervisor
Rewire operate_confidential_initialize, _measurement and _inject_secret
to call only the Supervisor interface: the initialize guards now read
supervisor.get_vm (status + confidential_mode) instead of the pool, and
the three endpoints no longer touch require_vm_pool, the pool or
QemuVmClient. The measurement response is preserved byte-for-byte as
{"sev_info": {7 fields}, "launch_measure": ...}; inject_secret now
returns {"status": "ok"} since the void supervisor method cannot carry
the post-injection QMP status (the agreed minor contract change).
* Add check_spec_admission for the spec create path
* Fold capacity admission into create_vm_from_spec atomically
* Remove agent-side admission in notify_allocation; surface boundary 503
* Route reserve_resources through the Supervisor interface
* fix(supervisor): derive measurement tee_backend from the VM config, not a hardcoded SEV
The TEE generation is determined by the VM's confidential configuration
(an input the supervisor already holds via the execution policy), so
get_measurement now derives tee_backend through _confidential_mode
instead of hardcoding TeeBackend.SEV. TeeBackend.SEV still covers SEV and
SEV-ES (refined by sev_info.policy); SEV-SNP maps distinctly.
* Enrich BackupOps engine: include_volumes, metadata, restore_from_image
Grow the supervisor backup interface so the agent can drop its pool-backed
backup/restore logic without any feature regression:
- BackupInfo carries checksum, volumes and source_sizes for completed
archives, so the agent builds the HTTP body and download sidecar headers
from engine metadata.
- start_backup gains include_volumes: the engine archives the VM's
non-read-only persistent volumes alongside the rootfs.
- restore_from_image(vm_id, image_path, max_virtual_size_bytes) restores from
a QCOW2 already staged on a host path (uploaded image or downloaded volume):
the engine validates, size-checks, swaps the rootfs and restarts.
- GrpcSupervisor stubs restore_from_image NotImplementedError (wired in Phase 2).
* Route backup/restore endpoints through the supervisor
Rewire the five backup/restore endpoints off the VmPool: they now reach VM
and disk work only through the Supervisor interface, keeping HTTP-only
concerns (owner auth, presigned download URLs and their verification,
multipart/JSON parsing, staging uploaded bytes, sidecar response headers).
- operate_backup calls start_backup(quiesce_guest, include_volumes): 202 while
RUNNING, metadata body (sourced from BackupInfo) plus a presigned URL when
COMPLETE; InvalidBackend -> 400, InsufficientResources -> 507.
- operate_backup_status reports via list_backups.
- operate_backup_download verifies the presigned URL, streams from
download_backup, and sets Content-Length / X-Backup-Checksum / X-Source-Size
from the backup metadata.
- operate_backup_delete delegates to delete_backup.
- operate_restore stages the uploaded QCOW2 or the volume_ref download to a
temp path, then calls restore_from_image; qemu-img rejection and oversized
disks surface as 400.
Drops the agent-side BackupState background-task tracking, the inline
QemuVmClient/fsfreeze logic, require_vm_pool and pool reads. Adds a guard test
asserting the five endpoints reference no pool symbols.
* Update restore-rejects-invalid-image test for the supervisor path
The test exercised the removed operator._do_restore / _parse_restore_upload
internals. Rewire it through operate_restore against a supervisor whose
restore_from_image raises InvalidBackendError (what the engine's qemu-img
check does on a non-QCOW2), still asserting the 4xx (not 500).
* test: stage volume_ref restore under tmp_path
* Add P2P-migration disk/VM seam to the Supervisor interface
The agent's P2P pull protocol keeps the network transport (compress, hash,
serve, download, rebase); add the four MigrationOps methods that own the
parts touching the VmExecution and the pool, which the agent must not reach
into directly:
- stop_vm_for_export: graceful stop + locate the persistent-volumes dir
- restart_after_failed_export: bring the VM back up after a failed export
- create_migrated_vm: create a persistent VM from a staged instance message
- release_migrated_vm: stop + forget a VM that has migrated away
Implemented for real in LocalSupervisor, stubbed NotImplementedError in
GrpcSupervisor (wired in Phase 2). Surface count 32 to 36.
* Route the P2P migration endpoints through the supervisor
Thread the supervisor (not the VmPool) through the migration runner and the
three migration endpoints. The P2P pull protocol stays the agent-side network
transport: the on-host HTTP disk transfer, tokens, ExportJob/ImportJob states
and request/response shapes are unchanged. Only the disk/VM work moves behind
the supervisor:
- migration_export gates via supervisor.get_vm (VmNotFoundError = 404) and
reads status/backend/confidential_mode off the returned VmInfo; run_export
asks the supervisor to stop the VM and report its volumes dir, then the agent
compresses and serves the files it finds there.
- migration_import gates the already-running check via supervisor.get_vm;
run_import downloads and rebases the peer disks (network transport) then calls
supervisor.create_migrated_vm. The dest-dir rmtree decision is made against a
single supervisor.get_vm probe instead of pool.executions.
- migration_cleanup releases the source VM via supervisor.release_migrated_vm
(engine: stop_vm + forget_vm) instead of pool.stop_vm/forget_vm.
The three endpoints (and the runner) now contain no require_vm_pool / vm_pool /
pool.create_a_vm / pool.executions references; a guard test asserts this via
inspect.getsource.
* Delete the directory-based migration flow
Remove the provisional directory-based export_vm / import_vm /
get_migration_status from the MigrationOps ABC, LocalSupervisor,
GrpcSupervisor and the gRPC server handlers, and the migrate.py
manifest module they relied on. Migration now rides the standard
lifecycle RPCs; only the export-time graceful stop survives as a
MigrationOps method (it needs a guest powerdown stop_vm does not do).
The orphan ExportVm / ImportVm / GetMigrationStatus proto RPCs stay in
supervisor.proto with a note to drop them in the Phase 2 proto pass, so
the generated bindings are unchanged and check_proto_clean stays green.
Surface count 36 to 30.
* Route migration import through the standard create_vm RPC
run_import now builds a CreateVmSpec from the fetched instance message
with build_create_vm_spec (the same path run.create_vm_execution uses for
a normal persistent instance) and calls supervisor.create_vm(spec)
instead of the bespoke create_migrated_vm. The spec's rootfs path is
PERSISTENT_VOLUMES_DIR/<vm_hash>/rootfs.qcow2, exactly where the
download+rebase already staged the overlay; build_create_vm_spec adopts
an already-present host-persistence overlay rather than recreating it, so
create_vm reuses the staged disk with no re-download (covered by
test_import_spec_build_reuses_staged_overlay).
The failed-export restart rides the standard start_vm RPC instead of
restart_after_failed_export. The disk/VM seam methods were removed in the
previous commit; this wires the runner onto the lifecycle RPCs.
* Route migration cleanup through the standard delete_vm RPC
migration_cleanup drops the migrated-away source through delete_vm(wipe=
False) instead of the bespoke release_migrated_vm. delete_vm stops the
VM, forgets the definition and removes the controller config; wipe=False
leaves the disks alone, which is correct for a source whose data now
lives on the destination. The view tests assert create_vm and delete_vm
are driven, mocking build_create_vm_spec on the import path.
* docs: revise migration design to lifecycle RPCs (delete directory-based flow)
* Add run_program_code to the Supervisor interface
Persistent programs are served through the supervisor: the engine runs
the request over the VM's guest channel (the agent cannot reach a
supervisor-owned channel across the boundary). Stubbed in GrpcSupervisor
for Phase 2.
* Emit persistent program specs from build_program_create_vm_spec
Thread the program message's on.persistent into the spec instead of
rejecting persistent programs; a persistent program boots under systemd
on the engine side.
* Boot persistent programs through the Firecracker spec create path
Drop the persistent-Firecracker guard in _create_firecracker_from_spec;
VmExecution.start already branches on self.persistent to boot under
systemd and wait for init.
* Route persistent programs through the supervisor, drop the legacy paths
Persistent on-request and on-event programs now create through the
supervisor spec path (build_program_create_vm_spec + create_vm) and are
served via supervisor.run_program_code; on-demand programs keep the
agent-side program_client.run_code. The no-idle-reap finally behavior is
preserved for persistent programs. Deletes _run_code_on_request_legacy
and _run_code_on_event_legacy.
* Final pool removal: the agent reaches VMs only through the supervisor
Drop the pool parameter from the run.py entry points; the one remaining
legacy create path (confidential / GPU / firecracker instances) reaches
the embedded pool through supervisor.pool, None across the gRPC boundary.
Stop exposing the raw pool to handlers (app["vm_pool"] -> private
app["_engine_pool"], read only by the process-lifecycle hooks). Point
resources.py at supervisor.get_host_info() for GPU inventory and available
disk (HostInfo grows available_disk_bytes / gpu_inventory / available_gpus).
Delete require_vm_pool and the dead get_execution_or_404. Add the capstone
guard test asserting the agent view/run modules are pool-free.
* feat(supervisor): carry GPU requests on CreateVmSpec
Add device_id/model request fields to GpuSpec and populate spec.gpus from
message.requirements.gpu in build_create_vm_spec. The spec now carries the
GPU REQUEST (device_id/model); pci_host stays empty until the engine resolves
a concrete host card. proto_convert keeps round-tripping pci_host/supports_x_vga
only; the new fields default to empty (Phase 2 adds them to the proto).
* feat(supervisor): resolve and reserve GPUs on the spec create path
pool.create_vm_from_spec now resolves each GPU REQUEST (device_id, empty
pci_host) on the spec to a concrete available host card, atomically under
creation_lock, and rewrites the spec with the resolved pci_host so
build_qemu_configuration and from_spec wire it through. The resolved HostGPUs
are attached to the execution so uses_gpu() holds them against other creates
and VmInfo reports them, exactly like the message path.
The spec has no owner address, so it cannot honour this user's own
pre-reservation. The agent clears its own reservation before calling create_vm
(it holds message.address); any reservation still valid here belongs to another
user and is skipped. Raises InsufficientResourcesError when a request cannot be
satisfied.
* feat(orchestrator): route GPU instances through the spec create path
Drop the GPU exclusion from _is_spec_eligible so GPU QEMU instances reach
build_create_vm_spec and the engine's GPU-capable spec path. Before driving
create_vm for a GPU spec, the agent releases its own GPU pre-reservation via
the new pool.release_user_reservations(content.address): the engine's spec-path
resolution carries no owner address and skips any reservation still standing,
so the owner's own hold must be cleared first or it would block the create.
In-process only; owner-aware reservation over the gRPC boundary is a later
slice.
* feat(supervisor): populate spec.tee for confidential instances in translate
build_create_vm_spec now resolves the trusted_execution firmware ref to a
host path and emits a TeeConfig (backend SEV, NO_DBG policy, per-VM session
dir, firmware path) instead of rejecting confidential instances. The launched
policy is NO_DBG, faithfully reproducing the message path which never reads
trusted_execution.policy.
Adds TeeConfig.firmware_path (defaulted so proto_convert keeps round-tripping).
* feat(supervisor): build confidential launch on the spec create path
pool.create_vm_from_spec now builds a QemuConfidentialVMConfiguration for a
spec carrying spec.tee (via the new build_qemu_confidential_configuration,
mirroring AlephQemuConfidentialInstance.configure), and creates an
AlephQemuConfidentialInstance through VmExecution.from_spec/create. The VM is
left in awaiting_confidential_init: start(write_config=False) does not
enable_and_start the controller because is_confidential is True, so the owner
drives the launch through initialize_confidential, exactly as the message path.
GPU resolution composes: the resolved pci_hosts land in the confidential config.
SAFETY-CRITICAL: a missing/unresolved firmware aborts the create with a typed
error; there is no fallback to a plain QemuVMConfiguration, so a confidential
VM can never boot unprotected.
* feat(orchestrator): route confidential instances through the spec path
_is_spec_eligible no longer excludes confidential instances, so they reach
build_create_vm_spec / supervisor.create_vm. create_vm_execution and
start_persistent_vm detect awaiting_confidential_init after create and skip the
wait-for-running barrier and port-forward setup (the VM is not up: only the
owner starts it via /confidential/initialize), mirroring the message path.
* test(supervisor): sort imports in confidential spec pool create tests
* feat(conf): default instances to QEMU hypervisor
Instances are QEMU-only. Change INSTANCE_DEFAULT_HYPERVISOR to QEMU and
drop the setup() override that forced instances to Firecracker when QEMU
support was disabled. Program Firecracker support is unaffected (programs
do not read INSTANCE_DEFAULT_HYPERVISOR).
* refactor: drop the Firecracker-instance concept
Instances are QEMU-only; a Firecracker instance never existed as a real
product concept. This removes the vestigial create path and unblocks
making the agent fully pool-free.
- models.py: remove the hypervisor==firecracker branches in
VmExecution.prepare()/create(); instances always build QEMU resources
and controllers. The hypervisor property now returns QEMU for instances
unconditionally (programs still resolve to Firecracker).
- delete the now-dead AlephFirecrackerInstance / AlephInstanceResources
module and drop them from the firecracker and orchestrator.vm exports.
AlephFirecrackerProgram (the legitimate Firecracker user) is untouched.
- run._is_spec_eligible: every InstanceContent is now spec-eligible, so
all instances reach build_create_vm_spec. An explicit Firecracker
instance is rejected there with InvalidBackendError (instances are
QEMU-only), rather than silently falling back to the legacy create path.
- translate.build_create_vm_spec: clearer QEMU-only rejection message.
- tests: remove the FC-instance test (test_instance.py, covered for QEMU
by test_qemu_instance.py); rewrite the routing test to assert an FC
instance is rejected via the spec path; add test_instance_hypervisor.py
asserting an instance always resolves to QEMU.
The legacy create_a_vm bridge and _engine_pool are intentionally left in
place; removing them is the next step.
* feat(supervisor): consume owner GPU reservation engine-side
Add owner_address to CreateVmSpec and populate it from message.address in
build_create_vm_spec. create_vm_from_spec now consumes the owner's own GPU
reservation and skips other users' valid reservations, so the engine fully
owns reservation handling. Drop the now-unused VmPool.release_user_reservations.
* refactor(agent): delete create_a_vm pool bridge from run.py
Remove the _engine_pool helper and the legacy create_a_vm fallback branch in
create_vm_execution. Every supported content type now routes through the
Supervisor abstraction: programs through the spec program path, instances
(plain, GPU, confidential) through the spec path. Unsupported content raises a
clear HTTPBadRequest instead of falling back to a pool. Drop the now-dead
VmPool import and the _forget pool cleanup helper.
* test(agent): tighten pool-free guard to forbid all pool tokens
Extend the forbidden-token list to cover create_a_vm, _engine_pool,
supervisor.pool and direct pool imports alongside the existing require_vm_pool /
vm_pool / .executions tokens, and assert they appear in none of the agent's
view/run/resources modules. The agent is now provably pool-free.
* style: apply ruff format and isort across Phase 1 changes
* refactor(migration): fold stop_vm_for_export into the standard stop_vm
The export-time stop no longer needs a dedicated supervisor method: stop_vm
already performs a graceful, disk-quiescing powerdown. stop_vm -> execution.stop()
stops the controller unit and blocks on wait_for_controller_stopped() until the
controller's SIGTERM handler has ACPI-powered the guest down and QMP-quit QEMU
with a cache flush, so the exported overlay is consistent once stop_vm returns.
The export runner now calls stop_vm and derives the volumes dir from
settings.PERSISTENT_VOLUMES_DIR itself (the same path the import runner stages
into). With stop_vm_for_export gone, MigrationOps holds no method and is removed
from the Supervisor aggregate; migration rides the standard lifecycle RPCs end
to end.
* fix(confidential): apply the requested SEV policy on both create paths
The requested policy (message.environment.trusted_execution.policy, schema-
defaulted to NO_DBG so always present) was dropped: both create paths hardcoded
NO_DBG. Now the supervisor holds no opinion and applies the requested policy
verbatim.
- spec path: build_create_vm_spec sets TeeConfig.policy from
trusted_execution.policy (was hardcoded NO_DBG).
- message path: VmExecution.create() passes confidential_policy from
trusted_execution.policy (was omitted, defaulting to NO_DBG).
- AlephQemuConfidentialInstance.confidential_policy loses its NO_DBG default and
becomes required, since both construction sites now pass it explicitly. The
engine (qemu_build / qemuvm) already applies the policy with no clamp.
* fix(supervisor): resolve mypy errors (grpc_server Path import, test await_args)
* fix(pool): admit spec programs against the program memory bucket
check_spec_admission hardcoded the instance bucket, so a program routed
through the spec create path was charged against the instance ceiling.
On a small host where physical - host_reserved - program_reserved <= 0
that ceiling is 0, refusing every program (the droplet runtime health
check 503'd on a 3915 MiB host). Select the bucket from the spec backend
(QEMU=instance, Firecracker=program), mirroring check_admission.
* fix(supervisor): preserve confidential inject_secret success body
The pre-Phase-1 operate_confidential_inject_secret endpoint returned the
QMP query-status dict as its `status` field, so the happy-path body was
{"status": {"status": "running", "running": true, "singlestep": false}}.
Phase 1 replaced that with a flat {"status": "ok"}, a breaking change in
both value and shape, while the supervisor still issued query-status only
to discard the result.
Reproduce the old running body in the endpoint (inject_secret returns only
on the success path, since a QMP failure raises, so the VM has resumed) and
drop the now-useless query_status() round-trip from LocalSupervisor.
* fix(migration): resolve import hypervisor like the create path
run_import rejected any instance whose message omitted environment.hypervisor
(the CLI never sets it) because it fell back to Firecracker, while the create
path (translate.build_create_vm_spec) falls back to INSTANCE_DEFAULT_HYPERVISOR,
now QEMU. The source happily exported (it checks the live QEMU backend) but the
destination import aborted with "Migration only supported for QEMU instances"
before create_vm, so migrated VMs never booted on the new CRN and the testnet
migration test timed out waiting for re-dispatch.
Use the same INSTANCE_DEFAULT_HYPERVISOR fallback so the two paths agree. Adds
a regression test covering a message with hypervisor unset.
* fix(migration): set up port forwards on the destination after import
A migrated instance booted on the destination but never got its host port
forwards: create_vm_from_spec only reloads persisted mappings (empty on a
fresh destination), and run_import stopped at create_vm. The normal create
path runs a post-create tail (wait-until-running + apply the agent's resolved
port forwards, always including SSH/22); migration skipped it, so mapped_ports
stayed empty, the VM was unreachable, and the testnet migration test timed out
waiting for the re-dispatched VM's port 22.
Extract that tail into run.finish_instance_create and call it from both the
normal create path and run_import, so a migrated instance is wired identically
to a freshly created one. Adds a regression assertion that run_import invokes
the tail.
#981) * docs: design for gRPC-only supervisor (full process split) Two-process split as the only supported deployment: a supervisor daemon owning the pool and serving the Supervisor interface over gRPC, and an agent that reaches VMs exclusively through that interface. Key decisions: - rename InProcessSupervisor -> LocalSupervisor (the pool-backed engine) - agent keeps a thin embedded seam for dev/tests only - fold admission + GPU reservation into create_vm (atomic) - HAProxy/L7 domain mapping moves fully to the agent (fed by list_vms), out of the pool; the supervisor stays L3/L4 - interface-first, two phases: decouple agent from pool, then transport and two-service packaging * docs: Phase 1 implementation plan for gRPC-only supervisor Decouple the agent from VmPool. P1.1 (rename + wiring) and P1.6a (recreate_network) are ready to build; backup/restore, migration and confidential measurement are blocked on three contract decisions noted in the plan. * Rename InProcessSupervisor to LocalSupervisor (module local.py) * Update production references to LocalSupervisor * Update supervisor tests to LocalSupervisor * Extract build_supervisor factory for explicit agent wiring * Remove throwaway rename smoke test * Add recreate_network to the Supervisor interface * Implement recreate_network in LocalSupervisor * Route recreate_network endpoint through the supervisor * feat(haproxy): add agent-side domain-mapping sync driven by supervisor Introduce orchestrator/haproxy_sync.py with sync_domain_mappings(supervisor), which derives each local VM's IP from VmInfo.ipv4.address and pushes the domain->IP map to HAProxy. Add build_map_entries_from_vm_ips and write_entries_to_backend to haproxy.py as the low-level building blocks. * refactor(haproxy): route domain-mapping callers through agent sync Replace pool.update_domain_mapping() calls in the orchestrator views and tasks with sync_domain_mappings(supervisor): regenerate_proxy, notify_allocation, operate_update, the periodic resync task, and the domains-aggregate handler. The supervisor's list_vms() output now drives HAProxy, not the in-process pool. * refactor(haproxy): remove pool HAProxy coupling, agent owns domain sync Delete VmPool.update_domain_mapping and its load-time call; the agent now seeds an initial forced sync in periodic_domain_resync at startup. Drop the dead instances-based helpers from haproxy.py (fetch_list_and_update, update_backends, _build_map_entries, _resolve_vm_ip), keeping fetch_list and the low-level socket/map helpers. Add a guard test. * docs: record resolved contract decisions (backup, migration, confidential) * feat(supervisor): extend Measurement with SEV info and launch measure Add a SevInfo dataclass mirroring the seven QEMU query-sev fields and carry it (plus the base64 launch_measure) on Measurement, defaulted so existing callers and the proto path keep working. This lets the confidential measurement endpoint preserve its current response shape once the logic moves into the engine. * feat(supervisor): implement confidential ops in LocalSupervisor Move the initialize/measurement/inject-secret logic out of the agent endpoints and into the engine: write the SEV session certificates and start the controller service, query the launch measurement plus the seven query-sev fields, and inject the secret then resume the guest. The confidential ops are no longer stubs, so drop them from the conformance STUB_METHODS set. * feat(agent): delegate confidential endpoints to the supervisor Rewire operate_confidential_initialize, _measurement and _inject_secret to call only the Supervisor interface: the initialize guards now read supervisor.get_vm (status + confidential_mode) instead of the pool, and the three endpoints no longer touch require_vm_pool, the pool or QemuVmClient. The measurement response is preserved byte-for-byte as {"sev_info": {7 fields}, "launch_measure": ...}; inject_secret now returns {"status": "ok"} since the void supervisor method cannot carry the post-injection QMP status (the agreed minor contract change). * Add check_spec_admission for the spec create path * Fold capacity admission into create_vm_from_spec atomically * Remove agent-side admission in notify_allocation; surface boundary 503 * Route reserve_resources through the Supervisor interface * fix(supervisor): derive measurement tee_backend from the VM config, not a hardcoded SEV The TEE generation is determined by the VM's confidential configuration (an input the supervisor already holds via the execution policy), so get_measurement now derives tee_backend through _confidential_mode instead of hardcoding TeeBackend.SEV. TeeBackend.SEV still covers SEV and SEV-ES (refined by sev_info.policy); SEV-SNP maps distinctly. * Enrich BackupOps engine: include_volumes, metadata, restore_from_image Grow the supervisor backup interface so the agent can drop its pool-backed backup/restore logic without any feature regression: - BackupInfo carries checksum, volumes and source_sizes for completed archives, so the agent builds the HTTP body and download sidecar headers from engine metadata. - start_backup gains include_volumes: the engine archives the VM's non-read-only persistent volumes alongside the rootfs. - restore_from_image(vm_id, image_path, max_virtual_size_bytes) restores from a QCOW2 already staged on a host path (uploaded image or downloaded volume): the engine validates, size-checks, swaps the rootfs and restarts. - GrpcSupervisor stubs restore_from_image NotImplementedError (wired in Phase 2). * Route backup/restore endpoints through the supervisor Rewire the five backup/restore endpoints off the VmPool: they now reach VM and disk work only through the Supervisor interface, keeping HTTP-only concerns (owner auth, presigned download URLs and their verification, multipart/JSON parsing, staging uploaded bytes, sidecar response headers). - operate_backup calls start_backup(quiesce_guest, include_volumes): 202 while RUNNING, metadata body (sourced from BackupInfo) plus a presigned URL when COMPLETE; InvalidBackend -> 400, InsufficientResources -> 507. - operate_backup_status reports via list_backups. - operate_backup_download verifies the presigned URL, streams from download_backup, and sets Content-Length / X-Backup-Checksum / X-Source-Size from the backup metadata. - operate_backup_delete delegates to delete_backup. - operate_restore stages the uploaded QCOW2 or the volume_ref download to a temp path, then calls restore_from_image; qemu-img rejection and oversized disks surface as 400. Drops the agent-side BackupState background-task tracking, the inline QemuVmClient/fsfreeze logic, require_vm_pool and pool reads. Adds a guard test asserting the five endpoints reference no pool symbols. * Update restore-rejects-invalid-image test for the supervisor path The test exercised the removed operator._do_restore / _parse_restore_upload internals. Rewire it through operate_restore against a supervisor whose restore_from_image raises InvalidBackendError (what the engine's qemu-img check does on a non-QCOW2), still asserting the 4xx (not 500). * test: stage volume_ref restore under tmp_path * Add P2P-migration disk/VM seam to the Supervisor interface The agent's P2P pull protocol keeps the network transport (compress, hash, serve, download, rebase); add the four MigrationOps methods that own the parts touching the VmExecution and the pool, which the agent must not reach into directly: - stop_vm_for_export: graceful stop + locate the persistent-volumes dir - restart_after_failed_export: bring the VM back up after a failed export - create_migrated_vm: create a persistent VM from a staged instance message - release_migrated_vm: stop + forget a VM that has migrated away Implemented for real in LocalSupervisor, stubbed NotImplementedError in GrpcSupervisor (wired in Phase 2). Surface count 32 to 36. * Route the P2P migration endpoints through the supervisor Thread the supervisor (not the VmPool) through the migration runner and the three migration endpoints. The P2P pull protocol stays the agent-side network transport: the on-host HTTP disk transfer, tokens, ExportJob/ImportJob states and request/response shapes are unchanged. Only the disk/VM work moves behind the supervisor: - migration_export gates via supervisor.get_vm (VmNotFoundError = 404) and reads status/backend/confidential_mode off the returned VmInfo; run_export asks the supervisor to stop the VM and report its volumes dir, then the agent compresses and serves the files it finds there. - migration_import gates the already-running check via supervisor.get_vm; run_import downloads and rebases the peer disks (network transport) then calls supervisor.create_migrated_vm. The dest-dir rmtree decision is made against a single supervisor.get_vm probe instead of pool.executions. - migration_cleanup releases the source VM via supervisor.release_migrated_vm (engine: stop_vm + forget_vm) instead of pool.stop_vm/forget_vm. The three endpoints (and the runner) now contain no require_vm_pool / vm_pool / pool.create_a_vm / pool.executions references; a guard test asserts this via inspect.getsource. * Delete the directory-based migration flow Remove the provisional directory-based export_vm / import_vm / get_migration_status from the MigrationOps ABC, LocalSupervisor, GrpcSupervisor and the gRPC server handlers, and the migrate.py manifest module they relied on. Migration now rides the standard lifecycle RPCs; only the export-time graceful stop survives as a MigrationOps method (it needs a guest powerdown stop_vm does not do). The orphan ExportVm / ImportVm / GetMigrationStatus proto RPCs stay in supervisor.proto with a note to drop them in the Phase 2 proto pass, so the generated bindings are unchanged and check_proto_clean stays green. Surface count 36 to 30. * Route migration import through the standard create_vm RPC run_import now builds a CreateVmSpec from the fetched instance message with build_create_vm_spec (the same path run.create_vm_execution uses for a normal persistent instance) and calls supervisor.create_vm(spec) instead of the bespoke create_migrated_vm. The spec's rootfs path is PERSISTENT_VOLUMES_DIR/<vm_hash>/rootfs.qcow2, exactly where the download+rebase already staged the overlay; build_create_vm_spec adopts an already-present host-persistence overlay rather than recreating it, so create_vm reuses the staged disk with no re-download (covered by test_import_spec_build_reuses_staged_overlay). The failed-export restart rides the standard start_vm RPC instead of restart_after_failed_export. The disk/VM seam methods were removed in the previous commit; this wires the runner onto the lifecycle RPCs. * Route migration cleanup through the standard delete_vm RPC migration_cleanup drops the migrated-away source through delete_vm(wipe= False) instead of the bespoke release_migrated_vm. delete_vm stops the VM, forgets the definition and removes the controller config; wipe=False leaves the disks alone, which is correct for a source whose data now lives on the destination. The view tests assert create_vm and delete_vm are driven, mocking build_create_vm_spec on the import path. * docs: revise migration design to lifecycle RPCs (delete directory-based flow) * Add run_program_code to the Supervisor interface Persistent programs are served through the supervisor: the engine runs the request over the VM's guest channel (the agent cannot reach a supervisor-owned channel across the boundary). Stubbed in GrpcSupervisor for Phase 2. * Emit persistent program specs from build_program_create_vm_spec Thread the program message's on.persistent into the spec instead of rejecting persistent programs; a persistent program boots under systemd on the engine side. * Boot persistent programs through the Firecracker spec create path Drop the persistent-Firecracker guard in _create_firecracker_from_spec; VmExecution.start already branches on self.persistent to boot under systemd and wait for init. * Route persistent programs through the supervisor, drop the legacy paths Persistent on-request and on-event programs now create through the supervisor spec path (build_program_create_vm_spec + create_vm) and are served via supervisor.run_program_code; on-demand programs keep the agent-side program_client.run_code. The no-idle-reap finally behavior is preserved for persistent programs. Deletes _run_code_on_request_legacy and _run_code_on_event_legacy. * Final pool removal: the agent reaches VMs only through the supervisor Drop the pool parameter from the run.py entry points; the one remaining legacy create path (confidential / GPU / firecracker instances) reaches the embedded pool through supervisor.pool, None across the gRPC boundary. Stop exposing the raw pool to handlers (app["vm_pool"] -> private app["_engine_pool"], read only by the process-lifecycle hooks). Point resources.py at supervisor.get_host_info() for GPU inventory and available disk (HostInfo grows available_disk_bytes / gpu_inventory / available_gpus). Delete require_vm_pool and the dead get_execution_or_404. Add the capstone guard test asserting the agent view/run modules are pool-free. * feat(supervisor): carry GPU requests on CreateVmSpec Add device_id/model request fields to GpuSpec and populate spec.gpus from message.requirements.gpu in build_create_vm_spec. The spec now carries the GPU REQUEST (device_id/model); pci_host stays empty until the engine resolves a concrete host card. proto_convert keeps round-tripping pci_host/supports_x_vga only; the new fields default to empty (Phase 2 adds them to the proto). * feat(supervisor): resolve and reserve GPUs on the spec create path pool.create_vm_from_spec now resolves each GPU REQUEST (device_id, empty pci_host) on the spec to a concrete available host card, atomically under creation_lock, and rewrites the spec with the resolved pci_host so build_qemu_configuration and from_spec wire it through. The resolved HostGPUs are attached to the execution so uses_gpu() holds them against other creates and VmInfo reports them, exactly like the message path. The spec has no owner address, so it cannot honour this user's own pre-reservation. The agent clears its own reservation before calling create_vm (it holds message.address); any reservation still valid here belongs to another user and is skipped. Raises InsufficientResourcesError when a request cannot be satisfied. * feat(orchestrator): route GPU instances through the spec create path Drop the GPU exclusion from _is_spec_eligible so GPU QEMU instances reach build_create_vm_spec and the engine's GPU-capable spec path. Before driving create_vm for a GPU spec, the agent releases its own GPU pre-reservation via the new pool.release_user_reservations(content.address): the engine's spec-path resolution carries no owner address and skips any reservation still standing, so the owner's own hold must be cleared first or it would block the create. In-process only; owner-aware reservation over the gRPC boundary is a later slice. * feat(supervisor): populate spec.tee for confidential instances in translate build_create_vm_spec now resolves the trusted_execution firmware ref to a host path and emits a TeeConfig (backend SEV, NO_DBG policy, per-VM session dir, firmware path) instead of rejecting confidential instances. The launched policy is NO_DBG, faithfully reproducing the message path which never reads trusted_execution.policy. Adds TeeConfig.firmware_path (defaulted so proto_convert keeps round-tripping). * feat(supervisor): build confidential launch on the spec create path pool.create_vm_from_spec now builds a QemuConfidentialVMConfiguration for a spec carrying spec.tee (via the new build_qemu_confidential_configuration, mirroring AlephQemuConfidentialInstance.configure), and creates an AlephQemuConfidentialInstance through VmExecution.from_spec/create. The VM is left in awaiting_confidential_init: start(write_config=False) does not enable_and_start the controller because is_confidential is True, so the owner drives the launch through initialize_confidential, exactly as the message path. GPU resolution composes: the resolved pci_hosts land in the confidential config. SAFETY-CRITICAL: a missing/unresolved firmware aborts the create with a typed error; there is no fallback to a plain QemuVMConfiguration, so a confidential VM can never boot unprotected. * feat(orchestrator): route confidential instances through the spec path _is_spec_eligible no longer excludes confidential instances, so they reach build_create_vm_spec / supervisor.create_vm. create_vm_execution and start_persistent_vm detect awaiting_confidential_init after create and skip the wait-for-running barrier and port-forward setup (the VM is not up: only the owner starts it via /confidential/initialize), mirroring the message path. * test(supervisor): sort imports in confidential spec pool create tests * feat(conf): default instances to QEMU hypervisor Instances are QEMU-only. Change INSTANCE_DEFAULT_HYPERVISOR to QEMU and drop the setup() override that forced instances to Firecracker when QEMU support was disabled. Program Firecracker support is unaffected (programs do not read INSTANCE_DEFAULT_HYPERVISOR). * refactor: drop the Firecracker-instance concept Instances are QEMU-only; a Firecracker instance never existed as a real product concept. This removes the vestigial create path and unblocks making the agent fully pool-free. - models.py: remove the hypervisor==firecracker branches in VmExecution.prepare()/create(); instances always build QEMU resources and controllers. The hypervisor property now returns QEMU for instances unconditionally (programs still resolve to Firecracker). - delete the now-dead AlephFirecrackerInstance / AlephInstanceResources module and drop them from the firecracker and orchestrator.vm exports. AlephFirecrackerProgram (the legitimate Firecracker user) is untouched. - run._is_spec_eligible: every InstanceContent is now spec-eligible, so all instances reach build_create_vm_spec. An explicit Firecracker instance is rejected there with InvalidBackendError (instances are QEMU-only), rather than silently falling back to the legacy create path. - translate.build_create_vm_spec: clearer QEMU-only rejection message. - tests: remove the FC-instance test (test_instance.py, covered for QEMU by test_qemu_instance.py); rewrite the routing test to assert an FC instance is rejected via the spec path; add test_instance_hypervisor.py asserting an instance always resolves to QEMU. The legacy create_a_vm bridge and _engine_pool are intentionally left in place; removing them is the next step. * feat(supervisor): consume owner GPU reservation engine-side Add owner_address to CreateVmSpec and populate it from message.address in build_create_vm_spec. create_vm_from_spec now consumes the owner's own GPU reservation and skips other users' valid reservations, so the engine fully owns reservation handling. Drop the now-unused VmPool.release_user_reservations. * refactor(agent): delete create_a_vm pool bridge from run.py Remove the _engine_pool helper and the legacy create_a_vm fallback branch in create_vm_execution. Every supported content type now routes through the Supervisor abstraction: programs through the spec program path, instances (plain, GPU, confidential) through the spec path. Unsupported content raises a clear HTTPBadRequest instead of falling back to a pool. Drop the now-dead VmPool import and the _forget pool cleanup helper. * test(agent): tighten pool-free guard to forbid all pool tokens Extend the forbidden-token list to cover create_a_vm, _engine_pool, supervisor.pool and direct pool imports alongside the existing require_vm_pool / vm_pool / .executions tokens, and assert they appear in none of the agent's view/run/resources modules. The agent is now provably pool-free. * style: apply ruff format and isort across Phase 1 changes * refactor(migration): fold stop_vm_for_export into the standard stop_vm The export-time stop no longer needs a dedicated supervisor method: stop_vm already performs a graceful, disk-quiescing powerdown. stop_vm -> execution.stop() stops the controller unit and blocks on wait_for_controller_stopped() until the controller's SIGTERM handler has ACPI-powered the guest down and QMP-quit QEMU with a cache flush, so the exported overlay is consistent once stop_vm returns. The export runner now calls stop_vm and derives the volumes dir from settings.PERSISTENT_VOLUMES_DIR itself (the same path the import runner stages into). With stop_vm_for_export gone, MigrationOps holds no method and is removed from the Supervisor aggregate; migration rides the standard lifecycle RPCs end to end. * fix(confidential): apply the requested SEV policy on both create paths The requested policy (message.environment.trusted_execution.policy, schema- defaulted to NO_DBG so always present) was dropped: both create paths hardcoded NO_DBG. Now the supervisor holds no opinion and applies the requested policy verbatim. - spec path: build_create_vm_spec sets TeeConfig.policy from trusted_execution.policy (was hardcoded NO_DBG). - message path: VmExecution.create() passes confidential_policy from trusted_execution.policy (was omitted, defaulting to NO_DBG). - AlephQemuConfidentialInstance.confidential_policy loses its NO_DBG default and becomes required, since both construction sites now pass it explicitly. The engine (qemu_build / qemuvm) already applies the policy with no clamp. * fix(supervisor): resolve mypy errors (grpc_server Path import, test await_args) * fix(pool): admit spec programs against the program memory bucket check_spec_admission hardcoded the instance bucket, so a program routed through the spec create path was charged against the instance ceiling. On a small host where physical - host_reserved - program_reserved <= 0 that ceiling is 0, refusing every program (the droplet runtime health check 503'd on a 3915 MiB host). Select the bucket from the spec backend (QEMU=instance, Firecracker=program), mirroring check_admission. * docs: Phase 2 implementation plan for gRPC-only supervisor * feat(supervisor): carry tee firmware, gpu request, owner, include_volumes over the wire * fix(supervisor): refresh stale Phase-1 wire docstrings; cover new fields in full round-trip * feat(supervisor): carry SEV info and launch measure over the wire * feat(supervisor): carry backup archive metadata over the wire * feat(supervisor): reconcile HostInfo hardware and reservation fields over the wire * feat(supervisor): wire recreate_network over gRPC * test(supervisor): document the mock gRPC fixture and simplify the ABC stub * feat(supervisor): wire restore_from_image over gRPC * feat(supervisor): wire run_program_code over gRPC (msgpack scope) * fix(supervisor): scale run_program_code gRPC deadline with the request timeout * feat(supervisor): reserve_resources takes a message-free resources DTO over gRPC * refactor(pool): unify capacity admission, drop orphaned reserve_resources; A8 cleanups * refactor(supervisor): drop the orphan directory-based migration RPCs * test(supervisor): guard that the gRPC surface is complete * feat(packaging): split supervisor daemon and agent into two systemd units (Phase 2 B1-B4) - aleph-vm-supervisor.service becomes the daemon (python3 -m aleph.vm.supervisor, owns the pool, serves gRPC). - New aleph-vm-agent.service runs the HTTP orchestrator (python3 -m aleph.vm.orchestrator --print-settings), ordered After/Wants the supervisor. - preinst stops both units (agent then supervisor); postinst enables both, restarts the daemon, waits for the socket, then restarts the agent; prerm disables/stops both. - supervisor.env sets ALEPH_VM_SUPERVISOR_GRPC_SOCKET so production runs the split; code defaults stay None for dev/tests. * test(supervisor): pin embedded-by-default, gRPC-when-socket-set wiring * refactor(supervisor): drop now-unused migration DTOs from types * refactor(pool): remove the dead legacy message create path * docs: refresh comments that named the removed create_a_vm/check_admission * fix(supervisor): preserve confidential inject_secret success body The pre-Phase-1 operate_confidential_inject_secret endpoint returned the QMP query-status dict as its `status` field, so the happy-path body was {"status": {"status": "running", "running": true, "singlestep": false}}. Phase 1 replaced that with a flat {"status": "ok"}, a breaking change in both value and shape, while the supervisor still issued query-status only to discard the result. Reproduce the old running body in the endpoint (inject_secret returns only on the success path, since a QMP failure raises, so the VM has resumed) and drop the now-useless query_status() round-trip from LocalSupervisor. * fix(migration): resolve import hypervisor like the create path run_import rejected any instance whose message omitted environment.hypervisor (the CLI never sets it) because it fell back to Firecracker, while the create path (translate.build_create_vm_spec) falls back to INSTANCE_DEFAULT_HYPERVISOR, now QEMU. The source happily exported (it checks the live QEMU backend) but the destination import aborted with "Migration only supported for QEMU instances" before create_vm, so migrated VMs never booted on the new CRN and the testnet migration test timed out waiting for re-dispatch. Use the same INSTANCE_DEFAULT_HYPERVISOR fallback so the two paths agree. Adds a regression test covering a message with hypervisor unset. * fix(migration): set up port forwards on the destination after import A migrated instance booted on the destination but never got its host port forwards: create_vm_from_spec only reloads persisted mappings (empty on a fresh destination), and run_import stopped at create_vm. The normal create path runs a post-create tail (wait-until-running + apply the agent's resolved port forwards, always including SSH/22); migration skipped it, so mapped_ports stayed empty, the VM was unreachable, and the testnet migration test timed out waiting for the re-dispatched VM's port 22. Extract that tail into run.finish_instance_create and call it from both the normal create path and run_import, so a migrated instance is wired identically to a freshly created one. Adds a regression assertion that run_import invokes the tail. * ci: trigger CI for #981 (base now dev; carries #980 fixes) * test(supervisor): declare FakePool reservation attrs for mypy CI now runs mypy on tests/ for this branch (it never did while #981 targeted a non-dev base). FakePool's check_capacity/reserve_gpus were assigned dynamically in test_reserve_resources_*; declare them so mypy's attr-defined check passes. * ci(deb): also dump aleph-vm-agent journal on droplet-test failure The HTTP API (port 4020, /control/*) is served by the agent in the two-service split; the failure handler only dumped the supervisor unit, leaving the agent side (where /control/allocations 503s) invisible. Capture both. * ci(deb): dump aleph-vm-agent journal in the post-test log export The after-failure export only captured the supervisor unit; the agent (HTTP API, /control/allocations) journal is where instance-create errors surface in the two-service split. * fix(pool): give SpecProgramResources a get_disk_usage_delta The shared capacity-admission path (calculate_available_disk) sums get_disk_usage_delta() over every execution's resources. SpecProgramResources (spec-driven Firecracker programs) never implemented it, so once a spec program was running, any subsequent create_vm crashed with AttributeError and the agent returned 503 on /control/allocations. Spec disk admission is deferred (DiskSpec carries no size yet), so it reserves nothing: return 0, mirroring check_spec_admission's disk_mib=0. * ci(deb): set SUPERVISOR_GRPC_SOCKET in the droplet supervisor.env The droplet test writes its own supervisor.env and installs with --force-confold, which keeps it over the packaged one - so it must carry the two-service socket the package ships (packaging/aleph-vm/etc/aleph-vm/supervisor.env). Without it the agent runs embedded next to the supervisor daemon, two pools fight over the VM tap ('File descriptor in bad state'), and program networking endpoints return 503. * fix(confidential): wait for the VM record on init-session instead of 404 A confidential instance is created via the allocation path, where create_vm (build config + start to the awaiting-init state) takes ~20s. The scheduler exposes placement earlier, so the owner's one-shot init-session (/confidential/initialize) can arrive mid-create — before create_vm_execution writes the agent registry record — and got a 404 (the testnet confidential test fails here). The pre-refactor path looked the VM up in the pool (registered early) and didn't race. Wait for the record (written exactly when create completes and the VM is awaiting init) up to a bounded cap instead of 404ing. * fix(confidential): don't reject init-session on a VM awaiting init After the record-wait fix, init-session got past the 404 but hit a 400 'vm_running': the endpoint rejects status in (RUNNING, BOOTING), and a spec-path confidential VM awaiting its owner reports BOOTING (start() sets starting_at without launching the controller). That VM is exactly what init-session initializes, so exempt awaiting_confidential_init from the running check. * fix(confidential): set up port forwards after secret injection A confidential VM returns awaiting_confidential_init from create, so the normal create-completion (run.finish_instance_create: wait-running + port forwards) is skipped. The VM only boots once the owner injects the secret, after which nothing mapped its host ports — so SSH/22 never appeared and the testnet confidential test timed out waiting for port 22. Reconcile the port forwards in the inject-secret endpoint now that the VM is running, mirroring the normal create path.
The pool keyed executions by aleph_message's ItemHash, the last aleph_message symbol in pool.py. Replace it with the message-free VmId (a NewType str) from supervisor.types, drop the import so the pool is fully message-free, and retype VmExecution.vm_hash to match so the pool's dict key type stays honest. ItemHash is a str subclass and VmId a str NewType, so stored keys and lookups stay compatible; the spec path now stores spec.vm_id directly instead of re-wrapping it. models.py keeps its message-content imports (ExecutableContent and friends), which drive the message path and are out of scope here.
…s; delete dead AlephFirecrackerProgram (#983) * docs: plan message-free VmExecution and controller layer * docs: drop seconds from HardwareResources, fold in AlephFirecrackerProgram deletion * feat(supervisor): add message-free HardwareResources DTO * refactor(supervisor): make VmExecution spec-only, drop aleph_message * refactor(controllers): delete dead message-path AlephFirecrackerProgram * refactor(controllers): take HardwareResources instead of MachineResources * test(supervisor): migrate VmExecution tests to spec path; drop message-path-only tests Add a shared CreateVmSpec factory in tests/supervisor/conftest.py (make_spec, parameterized over backend/tee/gpus/persistent/memory_mib/vcpus/internet) and route the migrated tests through VmExecution.from_spec(...). Converting the shared _make_execution/create_mock_execution/make_execution helpers in test_drain, test_wait_for_controller, test_firewall and test_run to build a spec cleared their breakage in one place each. Converted (still-live spec behavior preserved): - test_drain, test_wait_for_controller: _make_execution now builds from a spec (drain waiting, controller-readiness polling). - test_firewall: create_mock_execution builds from a spec (recreate_port_redirect reads only mapped_ports + the mocked vm.tap_interface). - test_run: make_execution builds a spec, confidential via TeeConfig (is_awaiting_confidential_init / is_running). - test_views: the executions-list tests build spec executions; vm_network uses VmType.instance directly. - test_supervisor_reattach, test_supervisor_spec_pool_create: .message-is-None assertions replaced by vm_spec identity checks. - test_supervisor_spec_execution: dropped .message/.original/.hypervisor and the save_record patch; kept the configure-not-awaited spec assertion. - test_supervisor_inprocess_query: make_execution gained is_program (backend now derives from is_program, not hypervisor); the no-is_instance-field test drives a program execution to assert the FIRECRACKER backend. Deleted (message-path-only / removed surface): - tests/supervisor/test_instance_hypervisor.py: tested the removed VmExecution.hypervisor property entirely via the message branch. - test_host_gpu_detail.py::test_prepare_gpus_retains_detail: exercised the removed prepare_gpus on a message-built execution; spec-path GPU detail is covered by test_supervisor_spec_pool_create. - test_port_mappings.py::test_fetch_port_redirect_config_does_not_call_get_port_mappings: exercised the removed fetch_port_redirect_config_and_setup via MessageSpec. - test_execution.py message/program tests (test_create_execution, test_create_execution_online, test_create_execution_from_fake_message, test_create_execution_volume_with_no_name, test_create_execution_legacy): built message-driven AlephFirecrackerProgram executions; both the program message path and AlephFirecrackerProgram are removed. Kept the two structural has_no_*_api tests. This removes the former jailman-chown baseline (those were the message/program executions). * docs: reframe controllers as agent-side; supervisor split is separate * refactor(controllers): drop dead get_volumes_for_program helper Its only caller was the deleted AlephFirecrackerProgram.load_configuration; the agent reimplements the logic as program_client.build_code_and_volumes. * test(qemu): migrate test_qemu_instance to the spec path #983 made VmExecution spec-only (no message=/original= constructor), but its test-migration commit missed test_qemu_instance.py, so CI failed with 'VmExecution.__init__() got an unexpected keyword argument message'. Build the CreateVmSpec via build_create_vm_spec() and construct with from_spec(), matching the other migrated supervisor tests and the production create path. * test(qemu): drive test_qemu_instance through the production spec path The first migration constructed via from_spec but still called start() (write_config=True), hitting the legacy in-process configure() -> _create_cloud_init_drive, which reads message_content (None on the spec path) -> AttributeError on authorized_keys. Production (VmPool.create_vm_from_spec) writes the controller config via build_qemu_configuration(spec) and starts with write_config=False. Mirror that: build+save the config, then start(write_config= False). The cloud-init keys come from spec.ssh_authorized_keys, message-free. * test(port-mappings): make AsyncSessionMaker redirect order-independent metrics.AsyncSessionMaker is a bare module annotation until setup_engine() binds it, so monkeypatch.setattr(raising=True) only worked when an earlier test in the session had run setup_engine(). #983's test changes shifted ordering and left it unbound at fixture time -> 'metrics has no attribute AsyncSessionMaker' (ERROR at setup x4). Use raising=False to redirect regardless of prior setup.
… settings slice (#1016)
There is no wire protocol to stay compatible with: both stubs regenerate from this file in the same deb, and the only mixed-version exposure is seconds of restart ordering during an upgrade, where failed calls retry. Dropping the reserved blocks (BACKEND_QEMU_SEV, DiskConfig.mount, the VmInfo network/gateway pairs and is_instance) and renumbering VmInfo 9-20 means the upcoming Rust contract is born without historical holes. ReserveResourcesRequest.is_instance stays: a reservation precedes the VM and carries no Backend, so the memory-accounting bucket cannot be derived and the client must state it.
Not ported to Rust, so removed rather than carried: - Snapshots: every concrete controller sets support_snapshot = False and SNAPSHOT_FREQUENCY defaults to 0, so snapshot_manager.py, snapshots.py, the pool/models wiring, the SNAPSHOT_* settings and the storage helpers (create_volume_snapshot, compress_volume_snapshot) are all unreachable. The devmapper snapshot in storage.py is a different mechanism and stays. - program.py: AlephProgramResources only survived in a models.py type union (nothing instantiates it since the spec path took over), and its legacy FileTooLargeError is raised nowhere (everything raises the supervisor_interface.errors one), which made the error_mapping branch for it unreachable. The guest-protocol re-exports already live in aleph.vm.program_config; the one test importing through program.py is repointed there. Also fixes the VmExecution.resources annotation, which listed AlephQemuConfidentialInstance (a controller) where prepare() actually assigns AlephQemuConfidentialResources.
Capacity reservation policies are now handled agent-side, while the supervisor now only enforces physical constraints.
…Info) (#1021) Start of the Rust port. - rust/ cargo workspace (stable 1.90): supervisor-proto (tonic-build on the frozen proto/supervisor.proto) and supervisor-daemon (binary aleph-vm-supervisor). - Health and GetHostInfo at field-for-field parity with the Python LocalSupervisor, lspci GPU inventory ported literally from resources.py; every other RPC returns UNIMPLEMENTED carrying the same ErrorDetail trailer the Python _abort sends. - Socket lifecycle matching the Python daemon, plus day-one hardening: bind under umask 0o077 with chmod 0700 backstop, identity-checked shutdown unlink (O_PATH inode pinning, never removes a newer instance's socket), 5s shutdown grace matching server.stop(grace=5). - supervisor_interface/client.py pins grpc.default_authority=localhost: gRPC C-core derives the HTTP/2 :authority for unix targets from the percent-encoded socket path, which hyper/h2 rejects, so no existing client could reach a Rust daemon. Both daemons ignore the value. - Conformance suite seed (tests/conformance, opt-in via ALEPH_VM_CONFORMANCE=1) driving the daemon with the production Python client, and a path-filtered cargo CI job (fmt, clippy -D warnings, test --locked, conformance). - docs/plans/rust-port-divergences.md: ledger of deliberate divergences from the Python daemon (10 entries). The daemon is not packaged or selectable yet; ALEPH_VM_SUPERVISOR_IMPL wiring ships with the packaging PR.
…1022) The Rust daemon now boots on a live EXECUTION_ROOT and serves its world read-only, mirroring the restarted Python daemon (design doc section 7, row 2; steps 1-4 of the boot sequence). World adoption: scan {EXECUTION_ROOT}/*-controller.json (sorted, keyed on the embedded vm_hash, duplicate-vm_index guard, WARN-skip on unparseable or non-regular files), controller unit states over zbus (5s method timeout; a bus failure at boot defers to live per-RPC queries instead of stamping VMs stopped), and port mappings from supervisor.sqlite3 via rusqlite with Python's dict-fold semantics. RPCs: GetVm, GetVmSpec, ListVms, ListPortForwards, GetLogs (journalctl, same matches and ordering as the sd-journal reader, plus a Rust-only 10k-line cap), and world-aware Health and GetHostInfo. GetVmSpec is asserted field-for-field against Python's spec_from_controller_configuration in the conformance suite. Python side: save_controller_configuration now writes atomically (temp file + os.replace); the in-place truncate could leave a config unreadable exactly during a daemon handoff. Verification: conformance fixtures generated by the real pydantic/SQLAlchemy models, byte-stable; both IPv6 allocators pinned against the Python allocators; divergence ledger grown to 17 entries. Gates: cargo fmt/clippy clean, 80 cargo tests, conformance 9/9, full Python suite green.
…#1023) After a supervisor daemon restart, load_persistent_executions reattaches running VMs from their on-disk controller configs but left VmExecution.gpus empty, so the cards a running guest physically holds looked free. Two consumers were wrong as a result: - GetHostInfo.available_gpus (VmPool.get_available_gpus) re-offered cards already passed through to adopted VMs after every restart. - The create-time mechanism check (_validate_spec_gpus) was blind to adopted VMs, so a new CreateVm could be handed a card a running VM owns: a real double-attach. _restore_running_execution_from_config now rebuilds execution.gpus from the config's vm_configuration.gpus (pci_host + supports_x_vga). A card still present in the pool inventory keeps its full HostGPU metadata; a card absent from it (removed hardware, changed vfio binding) is recorded as a bare HostGPU from the config alone, which keeps the accounting of the cards that do exist correct while still reporting the passthrough. The Rust port's world view already implements this exclusion; entry 14 of docs/plans/rust-port-divergences.md (od/rust-daemon-inc2 branch) tracks it as fix-in-python-later.
…launcher (#1024) The deb now builds rust/ (cargo, toolchain pinned by rust-toolchain.toml) and installs /opt/aleph-vm/bin/aleph-vm-supervisor next to a supervisor-launcher dispatch script; aleph-vm-supervisor.service execs the launcher, which picks the implementation from ALEPH_VM_SUPERVISOR_IMPL in /etc/aleph-vm/supervisor.env (default python). Cutover to Rust is a one-line env change and switching back is safe: both daemons read and write the same on-disk state. Build containers install rustup instead of the distro cargo (plain cargo ignores the toolchain pin, and jammy's rustc predates edition 2024), and protoc for tonic-build (upstream 21.12 on jammy, whose packaged 3.12 predates proto3 optional). The deb-content CI check asserts both new files. Proven by running the packaged binary and the launcher dispatch on clean ubuntu:22.04 and debian:12 containers. * docs(rust): flip ledger entry 14, the Python GPU reattach fix landed (#1023) The remaining gap moves to the Rust side: VmInfo.gpus for VMs adopted running must be rebuilt from the controller config in increment 3.
The package has shipped native x86_64 binaries for a long time (firecracker, jailer, sevctl, now the Rust supervisor daemon) while declaring Architecture: all, so dpkg would install it on any architecture and nothing would run. Artifact names are unaffected: dpkg-deb --build names the output explicitly.
…ftables) (#1026) * feat(rust): nftables rule engine, pure layer pinned to firewall.py captures Port firewall.py rule for rule: entity dedup (is_entity_present over supersets), base-chain discovery with the prerouting nat-vs-raw selection, initialize_nftables (both phases, including the duplicate add-table wart on an empty host), per-VM chain/masquerade/forward builders, port-redirect add/remove, remove_chain and the aleph-chain sweep, plus the two redirect predicates. The pure layer maps a ruleset snapshot to nftables JSON command batches so parity is testable without root: scripts/generate_rust_fixtures.py now captures the actual Python module's output (fetch/execute edges stubbed) into tests/fixtures/nftables.json, and a cargo test replays all eleven scenarios plus the predicate table against the same inputs. The apply edge is the NftExecutor seam: nft -j subprocess in production (same libnftables JSON dialect as the Python binding, -s -p matching its output options), a StaticRuleset fake for tests. * feat(rust): host mutation seams (systemd, tap, ndppd, port-mapping writes) - units.rs: extend the UnitStateSource trait with the SystemDManager mutations (start/stop/restart/enable/disable, GetUnitFileState, the per-unit ActiveState with the synthetic not-loaded/unknown values) plus the stop_and_disable / enable_and_start compounds, all on the existing zbus connection with its reconnect-once and 5s method timeout. A new FakeSystemd fake records mutations and flips states for lifecycle tests. - tap.rs: TapAssignment (the TapInterface surface the supervisor reads) and the TapBackend seam; production drives ip(8), same kernel operations and EEXIST tolerance as the pyroute2 path. - ndppd.rs: the NdpProxy map with the exact /etc/ndppd.conf rendering, the 0.5s debounced systemctl restart, and the update_service=false priming used at adoption; file/systemctl behind the NdppdEdge seam. - ports.rs: the write path against supervisor.sqlite3 with SQLAlchemy's exact DDL (byte-compared against the committed fixture) and datetime rendering, save/delete_port_mappings with the Python soft-delete semantics, and the host-port allocator (DB actives, per-candidate nftables re-fetch, TCP+UDP bind probe, rotating cursor). * feat(rust): persistent VM lifecycle, port-forward mutations, RecreateNetwork, adoption step 5 The increment 3 RPC surface, ported 1:1 from LocalSupervisor + VmPool + VmExecution (QEMU persistent instances only; ephemeral Firecracker programs are increment 4, confidential creation increment 6, both answering UNIMPLEMENTED with the established trailer shape): - CreateVm: idempotency on a live VM (identical spec returns it, a different one is ALREADY_EXISTS), the re-adoption of live-but-untracked controllers with the transitional-state refusal, the memory backstop and GPU attach validation under creation_lock, vm_index allocation skipping hidden VMs' claims, tap create (with the pre-existing-interface delete-and-recreate), ndppd range, per-VM nftables chains, the cloud-init seed (JSON body, semantically identical to the Python YAML), the atomically-written controller config (byte-identical to pydantic), enable_and_start plus the 30x2s wait with the crash-loop double check, and the persisted port-forward reload. Failures tear down and forget, leaving the config artifacts behind like Python. - StopVm/StartVm/RebootVm/ReinstallVm/DeleteVm with the exact Python stage stamps, graceful-stop wait (75x1s), port-redirect removal, tap/nftables teardown, store soft-deletes and the keep_port_mappings/wipe semantics; DeleteVm also honors deletes of hidden (failed-adoption) VMs like the Python discard_failed_reattach. - AddPortForward/RemovePortForward through the update_port_redirects diff (host ports always auto-allocated, matching the Python engine which ignores the requested host_port), persisting through the SQLAlchemy-shaped store. - RecreateNetwork: aleph-chain sweep, base re-initialize, per-VM chains and persisted redirects, same summary dict keys. - Adoption step 5 at boot: create-if-absent taps, primed ndppd map, per-VM chains, persisted redirects for VMs adopted running; a per-VM failure hides the VM like a failed Python reattach. - settings.check() slice at startup (/dev/kvm, hypervisor binaries, socket-path length, tool availability), plus the pool.setup() counterparts: store schema, forwarding sysctls, base nftables ruleset. - Ledger entry 14 closed: VmInfo.gpus is rebuilt from the inventory for VMs adopted running (post-#1023 Python) and set from the validated request at create. VmEntry grows the attachments, the original CreateVm spec (served by GetVmSpec and compared by the idempotency check, like a live Python daemon) and reserved-index bookkeeping; the service layer maps the closed RpcError vocabulary onto the same status codes and ErrorDetail trailers as grpc_server.py. * test(conformance): increment 3 lifecycle coverage, cloud-init parity, fixture drift guard - New lifecycle module: CreateVm rejection paths (increment staging for Firecracker/confidential, the memory backstop and GPU validation with their exact Python messages), the NOT_FOUND vocabulary of every mutation, StopVm idempotency on adopted-stopped VMs, the unprivileged NoBaseChainFound shape of AddPortForward, and DeleteVm's artifacts: definition sweep, wipe/keep_port_mappings semantics, soft-deleted rows read back through the actual SQLAlchemy models and a restarted daemon. - Cloud-init parity: the committed Rust seed fixtures must parse (JSON as a YAML subset) to exactly what encode_user_data / create_network_file / create_metadata_file produce for the same inputs. - nftables drift guard: the committed nftables.json oracle must equal what the current firewall.py produces (the generation payload is now an importable function). - conftest: hypervisor-path stubs for the ported settings.check(), and a short execution-root fixture (the check rejects pytest's long tmp paths through the 108-byte sun_path cap, exactly like the Python daemon). - test-rust.yml: install the lifecycle toolchain (acl, cloud-image-utils, qemu, ndppd) before the cargo tests, and widen the path filters to the network modules the fixtures now capture. * test(integration): drive the suite against either daemon via ALEPH_VM_SUPERVISOR_IMPL The suite stays implementation-agnostic; only the spawn differs. With ALEPH_VM_SUPERVISOR_IMPL=rust (the packaged launcher's selector) the harness runs the cargo-built binary (AVM_ITEST_RUST_BINARY overrides the default rust/target/debug path) with the same ALEPH_VM_* environment and --socket override. Ephemeral Firecracker tests (increment 4), WatchEvents and the backup suite (increment 5) are skipped under rust with explicit reasons; the persistent QEMU lifecycle, port forwards, volumes, resource release, error paths and the daemon-restart adoption all run. The 'Supervisor integration tests' job becomes a python/rust matrix; the rust leg installs the pinned toolchain and builds the daemon (with the shared cargo cache) before the run. * docs(rust): ledger the increment 3 decisions and close entry 14 - Entry 11: record the decision the entry deferred to increment 3: the Python destructive startup sweep is retired. Adopted-stopped VMs are startable and deletable, hidden VMs keep their vm_index claims and honor DeleteVm. - Entry 14: closed (VmInfo.gpus rebuilt for VMs adopted running, set from the validated request at create); disposition done. - Entry 9 updated to the post-increment-3 staging; entry 6 extended to RecreateNetwork.summary_json. - New entries 18-21: the JSON cloud-init seed representation, the networking-gated check() slice, the log-and-continue forwarding sysctls, and the real per-VM mutation locks. - Module headers updated for the increment 3 surface. * fix(test): close the CI gate-breaks in the integration and conformance harnesses Review batch for increment 3, harness findings: - The rust CI leg no longer fails on WatchEvents: only the event subscription and its assertions in test_qemu_stop_start_reboot_cycle are gated on ALEPH_VM_SUPERVISOR_IMPL (the Rust daemon serves WatchEvents in increment 4); the stop/start/reboot cycle itself runs on both legs. - An unknown ALEPH_VM_SUPERVISOR_IMPL value fails collection loudly instead of silently testing the python daemon. - All-skipped suites can no longer pass CI silently: with CI=true the integration conftest fails when /dev/kvm or qemu-system-x86_64 is missing, and the conformance pytestmarks (cargo_missing) fail collection when ALEPH_VM_CONFORMANCE=1 and cargo is absent. - New real-kernel coverage under requires_qemu, running on both matrix legs: AddPortForward must land a DNAT rule in nft list output (and serve the guest through the host port) with RemovePortForward taking it out, and RecreateNetwork must leave a running QEMU VM reachable with its persisted forwards reapplied. * fix(rust): concurrency, crash-safety and Python-parity batch from the increment 3 reviews Concurrency and crash (with regression tests for each): - One coarse host-network lock (DaemonState::net_lock, innermost after creation_lock and the per-VM locks, never held across systemd waits) serializes host-port allocation-through-persistence, tap/nftables setup and teardown, and RecreateNetwork's flush-and-rebuild: two concurrent AddPortForwards can no longer allocate the same host port and leak a live DNAT rule, and a recreate can no longer flush a freshly booted VM's chains mid-create. - The CreateVm boot closure runs under catch_unwind: a panic takes the same cleanup path as an error and reports INTERNAL. - check_memory_backstop saturates instead of overflowing on a hostile memory_mib. - The cloud-init hostname fallback slices characters (Python vm_hash[:63]); a multibyte vm_id no longer panics. - run_lifecycle documents the blocking-pool thread budget for the long-poll waits. Parity fixes (Rust changed to match the Python daemon): - Rootfs-disk validation answers INVALID_ARGUMENT with the INVALID_BACKEND trailer and types.py's exact messages, before any side effect (conformance-tested against the wire). - The legacy port-mapping migration (migrate_port_mappings_from_legacy_db) is ported 1:1 and runs at boot before adoption, over a fixture legacy DB generated by the real SQLAlchemy models. - Protocol iteration order is udp-then-tcp everywhere (SUPPORTED_PROTOCOL_FOR_REDIRECT), pinned against a fixture batch captured from the real VmExecution.recreate_port_redirect_rules. - VMs enumerate in insertion order (sorted adoption, then creation order) like the Python pool dict, via a per-entry ordinal; a replacement keeps its position. - DEVELOPER_SSH_KEYS / USE_DEVELOPER_SSH_KEYS are read (pydantic truthiness semantics included) and merged into the cloud-init keys. - Wire paths normalize like pathlib (collapse //, drop . and trailing slashes, keep ..) at spec ingestion, in the written config and the create-retry idempotency comparison. - The forwarding sysctls write only when the current value parses as int 0 (a host at "2" keeps its accept-RA behavior). - The host-port bind probe binds without SO_REUSEADDR (raw libc), so TIME_WAIT ports are refused like Python's plain socket. - Tap error tolerance matches interfaces.py exactly: EBUSY on create and set-link-up failures are warnings, deletion failures never fail a stop. - A failed ndppd.conf write propagates and fails the RPC. - erase_volumes and remove_controller_configuration propagate real unlink errors (missing files stay tolerated where Python passes missing_ok=True): ReinstallVm on an undeletable rootfs is INTERNAL. - The static-policy IPV6_SUBNET_PREFIX < 124 abort of StaticIPv6Allocator is ported into settings.check(). - RebootVm on a stopped VM is pinned: stopped_at/stopping_at survive, mapped_ports stay empty, the VM reports STOPPED (the shared restart-without-network wart is ledgered). Test quality: - RecreateNetwork cargo coverage (rebuild for running VMs, skip stopped, summary shape, redirect reapplication). - A shared chronological event log across FakeSystemd, FakeTapBackend and StaticRuleset pins tap-before-nft-before-unit ordering for create and start. - A conformance drift guard compares the committed written-controller-config fixture against the live pydantic model. - generate_rust_fixtures.py documents the SQLAlchemy index-order constraint; the sqlite fixture is regenerated with the venv's pinned version (index order only, content unchanged). - Reinstall wipe_volumes=false, the data-volume erase loop with the read-only skip, and reconcile_boot's failure branch are covered. - The wrong in-code claim in nft.rs that a missing nat prerouting chain fails boot like Python is corrected (the Rust boot logs and serves, ledger entry 22). * docs(rust): ledger the increment 3 review findings (entries 22-36) New entries: boot-time initialize_nftables degradation (22), no reattach retry loop (23, revisit in increment 4), RecreateNetwork cannot restore chains for VMs adopted under a bus outage (24, fix-in-rust in increment 4), Python is_service_active conflating enabled with active (25, fix-in-python-later), non-persistent QEMU create residue (26), ALLOW_VM_NETWORKING flipped off across restart (27), broken port-mapping store at adoption (28), mid-flight port mutation snapshots (29), readopt registration order (30), D-Bus per-call reconnect retry (31), crafted-state numeric/ruleset edges (32), cosmetics group narrowing entry 18's byte-identical metadata claim (33), the joint vm_id shape-validation gap with path traversal in both daemons (34, fix-in-both-later), the Python stop/teardown gating leak for internet_access=false VMs (35, fix-in-python-later; the Rust cleanup is kept), and the shared reboot-of-a-stopped-VM wart (36, fix-in-both-later). Extended: entry 13 (cross-reference to 24), entry 18 (non-ASCII metadata escaping), entry 19 (CONNECTOR_URL, FAKE_DATA_* and the confidential checks deliberately staged or agent-side, plus the ported IPV6_SUBNET_PREFIX abort), entry 20 (the int-0 write gate and the unparseable-value tolerance), entry 21 (the host-network lock and the documented lock order). * fix(test): assert the DNAT rule structurally, loopback cannot traverse PREROUTING The new port-forward integration test probed the redirect through 127.0.0.1, but locally generated traffic takes the OUTPUT hook and never reaches the PREROUTING chain, and the rule additionally matches iifname == NETWORK_INTERFACE; connection refused was the correct kernel behavior, for both daemons. Assert the full rule structure (interface, dport, dnat target) from nft -j against the live kernel instead; end-to-end traffic through the redirect stays with the aleph-testnets upgrade checks, whose client connects through the node's public interface. Also omit tests/integration from coverage like conformance: the coverage-measured job only imports these root-and-KVM files, and their uncovered lines sank codecov/patch.
…streams) (#1027) * feat(rust): reattach retry loop and RecreateNetwork IP rederivation Port the Python background reattach retry (pool.py run_reattach_retry_loop, REATTACH_RETRY_INTERVAL_SECONDS=30, REATTACH_RETRY_MAX_ATTEMPTS=5): hidden VMs (failed adoption or boot reconcile) are queued in the world view and retried under creation_lock with the Python liveness gate (a positively dead controller is stopped/disabled and dequeued; unknown or transitional states spend an attempt; exhaustion leaves the live controller alone). DeleteVm's hidden path now serializes with the retry pass under creation_lock and dequeues the VM, the discard_failed_reattach semantics (ledger entry 23). RecreateNetwork rederives missing IP assignments from vm_index/vm_hash before its running filter, so a node whose D-Bus was down at daemon boot can heal its VM chains through the RPC (ledger entry 24). * feat(rust): WatchEvents lifecycle event stream Port the Python LocalSupervisor event fan-out: an EventHub of unbounded per-subscriber queues (no replay; clients snapshot with ListVms first, as the proto documents), with emissions at exactly the Python points: CreateVm (DEFINED to the created status, on idempotent retries and readopts too), StopVm (including the idempotent already-stopped (STOPPED, STOPPED) pair), StartVm (nothing on the already-running short circuit), RebootVm's down-then-up pair, ReinstallVm (down, then up when the VM restarts) and DeleteVm. Event timestamps carry full time.time_ns precision, unlike the microsecond-truncated stage stamps. * feat(rust): StreamLogs journald follow Port LocalSupervisor.stream_logs. The live phase is the Python make_logs_queue semantics (an sd-journal follow on the vm-{hash}-stdout / -stderr identifiers) implemented as one journalctl --follow subprocess per stream, killed when the client drops the stream. One subprocess serves both phases: --lines=10000 (the ledger-16 server cap) replays the history gap-free when include_history is set, --lines=0 follows from now otherwise. An unknown or deleted VM is not an error: its journald history (when asked) is served and the stream ends, the Python executions.get() == None path. Deliberate divergences from the Python accidents (double history delivery, the proto-violating full replay on include_history=false, wall-clock stamping of replayed lines) are ledgered with increment 4. * feat(rust): ephemeral Firecracker programs and RunProgramCode Port the ephemeral launcher (design doc section 3): CreateVm for non-persistent FIRECRACKER specs boots the program as a direct child of the daemon, a literal MicroVM port. Jailer chroot prep (rm/mkdir/chown, hardlink-or-copy staging into /opt) and the unjailed fixed-path mode, setfacl, the Firecracker config JSON byte-for-byte against the pydantic serialization (pinned by the new firecracker-config.json fixture from scripts/generate_rust_fixtures.py; only the top-level keys carry the dash aliases, as in config.py), stdout/stderr wired to journald via the sd_journal_stream_fd protocol under vm-{hash}-stdout/-stderr, the vsock ready handshake on {vsock}_{ready_port} bounded by the spec's ready_timeout_secs (INIT_TIMEOUT fallback, new USE_JAILER/INIT_TIMEOUT/ PRINT_SYSTEM_LOGS settings), and kill-based teardown (halt attempt over the channel, SIGKILL, artifact removal). The lifecycle mirrors Python's non-persistent branches: microvm IPv6 addressing (VmType prefix 1), tap only when the spec asks for internet, no controller config and no port-mapping preload at create, stop/start answering the NotImplemented pair, delete dropping persisted mappings unconditionally (record_usage), reboot as a real recreation from the held spec, reinstall erasing the spec disks and returning the stopped VM, and the recreate-network wart where a program's DNAT redirects are not reapplied (is_instance gate). RunProgramCode speaks the CONNECT 52 + msgpack RunCodePayload exchange with the scope passed through opaquely (the fixmap header is prefixed to the client's own msgpack bytes), VmInitNotConnectedError and empty-timeout messages matching. A guest-channel-less or persistent FIRECRACKER spec is refused up front (InvalidBackend / Unimplemented; ledgered). * test: un-skip the Firecracker surface on the rust leg; conformance for the increment 4 RPCs The integration suite's rust leg now runs the full Firecracker surface (the boot + vsock ready-handshake test, port forwards, WatchEvents, StreamLogs history): FC_READY no longer excludes the rust daemon, the WatchEvents test leaves the skip list, and the CI=true guard fails the job when the Firecracker kernel or runtime squashfs is missing so neither matrix leg can silently skip FC surfaces. The backup assertions in the ephemeral error-path test follow the established impl gate (UNIMPLEMENTED under rust until increment 5). Conformance grows a streams module for what runs without root or KVM: WatchEvents fan-out/ordering/no-replay with the Python emission points, StreamLogs end-of-stream (not NOT_FOUND) for unknown VMs, RunProgramCode's error vocabulary, and the ephemeral create rejection paths with the exact Python messages and zero side effects. The lifecycle module tracks the new FC create contract (guest_channel required; persistent programs UNIMPLEMENTED) and gains the firecracker-config.json drift guard against the live pydantic models. Ledger: entries 23 and 24 closed (retry loop, RecreateNetwork rederivation), entry 9 restaged, entry 21 extended with the RunProgramCode lock note, new entries 37-40 (StreamLogs contract fixes, opaque msgpack passthrough, persistent programs, block-device rootfs). * docs(rust): align the world-view module notes with the ported retry queue * fix(rust): ephemeral-launcher parity batch from the increment 4 reviews Port the Python microvm.py behaviors the reviews found dropped or diverging, plus the launcher-local robustness fixes: - validate the guest ready payload like the wait_for_init callback: a non-empty payload must be one whole msgpack map with a "version" key (msgpack.unpackb parity via a new shape-validation helper); an unparseable payload discards that connection and the server keeps accepting until the init timeout fires (MICROVM_INIT_FAILED), and an empty payload stays the older-runtime default - propagate EXDEV for the kernel staging: enable_kernel catches only FileExistsError, so a cross-device kernel fails the create; the copy fallback stays where Python has it (enable_file_rootfs/enable_drive), with the filesystem calls injected so every branch is unit-tested - match the bare "Not a file or a block device: {path}" message of enable_rootfs (no OS error appended) - wire config_file_path for real: recorded on the jailed path only (start_jailed_firecracker parity), unlinked at teardown; the field was dead and its comment wrong - bound the teardown connect like Python's asyncio.wait_for(..., 5) around shutdown(): a wedged guest channel can no longer hang teardown - clamp float-seconds timeouts (Duration::try_from_secs_f64, saturating at 100 years) so a crafted RunProgramCode timeout_secs like 1e250 behaves like asyncio.wait_for instead of panicking New regression tests cover each behavior, including the read-phase stall (a guest that connects then goes silent times out instead of hanging CreateVm under the creation lock) and a note pinning the "OK <port>" ack line to Firecracker's vsock protocol. * fix(rust): lifecycle parity batch from the increment 4 reviews - erase_volumes on programs: stop inventing an extra-disk wipe. Python crashes on self.resources.volumes (SpecProgramResources has none) after the rootfs step, so the observable disk outcome is rootfs erased on reinstall-wipe and extra disks never touched; pin that outcome but answer success instead of reproducing the destructive-then-INTERNAL crash (ledger entry 43) - RunProgramCode validates scope_msgpack up front like grpc_server.py:158 (one whole msgpack value, strict str/bin map keys, before the VM lookup) and still forwards the original bytes untouched - non-positive vcpus/memory_mib fail INTERNAL before any spawn, the pydantic PositiveInt ValidationError point of MachineConfig - RecreateNetwork excludes failed chain removals from removed_chains like remove_all_aleph_chains (failed removals are a WARN and never flip the success flag); StaticRuleset gains batch-failure injection to pin it Also adds cargo-tier RunProgramCode coverage over a live guest-channel socket (create, dial, ack line, raw reply) and clamps the settings-sourced init_timeout through the new duration helper. * fix(rust): StreamLogs robustness: follow cap, reap on every pump exit Two blocking-pool hazards from the increment 4 reviews: - each live follow pins one blocking-pool thread for its whole lifetime, so ~512 concurrent follows would exhaust tokio's blocking pool and wedge every lifecycle RPC. Concurrent follows are now bounded by a semaphore (64, far below the 512-thread pool); the excess request answers RESOURCE_EXHAUSTED and the slot frees when the stream drops (ledger entry 44) - the follow subprocess is now killed AND reaped on every exit path: KillChild::stop waits after the kill, and the pump stops the follow when it exits (the mid-send break on a dropped client never reached the reader's EOF-path wait, leaving a zombie journalctl) The cleanup contract is pinned by mutation-sensitive tests: a recording stopper fails the suite if stop() is removed from StreamWithCleanup::drop or the pump exit path, and a real child test asserts the killed process is actually reaped (no Z state left). * test: assert the lifecycle event stream on the rust leg too The WatchEvents gate in test_qemu_stop_start_reboot_cycle predates increment 4: the Rust daemon serves WatchEvents now, so the stop/start/reboot event sequence is asserted on both matrix legs (the backup-surface gates in test_error_paths.py are the only ones left, correctly waiting on increment 5). The subscribe-race sleeps keep a comment noting the flake risk: the proto has no subscription ack (WatchEvents deliberately has no replay), so no deterministic readiness signal exists. * docs(rust): ledger the increment 4 review findings (entries 41-46) New entries: the RunProgramCode-vs-failing-boot race (41, Rust answers NOT_FOUND where Python hangs), the jail-staging edges including the shared duplicate-basename aliasing wart (42), the program erase_volumes AttributeError whose safe subset the Rust daemon pins (43), the Rust-only follow cap and timeout clamps (44), mid-boot guest_channel_path visibility (45) and the shared retry-exhaustion observability gap (46). Entry 38 narrows to the shape-validated pass-through and entry 40 records the now byte-identical rootfs message.
…al) (#1028) * feat(rust): supervisor daemon increment 5 (backups and restore) Port the backup/restore RPC surface 1:1 from the Python LocalSupervisor (src/aleph/vm/supervisor/local.py) and its qemu controller backup helpers: StartBackup, GetBackupStatus, ListBackups, DownloadBackup, DeleteBackup, RestoreBackup and RestoreFromImage. - src/backup.rs: the archive-management half. A qemu-img seam (DiskTools: convert -c / check / info) keeps the tar + sha256 + meta.json format logic unit-testable without a live QEMU. Completed archives are the record; only in-flight and FAILED runs live in the registry (the Python _backup_jobs / _backup_tasks / _backup_locks triple). StartBackup spawns the background run; idempotency answers a running job or a fresh archive. - src/qmp.rs: blocking QGA client for the best-effort guest fs-freeze, plus the QMP client the confidential surface uses next. - Restore (stop / swap rootfs / restart) lives in src/lifecycle.rs, reusing the stop/start execution machinery; it takes the vm_lock plus the shared per-VM backup disk lock. - RpcError::BackupNotFound maps to NOT_FOUND + the BACKUP_NOT_FOUND trailer; DownloadBackup streams the archive in 1 MiB offset chunks. - Un-skip tests/integration/test_backup_restore.py on the rust leg (its cycle is KVM-gated); conformance covers the no-root slice (empty listing, unknown-VM and malformed-id rejections). Ledger entries 47 (backup metadata cosmetics), 48 (the second-precision intermediate-name collision, ported bug-for-bug), 50 (QGA/QMP read timeouts) and 51 (restore lock discipline). * feat(rust): supervisor daemon increment 6 (confidential VM surface) Port the confidential surface 1:1 from the Python LocalSupervisor, matching Python's behavior exactly (where it acts vs where it needs SEV hardware). The real attestation stack stays Phase 3 (aleph-cvm); nothing here invents confidential behavior beyond what Python does. - src/confidential.rs: InitializeConfidential writes the owner's vm_session.b64 / vm_godh.b64 under CONFIDENTIAL_SESSION_DIRECTORY/<vm_id> and enable_and_starts the controller unit (no hardware, fully testable). GetMeasurement and InjectSecret are QMP passthrough to a running confidential QEMU (query-sev, query-sev-launch-measure, sev-inject-launch-secret, cont): the QMP protocol client is ported and unit-tested against a fake monitor socket, but the SEV data path is hardware-gated (Tier 2), like the Python path. - Confidential CreateVm (src/lifecycle.rs, src/controller_config.rs, src/cloudinit.rs): build_qemu_confidential_configuration produces the four SEV fields plus the confidential cloud-init (LUKS growpart bootcmds, no guest agent); execution.start leaves the unit down, so the VM reports awaiting_confidential_init until InitializeConfidential runs. A confidential spec without a resolved firmware_path is refused InvalidBackendError. - src/checks.rs: the ENABLE_CONFIDENTIAL_COMPUTING settings.check() block (SEV_CTL_PATH, the SEV/SEV-ES kernel-module gates), gated on the flag. Ledger entry 49 (the confidential boundary: non-hardware work reproduced exactly, SEV work marked Tier 2); entries 9 and 19 updated (both surfaces landed). * fix(rust): backup/restore serialization, download cap, and int(x,0) SEV policy Increment 5-6 review batch (robustness + parity for the backup and confidential-create surfaces): - DeleteVm (tracked path) takes the per-VM backup disk lock around the world-entry removal and the wipe erase, and reaps the VM's backup registry entry (jobs, task slot, disk lock), so a delete cannot race a StartBackup's qemu-img read of the same disks and no completed/failed run lingers in ListBackups for a deleted VM (R1/R3). - StartBackup admits the check-and-register critical section under a registry lock, so two concurrent StartBackups for one VM cannot both spawn a run and orphan a JoinHandle; the second returns the running job (R2). - DownloadBackup is bounded by a semaphore (cap 16) like StreamLogs, rejecting the excess RESOURCE_EXHAUSTED with the InsufficientResources trailer (R5). - cleanup_expired_backups sweeps stale intermediate .qcow2 files a killed run leaves behind (R6). - In-memory backup jobs list in a deterministic insertion order, matching Python's dict iteration (P2). - parse_sev_policy matches CPython int(x, 0) exactly (0x/0o/0b prefixes, no leading-zero decimals, PEP 515 underscore placement), rejecting the unrepresentable negative/bignum edges Python would accept (P1); pinned against the CPython truth table. - Correct the archive-format doc comment to state the real cross-read contract instead of a false byte-for-byte claim (P4). Regression tests: concurrent-start single-run, registry reap, delete reaps, restore data path (extract good/no-member/non-file, swap and preserve-on-failure), and a Python-built archive fixture read back by the Rust reader. The CI integration guard now fails loud on a missing QEMU cloud image (T2/T3/T5/T7). * fix(rust): bound QMP/QGA socket I/O and pin the command protocol Increment 6 review batch (QMP robustness + test quality): - Set a write timeout matching the read timeout on both the QMP monitor and QGA guest-agent connections, so a peer that never drains its socket cannot block write_all forever and park a blocking-pool thread; cap a single response line at 8 MiB so a newline-less stream cannot grow the read buffer unbounded (R4). - The fake QMP/QGA test server now records the exact request bytes. New tests pin the command names and argument keys against the Python QemuVmClient / QemuGuestAgentClient: qmp_capabilities, query-sev, query-sev-launch-measure, sev-inject-launch-secret with packet-header/secret, cont, and guest-fsfreeze-freeze/thaw. A mutation renaming any of these fails a test (T1). - Cover get_measurement/inject_secret with a VM present but no QMP socket (the pre-hardware path), asserting the Python "VM is not running" INTERNAL (T4). - A direct test of the capped line reader rejects an overlong newline-less line. * docs: record the increment 5-6 review-batch divergences Extend the port divergence ledger for the review-fix batch: - entry 44: the DownloadBackup concurrency cap. - entry 49: parse_sev_policy now matches CPython int(x, 0) exactly, with the one intentional narrowing (negative/bignum policies a u32 cannot hold); plus the query_sev_info field-level leniency (P3). - entry 50: the QMP/QGA write timeout and the 8 MiB response-line cap. - entry 51: the DeleteVm disk-lock-and-reap serialization. - entry 52 (new): the dropped-but-unreachable restore persistence guard (P5). - entry 53 (new): the stale-intermediate .qcow2 cleanup sweep and the best-effort fs-freeze quiesce note (R6, T8/T9). * fix(rust): write backup archives with regular members, not GNU sparse The rust integration leg caught RestoreBackup rejecting the daemon's own archive: 'Backup member rootfs.qcow2 ... is not a regular file'. A qemu-img converted rootfs is sparse on disk, and tar::Builder defaults to sparse detection, so it wrote the member as an EntryType::GNUSparse ('S') entry; extract_rootfs_member's is_file() check is true only for Regular. Python's tarfile reconstructs sparse entries transparently (the download step passed), which hid it. builder.sparse(false) writes plain regular members, matching Python's tarfile, which never emits GNU sparse. Regression test appends a genuinely sparse source file (the existing tests used small non-sparse files). Also drop the now-stale test_error_paths rust-branch: backups are implemented, so the rust daemon raises VmNotFound/BackupNotFound like python, not NotImplemented.
… parity (#1030) First foundation increment of Phase 3 (stacks on #1029). Adds a new Rust binary crate `aleph-vm-controller` that reads the existing `{vm_hash}-controller.json` and, for a non-confidential persistent QEMU VM, produces a QEMU argv byte-identical to the Python `QemuVM.start()`, spawns it, streams stdout/stderr to the systemd journal, and shuts it down gracefully on SIGTERM. This is a faithful 1:1 port; the Python controller is the parity oracle. Confidential/SEV is A2; packaging cutover is A3 (this PR wires nothing into systemd yet). **Parity is pinned by an oracle-generated conformance suite:** 14 argv fixtures are generated by monkeypatching the real Python `QemuVM.start()` (capturing the exact argv it would exec), and a Rust integration test asserts the Rust builder matches each byte-for-byte. Mutation testing killed 23/23 deliberate breakages after the review batch. **Reviewed by three adversarial lenses** (parity vs Python, Rust robustness, test quality with mutation). Fixes applied in separate commits, each with a regression test: - QMP reader given a wall-clock deadline + line/skip caps, and the graceful-stop QMP calls wrapped in a timeout so a chatty or wedged monitor can no longer hang the stop escalation (the qcow2-corruption path). - Dispatch now branches on the `hypervisor` field first (matching Python precedence): firecracker and confidential-shaped configs are rejected in A1, not booted. - `mem_size_mb` coercion fails closed on negative/non-finite (no silent `-m 0`). - Fixtures added for the three falsy corners the mutation tester found unpinned (`qga_socket_path=None` "None"-literal wart, empty-string `interface_name` / `cloud_init_drive_path`). journal header protocol and escalation-budget tests added. Deliberate divergences from Python recorded in `docs/plans/rust-port-divergences.md` (entries 54-57): fail-open journal fallback, exit-0-on-qemu-crash (Python wart, ported), omitted `-i` flag.
…#1031) Second foundation increment of Phase 3 (stacks on #1030). Extends the Rust `aleph-vm-controller` to launch an existing SEV / SEV-ES confidential QEMU VM with a QEMU argv byte-identical to the Python `QemuConfidentialVM.start()`. SEV-SNP is B1; this reproduces only the current `-object sev-guest` path. Packaging is A3. **What it adds:** - `build_confidential_argv` (OVMF pflash, qcow2 rootfs, `--no-reboot`, paused `-S`, `-object sev-guest` with policy/cbitpos/reduced-phys-bits/dh-cert/session, `q35,confidential-guest-support=sev0`, always `host-phys-bits-limit`, NIC without `rombar=0`, and the confidential `-qmp`-before-qga ordering). The plain-path helpers (host volumes, GPU, qga) are factored out and shared; the A1 spawn/journal/graceful- stop path is reused verbatim (verified byte-identical, no plain-path regression). - `cpuid.rs`: a `SevHostInfo` reader for cbitpos/reduced-phys-bits from CPUID leaf 0x8000001F EBX, matching the Python `SecureEncryptionEbx` field layout. It fails closed on the SEV feature bit (a deliberate, documented safety improvement over Python, divergence 58). The argv builder takes an injected `SevHostInfo` so parity fixtures are deterministic on non-SEV CI hardware. - Dispatch branches on the `hypervisor` field first (Python precedence): a confidential payload mislabelled firecracker still fails closed. **Parity + tests:** 8 confidential fixtures generated from the real `QemuConfidentialVM.start()` (SEV 0x1, SEV-ES 0x5, policy 0x0, GPU double `host-phys-bits-limit`, host volume, cloud-init, and a distinct-CPUID case that makes the injected values load-bearing). Reviewed by three adversarial lenses: no argv, policy, bit-extraction, generator, or dispatch divergence. Fix batch pinned the one surviving mutation (shared CPUID constants) and added prelaunch-guard tests (SEV-platform refusal, missing godh/session).
… controller (#1032) Third foundation increment of Phase 3 (stacks on #1031). Wires the Rust `aleph-vm-controller` (A1 plain + A2 confidential) into packaging behind the existing impl flag and validates it booting a real QEMU VM on KVM. After this, Rust owns the persistent-QEMU launch. **Cutover (mirrors the daemon's supervisor-launcher):** - New `packaging/controller-launcher` (POSIX sh): dispatches `aleph-vm-controller@` on `ALEPH_VM_SUPERVISOR_IMPL` (rust -> the Rust binary, python -> the module, unknown -> python fallback that never bricks the node), forwarding `--config`. - The controller unit now `EnvironmentFile=`s the same `/etc/aleph-vm/supervisor.env` the daemon reads, so one flag switches BOTH the daemon and every newly (re)started controller. `KillMode=mixed` / `TimeoutStopSec=60` / PYTHONPATH preserved. - The deb builds and installs `aleph-vm-controller` + `controller-launcher` to `/opt/aleph-vm/bin/` (deb-contents CI check greps for both). Rollback = flip the flag. **Real-KVM validation:** the integration harness previously always ran the python controller even on the rust leg; it now selects the Rust controller under `impl=rust` and fails loudly if its binary is missing. A new `assert_controller_matches_impl` guard isolates "wrong controller ran" from "controller ran but VM did not serve", so the rust leg cannot false-green on python. Tests covering boot -> RUNNING/serves -> graceful stop and daemon-restart re-adoption now exercise the Rust controller.
…ng (#1033) First SNP increment of Phase 3 (stacks on #1032). Ports aleph-cvm's aleph-tee crate into rust/crates/aleph-tee: the TeeBackend trait, SEV-SNP attestation report generation and parsing, AMD-KDS certificate-chain verification, the sev-snp-guest QEMU argument generator, and X.509 attestation embedding (OID 1.3.6.1.4.1.60000.1.1). This is the shared library B1 (SNP host launch) and B2b (attest-cli) build on. The security constants (report offsets, SIGNED_REPORT_SIZE=0x2A0, r/s layout, VMPL gate, KDS URL, cbitpos/policy, OID) are byte-identical to the donor. Hardened beyond the donor (reviewed by three lenses; the donor's attestation had a forgeable trust anchor and near-zero crypto test coverage, both fixed here): AMD ARK root pinning: verify_cert_chain now pins the chain's ARK against AMD's genuine root per product (SubjectPublicKeyInfo compared to the sev crate's vendored builtin::{milan,genoa,turin} roots). The donor tied the root to AMD by a forgeable CN/O string check alone, so a fabricated self-signed ARK passed; that is now rejected. A wrong pin fails closed (rejects genuine reports), the safe direction. Crypto now tested: synthetic P-384 key/cert tests kill the five mutations that previously survived (report-signature verify, VMPL>1 gate, each cert-chain link + ARK identity, cbitpos=51, the OID value). A tampered signature and a forged AMD ARK are asserted rejected. Robustness: cert validity/expiry checking, a 256 KiB KDS response cap (anti-OOM), a product allowlist (anti path-traversal/URL-injection), and a DER-length parse that can no longer panic on a crafted length.
…1034) **1. `aleph-attest-agent` crate** (`rust/crates/aleph-attest-agent`) - the in-guest attested-TLS sidecar: generates an ephemeral P-384 key, gets a key-bound SEV-SNP report, embeds it in a self-signed cert via the aleph-tee OID extension, serves fresh attestation + one-shot secret injection, and reverse-proxies to the workload. Ported from aleph-cvm; unifies rustls onto the ring provider (aws-lc kept out of the tree). **2. Core measured-boot Nix flake** (`nix/`) - measured OVMF + kernel + initrd (with the attest-agent baked in) + a dm-verity rootfs, and a precomputed `sev-snp-measure` measurement. Compose-rootfs, LUKS/encrypted, and the demo workload are scoped out (design non-goals). **The donor's measurement was non-reproducible (random mkfs/verity seeds); this makes it deterministic** (fixed UUID, non-zero hash_seed, SOURCE_DATE_EPOCH, pinned verity salt/uuid), and `nix build --rebuild` of rootfs/verity/initrd/measurement is byte-identical, resolving the top SNP risk (measurement reproducibility) before testnets. **Hardened beyond the donor** (three review lenses; the crypto fix is security-critical): - **Attested-key-confusion MITM closed:** the fresh-attestation endpoint no longer places the raw client nonce into `report_data`. The canonical `report_data` scheme is now domain-separated and channel-bound (`SHA-384(DOMAIN || pubkey [|| nonce])`), placed in the shared `aleph-tee` crate so the agent and the B2b verifier cannot drift. A relayed fresh report can no longer be reused with an attacker key (attack-impossibility test). - Secrets bind-mounted into the chroot (delivery was silently broken); secret store symlink-hardened (O_EXCL/O_NOFOLLOW, 0700); an nft ruleset firewalls the guest to only the attested port (prevents attestation bypass if the app binds 0.0.0.0); one-shot concurrency test; honest zeroization; hop-by-hop header stripping in the proxy. Report generation is hardware-gated; the live-KDS test is `#[ignore]`d. Divergences 63-67 recorded, with an upstreaming recommendation for the domain-separation fix. `cargo fmt`/`clippy -D warnings`/`test` green; Nix outputs build and rebuild reproducibly. Known deferred (tracked, ledger 66): inject-secret is unauthenticated (owner-auth is a later confidential-flow design step, not needed for the attest-only testnets path).
…1035) - **Controller** (`supervisor-controller`): `build_snp_argv` emits a measured direct-kernel boot (`-bios` measured OVMF, `-kernel`/`-initrd`, `-append` the exact measured cmdline, `-cpu EPYC-v4`, `sev-snp-guest` object with `kernel-hashes=on`, `policy=0x30000`, `memory-backend-memfd`). No `-S`, no session/godh: SNP has no guest-owner launch-secret handshake; secrets arrive at runtime via the in-guest attest-agent (B2a). The TEE fragment is pinned byte-for-byte against the `aleph-tee` generator by an oracle test. - **Daemon** (`supervisor-daemon`): `TeeBackend::SEV_SNP` selects the SNP path, writes the measured-boot fields into the controller JSON, and derives the kernel cmdline from the verity roothash sidecar so it matches B2a's `sev-snp-measure` inputs exactly. `HostInfo.sev_snp_supported` reads `/sys/module/kvm_amd/parameters/sev_snp`; SNP VMs report `ConfidentialMode::SEV_SNP` and start immediately (no session to await). The SEV/SEV-ES (A2) path is untouched (backend switch only, written-config bytes unchanged for plain/SEV). **Measurement match** (the load-bearing property, verified by the review): the launch presents the same OVMF/kernel/initrd/cmdline/EPYC-v4/vCPU config B2a measured. The roothash is hex-validated (cmdline-injection barrier) and the derived cmdline is asserted as an exact literal.
…fore-send (SEV-SNP complete) (#1036) Ports the client verifier that connects to a confidential VM's in-guest attest-agent (B2a), verifies the embedded SEV-SNP attestation, pins the measurement, and injects secrets over the attested channel. After this, **SEV-SNP is complete** (B0 crypto lib + B2a image/agent + B1 host launch + B2b verifier). - Uses the shared, domain-separated `aleph_tee::report_data` scheme (matching the B2a agent), so a relayed/key-confusion report is rejected. Inherits AMD **ARK root pinning** from B0 automatically. **Security-hardened beyond the donor** (three review lenses found the donor's verifier `EXPLOITABLE` two ways; both fixed, and aleph-cvm needs the same upstream): - **Trust is now blob-derived.** The unsigned JSON `report_data`/`measurement` copies were removed from `AttestationReport` entirely (it carries only the AMD-signed `data` blob), so gating trust on an attacker-settable copy is structurally impossible. Every check parses the signed blob. Previously an attacker could replay any genuine report as `data` and set the JSON copies to pass key-binding + measurement pinning. - **Full AMD chain verified before any secret is sent.** `SnpCertVerifier` is now the sole authority: it runs the complete verification (blob-derived key-binding + measurement + the KDS/ARK chain via `block_in_place`) at the handshake, so the connection only completes for a fully-verified TEE and no secret ever reaches an unverified endpoint. Previously the secret POST went on the wire before the chain was checked. Mutation-confirmed tests: a report with a borrowed-genuine blob but attacker JSON is rejected; `inject_secret` transmits zero bytes when the chain fails (a local attested-TLS server records receipt). 410 workspace tests pass, 1 ignored (network KDS). Divergences 71-72 recorded with an upstreaming recommendation. Deferred (tracked): inject-secret owner-auth (ledger 66) is a separate confidential-flow design step; this fix ensures secrets only reach a verified TEE, not that the caller is authorized.
#1037) First NUMA increment: the supervisor auto-places VMs on NUMA nodes (pack-first) and pins their vCPUs via the systemd `AllowedCPUs` property. Independent of the QEMU argv (CPU pinning is a systemd unit property); the memory-backend NUMA binding and hugepages are C2. - **`numa.rs`** (ported from aleph-cvm): `NumaTopology` (sysfs parse, cpulist/MemTotal/ hugepage counts) + `NumaAllocator` (pack-first vCPU placement ledger, cpuset formatting). - **`HostInfo.numa_nodes`** populated from the detected topology (was empty). - **Placement** (decision 4, supervisor auto-places): a requested `VmSpec.numa_node` is honored (validated), else pack-first; the effective node is reported in `VmInfo.numa_node` (was hard-coded `None`). - **Pinning** via a systemd drop-in (`aleph-vm-controller@{hash}.service.d/numa.conf`, `AllowedCPUs={cpuset}`) written + `daemon-reload`ed before start, removed on delete. - **Reconcile** reconstructs the placement ledger from each adopted unit's `AllowedCPUs`; a VM with unknown placement (pre-NUMA) stays unpinned and uncounted, never silently node 0 (design §8). **Placement/pinning activate only with >1 NUMA node** (a single-socket `CONFIG_NUMA` host reports its node but writes no drop-in and reserves nothing, so non-multi-node hosts behave as before). Empty/failed topology degrades to inert.
DRAFT — must NOT merge/deploy until the agent/supervisor DB split (#1010) has shipped to every node. Port mappings now live in the supervisor DB; the split's one-time migration copies the rows OUT of this agent-DB table on first start. Dropping the table before every node has run that copy would destroy live VMs' host-port forwards. This is the second phase of a two-phase migration: copy in the split release, drop here in a later one. Queued now so it is not forgotten. Claude-Session: https://claude.ai/code/session_012AjsGWxtU8aRywLnquMUfD
…e canonical schema The downgrade recreated port_mappings via hand-written DDL including AUTOINCREMENT, a schema that never existed: 0004 creates the table via op.create_table, which on SQLite emits a plain integer primary key. Rebuild the table in downgrade() by mirroring 0004's upgrade() exactly (same columns, defaults, and index creations, including the partial unique index on active host_port rows), verified by comparing sqlite_master before the drop and after the downgrade. Also drop the explicit drop_index in upgrade(): on SQLite, DROP TABLE removes the table's indexes, and the guard only checks for the table, so a table-without-index state would have aborted the migration on a needless statement. Remove the unused logging import as well.
odesenfans
force-pushed
the
od/drop-vestigial-agent-port-mappings
branch
from
July 30, 2026 15:35
a68cd2d to
f7eaf2e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this is gated
This is phase 2 of a two-phase migration. After the split (#1010), port mappings live in the supervisor DB. On first start, the supervisor's
migrate_port_mappings_from_legacy_db()copies the rows out of this agent-DBport_mappingstable. Dropping that table is only safe once every node has run the copy.If this merged in the same release as the split, the agent's Alembic (
cli.py:407) would run before the supervisor's data migration (pool.setup(), reached viasupervisor.run()) in-process — and unordered across processes in split mode — so theDROPcould execute ahead of the copy and destroy live VMs' host-port forwards.What it does
A new Alembic revision (
f1a2b3c4d5e6, on top of theport_mappings-creatinga1b2c3d4e5f6): drops the now-unusedport_mappingstable + its partial-unique index from the agent DB (guarded by an existence check).downgrade()recreates the table and indexes.Merge criteria
PortMappingno longer on the agentBase.metadata, so the drop sticks).b3c4d5e6f7a8with the samedown_revision = a1b2c3d4e5f6; whichever merges second must re-chain itsdown_revisionoralembic upgrade headfails at agent startup with multiple heads.🤖 Generated with Claude Code