From 7fc4a3264fe7ae4511c6f8a3897588cc31bf9633 Mon Sep 17 00:00:00 2001 From: Hengqi Chen Date: Mon, 8 Jun 2026 11:49:19 +0000 Subject: [PATCH] Add auto-pause/auto-resume capability across the stack MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implement an e2b-compatible auto-pause/auto-resume feature end-to-end: sandboxes opted in via the SDK's `lifecycle` config are paused after their idle timeout and transparently resumed on the next request, with no caller visible to the unwinding/winding work. The feature is always-on at deploy time — opt-in is per-sandbox, not per-cluster. Architecture: CubeMaster create/destroy hooks publish a Redis HSet snapshot (cube:sandbox:meta) plus an append-only event stream (cube:sandbox:events) into the RedisWrite pool — the same pool localcache already uses for the bypass-host proxy map, so no extra Redis configuration is required. Writes are warn-only: a Redis hiccup never fails a sandbox lifecycle op. CubeProxy log_phase stamps per-sandbox last-active into a worker shared dict; rewrite_phase consults a state dict and either passes traffic, returns 503+Retry-After, or fires an internal sub-request to the sidecar's /internal/resume before forwarding. A loopback admin server (127.0.0.1:8082) lets the sidecar push state and pull last-active on a poll. The /_sidecar_resume location appends ?$args to proxy_pass and forces Content-Length: 0 — both load-bearing fixes for the sub-request hand-off. cube-proxy- New Go component at CubeProxy/sidecar/, bundled inside sidecar the cube-proxy container image. The binary is built static (CGO=0, -tags 'netgo osusergo') for the musl-based Alpine runtime. At startup it bootstraps from the meta HSet, consumes the event stream via XREADGROUP, sweeps the registry on a fixed interval, and drives CubeMaster's /cube/sandbox/update for pause/resume. Concurrent resumes are coalesced in-process; cross-replica coordination uses Redis SETNX state locks. Failure paths are explicit: * not-found (ret_code 130483) → evict registry + proxy meta, no retry. * already-in-state (130490) → reconcile + treat as success. * generic RPC failure → roll back, sweep retries. The sweeper has a bootstrap-warmup window so a fresh sidecar doesn't pause everything before its first last_active poll lands. CubeAPI & SDK gains a single `lifecycle` kwarg on Sandbox.create Python SDK matching e2b verbatim: lifecycle={"on_timeout": "pause"|"kill", "auto_resume": bool} Wire shape is the camelCase nested form e2b uses (`lifecycle.onTimeout`, `lifecycle.autoResume`) — drop- in compatibility for existing e2b clients. CubeAPI translates the nested object into CubeMaster's two internal bools so the master-side protocol is unchanged. When `lifecycle` is unset, payloads stay byte-identical to pre-feature behaviour; regression tests lock the absent-field shape down. Deployment deploy/one-click renders 8 CUBE_SIDECAR_* env vars (Redis, CubeMaster URL, listen addr, idle timeout, admin token, …) and 2 nginx-side knobs ($cube_sidecar_addr, $cube_admin_token) into the cube-proxy compose file and global.conf. CubeMaster URL defaults to :8089 (the canonical http_port). No feature-flag env var: lifecycle is always live in the control plane; opt-in happens per-sandbox via the SDK's `lifecycle` argument. Documentation: * docs/guide/lifecycle.md + zh translation under "Core Concepts" / "核心概念" — covers the state machine, every lifecycle method, auto-pause/auto-resume semantics, and operational notes. * examples/code-sandbox-quickstart/auto-resume.py — end-to-end TUI demo (paired with the existing pause.py) verifying state survives the auto-pause / auto-resume cycle across kernel memory and the filesystem. Tests: * CubeMaster: lifecycle store + multi-hook chain (Go unit, race- clean). Pool resolution falls back through RedisWrite cleanly. * CubeProxy: nginx -t against openresty:1.21.4.1 with the rendered global.conf; luajit -b on every touched .lua file. * Sidecar: 34 unit tests across cubemasterclient/registry/ sweeper/resumer/proxypush/httpapi/lifecycle. Failure- code reconciliation, bootstrap-warmup gating, terminal-paused ownership, in-flight de-dup, peer- lock waits, and HTTP path semantics are all covered; deps injected via interfaces so no live Redis is required, -race green. * CubeAPI: inbound lifecycle deserialization → CubeMaster bool translation across all five meaningful shape combinations; outbound CreateSandboxRequest wire-shape snapshot. * Python SDK: 6 lifecycle wire-format snapshots (default omitted, pause-only, pause+resume, kill explicit, auto_resume-only, invalid on_timeout raises); full suite (172 tests) still passes. Signed-off-by: Hengqi Chen --- .gitignore | 1 + CubeAPI/src/cubemaster/mod.rs | 13 + CubeAPI/src/handlers/agenthub.rs | 6 +- CubeAPI/src/models/mod.rs | 50 +- CubeAPI/src/services/sandboxes.rs | 127 ++++- CubeMaster/cmd/cubemaster/app/main.go | 9 + CubeMaster/pkg/lifecycle/init.go | 89 ++++ CubeMaster/pkg/lifecycle/schema.go | 68 +++ CubeMaster/pkg/lifecycle/store.go | 112 ++++ CubeMaster/pkg/lifecycle/store_test.go | 196 +++++++ .../pkg/service/sandbox/runtime_ref_hook.go | 100 +++- .../service/sandbox/runtime_ref_hook_test.go | 74 +++ CubeMaster/pkg/service/sandbox/types/types.go | 11 + CubeMaster/pkg/task/runtime_ref_hook.go | 50 +- CubeProxy/.gitignore | 1 + CubeProxy/Dockerfile | 13 + CubeProxy/Makefile | 46 +- CubeProxy/lua/admin_phase.lua | 206 ++++++++ CubeProxy/lua/log_phase.lua | 45 ++ CubeProxy/lua/path_rewrite_phase.lua | 6 + CubeProxy/lua/rewrite_phase.lua | 9 + CubeProxy/lua/sandbox_state.lua | 120 +++++ CubeProxy/nginx.conf | 89 ++++ CubeProxy/sidecar/cmd/sidecar/main.go | 272 ++++++++++ CubeProxy/sidecar/go.mod | 15 + CubeProxy/sidecar/go.sum | 34 ++ CubeProxy/sidecar/internal/config/config.go | 216 ++++++++ .../internal/cubemasterclient/client.go | 156 ++++++ CubeProxy/sidecar/internal/httpapi/server.go | 136 +++++ .../sidecar/internal/httpapi/server_test.go | 192 +++++++ .../sidecar/internal/lifecycle/parity_test.go | 40 ++ .../sidecar/internal/lifecycle/schema.go | 62 +++ .../sidecar/internal/proxypush/client.go | 205 +++++++ .../sidecar/internal/proxypush/client_test.go | 209 ++++++++ .../sidecar/internal/redisstream/stream.go | 203 +++++++ .../sidecar/internal/registry/registry.go | 150 ++++++ .../internal/registry/registry_test.go | 98 ++++ CubeProxy/sidecar/internal/resumer/iface.go | 32 ++ CubeProxy/sidecar/internal/resumer/resumer.go | 306 +++++++++++ .../sidecar/internal/resumer/resumer_test.go | 320 +++++++++++ CubeProxy/sidecar/internal/sweeper/iface.go | 36 ++ CubeProxy/sidecar/internal/sweeper/sweeper.go | 263 +++++++++ .../sidecar/internal/sweeper/sweeper_test.go | 500 ++++++++++++++++++ CubeProxy/start.sh | 53 +- Makefile | 6 + deploy/one-click/build-release-bundle.sh | 43 ++ .../cubeproxy/docker-compose.yaml.template | 16 + .../one-click/cubeproxy/global.conf.template | 8 + .../scripts/one-click/up-cube-proxy.sh | 33 +- docs/.vitepress/config.mjs | 2 + docs/guide/lifecycle.md | 167 ++++++ docs/zh/guide/lifecycle.md | 166 ++++++ examples/code-sandbox-quickstart/README.md | 22 + examples/code-sandbox-quickstart/README_zh.md | 21 + .../code-sandbox-quickstart/auto-resume.py | 361 +++++++++++++ sdk/python/cubesandbox/sandbox.py | 52 +- sdk/python/tests/test_sandbox.py | 67 +++ 57 files changed, 5863 insertions(+), 40 deletions(-) create mode 100644 CubeMaster/pkg/lifecycle/init.go create mode 100644 CubeMaster/pkg/lifecycle/schema.go create mode 100644 CubeMaster/pkg/lifecycle/store.go create mode 100644 CubeMaster/pkg/lifecycle/store_test.go create mode 100644 CubeMaster/pkg/service/sandbox/runtime_ref_hook_test.go create mode 100644 CubeProxy/.gitignore create mode 100644 CubeProxy/lua/admin_phase.lua create mode 100644 CubeProxy/lua/sandbox_state.lua create mode 100644 CubeProxy/sidecar/cmd/sidecar/main.go create mode 100644 CubeProxy/sidecar/go.mod create mode 100644 CubeProxy/sidecar/go.sum create mode 100644 CubeProxy/sidecar/internal/config/config.go create mode 100644 CubeProxy/sidecar/internal/cubemasterclient/client.go create mode 100644 CubeProxy/sidecar/internal/httpapi/server.go create mode 100644 CubeProxy/sidecar/internal/httpapi/server_test.go create mode 100644 CubeProxy/sidecar/internal/lifecycle/parity_test.go create mode 100644 CubeProxy/sidecar/internal/lifecycle/schema.go create mode 100644 CubeProxy/sidecar/internal/proxypush/client.go create mode 100644 CubeProxy/sidecar/internal/proxypush/client_test.go create mode 100644 CubeProxy/sidecar/internal/redisstream/stream.go create mode 100644 CubeProxy/sidecar/internal/registry/registry.go create mode 100644 CubeProxy/sidecar/internal/registry/registry_test.go create mode 100644 CubeProxy/sidecar/internal/resumer/iface.go create mode 100644 CubeProxy/sidecar/internal/resumer/resumer.go create mode 100644 CubeProxy/sidecar/internal/resumer/resumer_test.go create mode 100644 CubeProxy/sidecar/internal/sweeper/iface.go create mode 100644 CubeProxy/sidecar/internal/sweeper/sweeper.go create mode 100644 CubeProxy/sidecar/internal/sweeper/sweeper_test.go create mode 100644 docs/guide/lifecycle.md create mode 100644 docs/zh/guide/lifecycle.md create mode 100755 examples/code-sandbox-quickstart/auto-resume.py diff --git a/.gitignore b/.gitignore index 84715e133..9a098a42b 100644 --- a/.gitignore +++ b/.gitignore @@ -47,4 +47,5 @@ docs/image.png __pycache__ /design +/plan /tests diff --git a/CubeAPI/src/cubemaster/mod.rs b/CubeAPI/src/cubemaster/mod.rs index 50dc07bcd..6da9f2b7a 100644 --- a/CubeAPI/src/cubemaster/mod.rs +++ b/CubeAPI/src/cubemaster/mod.rs @@ -657,6 +657,19 @@ pub struct CreateSandboxRequest { skip_serializing_if = "Option::is_none" )] pub cube_network_config: Option, + + /// Auto-pause: when true, CubeMaster publishes this sandbox to the + /// auto-pause registry consumed by CubeProxy-sidecar; once the proxy + /// reports it idle for `timeout` seconds the sidecar pauses it. + /// Field name matches CubeMaster's `auto_pause` JSON tag. + #[serde(skip_serializing_if = "std::ops::Not::not", default)] + pub auto_pause: bool, + + /// Auto-resume: when true, an incoming request hitting a paused sandbox + /// is transparently resumed instead of erroring. Field name matches + /// CubeMaster's `auto_resume` JSON tag. + #[serde(skip_serializing_if = "std::ops::Not::not", default)] + pub auto_resume: bool, } /// Network egress control sent to CubeMaster. diff --git a/CubeAPI/src/handlers/agenthub.rs b/CubeAPI/src/handlers/agenthub.rs index b5f5573da..0b97861ee 100644 --- a/CubeAPI/src/handlers/agenthub.rs +++ b/CubeAPI/src/handlers/agenthub.rs @@ -458,8 +458,7 @@ pub async fn create_agent_instance( .create_sandbox(NewSandbox { template_id: template_id.clone(), timeout, - auto_pause: false, - auto_resume: None, + lifecycle: None, secure: None, allow_internet_access: Some(true), network: network_config, @@ -1995,8 +1994,7 @@ pub async fn clone_agent_instance( .create_sandbox(NewSandbox { template_id: snapshot_id.clone(), timeout, - auto_pause: false, - auto_resume: None, + lifecycle: None, secure: None, allow_internet_access: Some(true), network: network_config, diff --git a/CubeAPI/src/models/mod.rs b/CubeAPI/src/models/mod.rs index 328a9ed18..a97966f46 100644 --- a/CubeAPI/src/models/mod.rs +++ b/CubeAPI/src/models/mod.rs @@ -116,10 +116,37 @@ pub struct EgressRuleInject { pub format: Option, } -/// Auto-resume configuration for paused sandboxes. -#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] -pub struct SandboxAutoResumeConfig { - pub enabled: bool, +/// Sandbox lifecycle configuration. Mirrors the e2b SDK's `lifecycle` object — +/// see https://e2b.dev/docs/sandbox/auto-resume for the canonical reference. +/// +/// `on_timeout` decides what happens when the sandbox idle timer fires; the +/// historical default is "kill" (delete the sandbox) which matches today's +/// behaviour. `auto_resume` only takes effect when `on_timeout = "pause"` — +/// it tells the proxy/sidecar to wake a paused sandbox up automatically when +/// activity arrives, instead of returning an error. +#[derive(Debug, Clone, Default, Serialize, Deserialize, ToSchema)] +pub struct SandboxLifecycleConfig { + /// "kill" (default) | "pause". + #[serde(rename = "onTimeout", default)] + pub on_timeout: SandboxOnTimeout, + + /// Auto-resume on activity. Defaults to false. Only meaningful when + /// `on_timeout` is set to "pause". + #[serde(rename = "autoResume", default)] + pub auto_resume: bool, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +#[serde(rename_all = "lowercase")] +pub enum SandboxOnTimeout { + Kill, + Pause, +} + +impl Default for SandboxOnTimeout { + fn default() -> Self { + Self::Kill + } } /// Volume mount inside the sandbox. @@ -133,8 +160,9 @@ pub struct SandboxVolumeMount { /// Request body for POST /sandboxes /// Field names match exactly what the E2B SDK sends. -/// Rule: ID abbreviations → uppercase (templateID, sandboxID, envVars, autoPause); -/// allow_internet_access is a known SDK snake_case quirk. +/// Rule: ID abbreviations → uppercase (templateID, sandboxID, envVars); +/// allow_internet_access is a known SDK snake_case quirk; +/// lifecycle is a nested object — see SandboxLifecycleConfig. #[derive(Debug, Deserialize, Validate, ToSchema)] #[allow(dead_code)] pub struct NewSandbox { @@ -145,11 +173,11 @@ pub struct NewSandbox { #[serde(default = "default_timeout")] pub timeout: i32, - #[serde(rename = "autoPause", default)] - pub auto_pause: bool, - - #[serde(rename = "autoResume", skip_serializing_if = "Option::is_none")] - pub auto_resume: Option, + /// Sandbox lifecycle configuration. Maps to e2b's `lifecycle` object so + /// callers that already speak e2b can pass through unchanged. Absent + /// (None) means today's behaviour: idle sandboxes are killed. + #[serde(skip_serializing_if = "Option::is_none")] + pub lifecycle: Option, #[serde(skip_serializing_if = "Option::is_none")] pub secure: Option, diff --git a/CubeAPI/src/services/sandboxes.rs b/CubeAPI/src/services/sandboxes.rs index 63c8ae3fd..31bb2b2b7 100644 --- a/CubeAPI/src/services/sandboxes.rs +++ b/CubeAPI/src/services/sandboxes.rs @@ -138,6 +138,21 @@ impl SandboxService { let cube_network_config = build_cube_network_config(body.allow_internet_access, body.network.as_ref())?; + // Derive the two CubeMaster-side bools from the e2b-shaped lifecycle + // object. Absent lifecycle keeps today's behaviour: idle sandboxes + // are killed (auto_pause = false), and auto_resume defaults off. + let (auto_pause, auto_resume) = body + .lifecycle + .as_ref() + .map(|lc| { + use crate::models::SandboxOnTimeout; + ( + matches!(lc.on_timeout, SandboxOnTimeout::Pause), + lc.auto_resume, + ) + }) + .unwrap_or((false, false)); + let req = CreateSandboxRequest { request_id: new_request_id(), instance_type: self.instance_type.clone(), @@ -150,6 +165,8 @@ impl SandboxService { exposed_ports: vec![], network_type: Some("tap".to_string()), cube_network_config, + auto_pause, + auto_resume, }; let resp = self @@ -718,7 +735,7 @@ mod tests { use std::collections::HashMap; use super::{build_cube_network_config, filter_by_metadata, from_cubemaster_info}; - use crate::cubemaster::{ListSandboxResponse, SandboxInfo}; + use crate::cubemaster::{CreateSandboxRequest, ListSandboxResponse, SandboxInfo}; use crate::models::{ EgressRule, EgressRuleAction, EgressRuleInject, EgressRuleMatch, SandboxNetworkConfig, SandboxState, @@ -953,4 +970,112 @@ mod tests { .iter() .all(|sandbox| sandbox.state == SandboxState::Paused)); } + + /// CubeMaster keys lifecycle metadata off these exact JSON field names — + /// `auto_pause` / `auto_resume`. If they ever rename or get dropped during + /// serialization the auto-pause sidecar silently treats every new sandbox + /// as opted-out. Lock the wire shape down with a serialization snapshot. + #[test] + fn create_sandbox_request_serializes_lifecycle_flags() { + let mut req = CreateSandboxRequest { + request_id: "req-1".to_string(), + instance_type: "cubebox".to_string(), + timeout: Some(60), + annotations: HashMap::new(), + labels: None, + volumes: None, + containers: vec![], + exposed_ports: vec![], + network_type: None, + cube_network_config: None, + auto_pause: false, + auto_resume: false, + }; + + // Both false → both fields are omitted (skip_serializing_if = Not::not). + let json = serde_json::to_value(&req).unwrap(); + assert!( + json.get("auto_pause").is_none(), + "auto_pause=false should be omitted, got: {json}" + ); + assert!( + json.get("auto_resume").is_none(), + "auto_resume=false should be omitted, got: {json}" + ); + + // Flip on → fields appear with snake_case key matching CubeMaster's + // `json:"auto_pause,omitempty"` and `json:"auto_resume,omitempty"`. + req.auto_pause = true; + req.auto_resume = true; + let json = serde_json::to_value(&req).unwrap(); + assert_eq!(json.get("auto_pause"), Some(&serde_json::Value::Bool(true))); + assert_eq!( + json.get("auto_resume"), + Some(&serde_json::Value::Bool(true)) + ); + } + + /// The inbound API mirrors the e2b `lifecycle` object (camelCase nested + /// struct). CubeAPI then translates it to the two CubeMaster-side bools + /// when constructing the create-sandbox RPC. Verify the translation + /// covers each meaningful combination. + #[test] + fn lifecycle_object_translates_to_cubemaster_bools() { + use crate::models::{NewSandbox, SandboxLifecycleConfig, SandboxOnTimeout}; + + // Helper that mimics services::create_sandbox's lifecycle decoding. + fn translate(body: &NewSandbox) -> (bool, bool) { + body.lifecycle + .as_ref() + .map(|lc| { + ( + matches!(lc.on_timeout, SandboxOnTimeout::Pause), + lc.auto_resume, + ) + }) + .unwrap_or((false, false)) + } + + // Absent lifecycle => preserve historical behaviour. + let absent: NewSandbox = serde_json::from_value(serde_json::json!({ + "templateID": "tpl", + })) + .unwrap(); + assert_eq!(translate(&absent), (false, false)); + + // Explicit kill (with auto_resume=true) is still kill — auto_resume + // doesn't auto-imply pause. Server-side enforcement of the e2b + // semantic ("auto_resume only meaningful when on_timeout=pause") is + // delegated to CubeMaster. + let kill: NewSandbox = serde_json::from_value(serde_json::json!({ + "templateID": "tpl", + "lifecycle": {"onTimeout": "kill", "autoResume": true}, + })) + .unwrap(); + assert_eq!(translate(&kill), (false, true)); + + // Pause + auto_resume — the canonical e2b auto-resume case. + let pause_with_resume: NewSandbox = serde_json::from_value(serde_json::json!({ + "templateID": "tpl", + "lifecycle": {"onTimeout": "pause", "autoResume": true}, + })) + .unwrap(); + assert_eq!(translate(&pause_with_resume), (true, true)); + + // Pause without auto_resume — caller must call connect() manually. + let pause_only: NewSandbox = serde_json::from_value(serde_json::json!({ + "templateID": "tpl", + "lifecycle": {"onTimeout": "pause"}, + })) + .unwrap(); + assert_eq!(translate(&pause_only), (true, false)); + + // Empty lifecycle object — defaults: kill on timeout, no auto-resume. + let empty: NewSandbox = serde_json::from_value(serde_json::json!({ + "templateID": "tpl", + "lifecycle": {}, + })) + .unwrap(); + assert_eq!(translate(&empty), (false, false)); + } } diff --git a/CubeMaster/cmd/cubemaster/app/main.go b/CubeMaster/cmd/cubemaster/app/main.go index a5ff95bdc..b624ddc1f 100644 --- a/CubeMaster/cmd/cubemaster/app/main.go +++ b/CubeMaster/cmd/cubemaster/app/main.go @@ -26,6 +26,7 @@ import ( "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/cubelet/grpcconn" "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/errorcode" "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/instancecache" + "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/lifecycle" "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/localcache" "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/nodemeta" "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/scheduler" @@ -162,6 +163,14 @@ func coreInit(ctx context.Context, cfg *config.Config) error { return err } + // lifecycle wires the auto-pause / auto-resume metadata channel into the + // sandbox create/destroy hooks. It is non-fatal: a Redis hiccup must not + // block CubeMaster from serving sandboxes, only the sidecar's view goes + // stale until the next reconcile. + if err := lifecycle.Init(ctx); err != nil { + log.G(ctx).Warnf("lifecycle init fail (non-fatal): %v", err) + } + scheduler.InitScheduler(ctx) if err := sandbox.Init(ctx, cfg); err != nil { diff --git a/CubeMaster/pkg/lifecycle/init.go b/CubeMaster/pkg/lifecycle/init.go new file mode 100644 index 000000000..cc7f98bd0 --- /dev/null +++ b/CubeMaster/pkg/lifecycle/init.go @@ -0,0 +1,89 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +package lifecycle + +import ( + "context" + "time" + + "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/base/log" + "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/base/wrapredis" + "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/service/sandbox" + sandboxtypes "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/service/sandbox/types" + "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/task" +) + +// Init wires the lifecycle metadata channel into the sandbox create/destroy +// hooks. Call exactly once at process start, after wrapredis is reachable. +// +// Failures here are intentionally non-fatal: lifecycle metadata is an +// observability/coordination side channel for the auto-pause sidecar; if it +// is missing the rest of CubeMaster keeps working and sandboxes still serve +// traffic. Callers (main.go) should log a warning and proceed. +// +// We use the single shared wrapredis pool. The sidecar consumes lifecycle +// metadata and the bypass_host_proxy map from the same Redis instance, so +// any pool that can write proxy entries can also write lifecycle entries. +func Init(ctx context.Context) error { + pool := wrapredis.GetRedis() + if isNilPool(pool) { + log.G(ctx).Warnf("lifecycle: redis pool unavailable; auto-pause metadata channel disabled") + return nil + } + + store := NewStore(pool) + setDefaultStore(store) + + sandbox.RegisterAfterCreateSandboxSuccessHook(onAfterCreate) + // Both the synchronous destroy path (sandbox_remove.callCubelet) and the + // asynchronous task executor end with their own success hook. Register on + // both so we publish exactly once for either deletion mode. + sandbox.RegisterAfterDestroySandboxSuccessHook(onAfterDestroy) + task.RegisterAfterDestroyTaskSuccessHook(onAfterDestroy) + + log.G(ctx).Infof("lifecycle: auto-pause metadata channel ready (key=%s, stream=%s)", + MetaKey, EventStreamKey) + return nil +} + +// isNilPool guards against wrapredis.GetRedis returning a typed-nil +// (*RedisWrap)(nil) — that satisfies a nil interface check via != nil but +// is functionally unusable. We unwrap by inspecting the concrete pool. +func isNilPool(w *wrapredis.RedisWrap) bool { + return w == nil || w.RedisConnPool == nil +} + +func onAfterCreate(ctx context.Context, sandboxID, hostID, hostIP string, req *sandboxtypes.CreateCubeSandboxReq) error { + store := getDefaultStore() + if store == nil || req == nil { + return nil + } + meta := &SandboxLifecycleMeta{ + SandboxID: sandboxID, + HostID: hostID, + HostIP: hostIP, + InstanceType: req.InstanceType, + TimeoutSeconds: req.Timeout, + AutoPause: req.AutoPause, + AutoResume: req.AutoResume, + CreatedAt: time.Now().UnixMilli(), + } + if req.Annotations != nil { + // Template ID is conventionally carried via annotations from CubeAPI; + // the field is informational so we tolerate it being absent. + if v, ok := req.Annotations["template_id"]; ok { + meta.TemplateID = v + } + } + store.PublishCreate(ctx, meta) + return nil +} + +func onAfterDestroy(ctx context.Context, sandboxID string) error { + if store := getDefaultStore(); store != nil { + store.PublishDelete(ctx, sandboxID) + } + return nil +} diff --git a/CubeMaster/pkg/lifecycle/schema.go b/CubeMaster/pkg/lifecycle/schema.go new file mode 100644 index 000000000..aa5c938f6 --- /dev/null +++ b/CubeMaster/pkg/lifecycle/schema.go @@ -0,0 +1,68 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +// Package lifecycle owns the cross-process metadata channel used by +// CubeProxy-sidecar to track auto-pause / auto-resume decisions. +// +// CubeMaster is the single writer for the canonical view: +// +// - cube:sandbox:meta HSet, field=sandboxID, value=JSON snapshot. +// Sidecars HGETALL it on startup to bootstrap the registry. +// - cube:sandbox:events Stream, append-only event log of +// create/delete operations. Sidecars consume via XREADGROUP for +// incremental updates after the bootstrap. +// +// Updates (pause/resume action) intentionally do NOT publish to the stream: +// state transitions are driven and observed by the sidecar itself, so making +// CubeMaster also publish them would just be a redundant round trip. +package lifecycle + +// Redis key / field constants. Keep them centralized so the sidecar (Go) and +// any other consumer can import the same source of truth. +const ( + // MetaKey is the HSet snapshot of every live sandbox the sidecar should + // know about. Field = sandbox ID, value = JSON-encoded SandboxLifecycleMeta. + MetaKey = "cube:sandbox:meta" + + // EventStreamKey is the append-only stream of create/delete events. The + // sidecar maintains a consumer group on it; entries trim with MAXLEN ~. + EventStreamKey = "cube:sandbox:events" + + // EventStreamMaxLen caps the stream so an offline sidecar cannot drive + // unbounded Redis growth. Sidecars also bootstrap from MetaKey, so any + // trimmed events are recovered on the next full sync. + EventStreamMaxLen = 100000 +) + +// Event op codes carried in stream entries. +const ( + OpCreate = "create" + OpDelete = "delete" +) + +// Stream entry field names. Stream values are flat key/value pairs in redigo, +// so we model the schema as constants rather than a struct. +const ( + FieldOp = "op" + FieldSandboxID = "sandbox_id" + FieldPayload = "payload" + FieldTimestamp = "ts" +) + +// SandboxLifecycleMeta is the JSON value stored under MetaKey[sandboxID] and +// also the payload field of OpCreate stream entries. OpDelete entries omit +// the payload field — the sandbox ID is enough to drop a registry entry. +type SandboxLifecycleMeta struct { + SandboxID string `json:"sandbox_id"` + TemplateID string `json:"template_id,omitempty"` + HostID string `json:"host_id,omitempty"` + HostIP string `json:"host_ip,omitempty"` + InstanceType string `json:"instance_type,omitempty"` + TimeoutSeconds int `json:"timeout_seconds,omitempty"` + AutoPause bool `json:"auto_pause,omitempty"` + AutoResume bool `json:"auto_resume,omitempty"` + // CreatedAt is unix milliseconds. Sidecars use it as the initial + // "last active" baseline before they ever observe a real request. + CreatedAt int64 `json:"created_at,omitempty"` +} diff --git a/CubeMaster/pkg/lifecycle/store.go b/CubeMaster/pkg/lifecycle/store.go new file mode 100644 index 000000000..9f15190e7 --- /dev/null +++ b/CubeMaster/pkg/lifecycle/store.go @@ -0,0 +1,112 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +package lifecycle + +import ( + "context" + "encoding/json" + "strconv" + "sync/atomic" + "time" + + "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/base/log" +) + +// redisDoer is the minimal redigo-shaped surface the writer needs. wrapredis's +// *RedisWrap satisfies it; tests substitute a fake. +type redisDoer interface { + Do(cmd string, args ...interface{}) (interface{}, error) +} + +// Store performs the actual Redis writes. It is intentionally tiny and never +// returns errors to its callers — every error is logged at warn level and +// swallowed so a Redis hiccup cannot fail a sandbox create/destroy. +type Store struct { + doer redisDoer + enabled atomic.Bool +} + +// NewStore wires a Store onto the supplied redis client. +func NewStore(doer redisDoer) *Store { + s := &Store{doer: doer} + s.enabled.Store(true) + return s +} + +// SetEnabled toggles all writes. When disabled the Store becomes a no-op so +// the lifecycle subsystem can be feature-flagged off without recompiling. +func (s *Store) SetEnabled(v bool) { + if s == nil { + return + } + s.enabled.Store(v) +} + +// PublishCreate persists a freshly-created sandbox to the registry: HSET the +// meta snapshot, then XADD an OpCreate event. +func (s *Store) PublishCreate(ctx context.Context, meta *SandboxLifecycleMeta) { + if s == nil || !s.enabled.Load() || s.doer == nil || meta == nil || meta.SandboxID == "" { + return + } + + payload, err := json.Marshal(meta) + if err != nil { + log.G(ctx).Warnf("lifecycle: marshal meta sandbox=%s: %v", meta.SandboxID, err) + return + } + + if _, err := s.doer.Do("HSET", MetaKey, meta.SandboxID, payload); err != nil { + log.G(ctx).Warnf("lifecycle: HSET %s %s failed: %v", MetaKey, meta.SandboxID, err) + // Continue: stream event is still useful for sidecars that already + // have a partial view, and the next reconcile cycle will retry. + } + + if _, err := s.xadd(OpCreate, meta.SandboxID, payload); err != nil { + log.G(ctx).Warnf("lifecycle: XADD create %s failed: %v", meta.SandboxID, err) + } +} + +// PublishDelete drops the registry entry. Stream payload is empty; sidecars +// only need the sandbox ID to evict. +func (s *Store) PublishDelete(ctx context.Context, sandboxID string) { + if s == nil || !s.enabled.Load() || s.doer == nil || sandboxID == "" { + return + } + + if _, err := s.doer.Do("HDEL", MetaKey, sandboxID); err != nil { + log.G(ctx).Warnf("lifecycle: HDEL %s %s failed: %v", MetaKey, sandboxID, err) + } + + if _, err := s.xadd(OpDelete, sandboxID, nil); err != nil { + log.G(ctx).Warnf("lifecycle: XADD delete %s failed: %v", sandboxID, err) + } +} + +// xadd builds an XADD ... MAXLEN ~ * op sandbox_id [payload

] +// ts command and dispatches it. +func (s *Store) xadd(op, sandboxID string, payload []byte) (interface{}, error) { + args := make([]interface{}, 0, 12) + args = append(args, + EventStreamKey, + "MAXLEN", "~", strconv.Itoa(EventStreamMaxLen), + "*", + FieldOp, op, + FieldSandboxID, sandboxID, + FieldTimestamp, time.Now().UnixMilli(), + ) + if len(payload) > 0 { + args = append(args, FieldPayload, payload) + } + return s.doer.Do("XADD", args...) +} + +// defaultStore is the package-level singleton wired by Init(). Hooks call into +// it from createSandbox / destroySandbox, where threading a *Store explicitly +// would require reaching into the sandbox package's hook signatures. +var defaultStore atomic.Pointer[Store] + +func setDefaultStore(s *Store) { defaultStore.Store(s) } + +func getDefaultStore() *Store { return defaultStore.Load() } diff --git a/CubeMaster/pkg/lifecycle/store_test.go b/CubeMaster/pkg/lifecycle/store_test.go new file mode 100644 index 000000000..c6a52a062 --- /dev/null +++ b/CubeMaster/pkg/lifecycle/store_test.go @@ -0,0 +1,196 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +package lifecycle + +import ( + "context" + "encoding/json" + "errors" + "sync" + "testing" +) + +// recordedCall captures one Do invocation for later assertion. +type recordedCall struct { + cmd string + args []interface{} +} + +type fakeRedis struct { + mu sync.Mutex + calls []recordedCall + // errOn maps command name -> error to return on the Nth call (counter-based). + failHSET bool + failHDEL bool + failXADD bool +} + +func (f *fakeRedis) Do(cmd string, args ...interface{}) (interface{}, error) { + f.mu.Lock() + defer f.mu.Unlock() + f.calls = append(f.calls, recordedCall{cmd: cmd, args: args}) + switch cmd { + case "HSET": + if f.failHSET { + return nil, errors.New("HSET boom") + } + case "HDEL": + if f.failHDEL { + return nil, errors.New("HDEL boom") + } + case "XADD": + if f.failXADD { + return nil, errors.New("XADD boom") + } + } + return "OK", nil +} + +func (f *fakeRedis) snapshot() []recordedCall { + f.mu.Lock() + defer f.mu.Unlock() + out := make([]recordedCall, len(f.calls)) + copy(out, f.calls) + return out +} + +func TestSandboxLifecycleMeta_JSONRoundTrip(t *testing.T) { + in := SandboxLifecycleMeta{ + SandboxID: "sbx-1", + TemplateID: "tpl-1", + HostID: "host-1", + HostIP: "10.0.0.1", + InstanceType: "cubebox", + TimeoutSeconds: 60, + AutoPause: true, + AutoResume: true, + CreatedAt: 1700000000000, + } + b, err := json.Marshal(in) + if err != nil { + t.Fatalf("marshal: %v", err) + } + var out SandboxLifecycleMeta + if err := json.Unmarshal(b, &out); err != nil { + t.Fatalf("unmarshal: %v", err) + } + if out != in { + t.Fatalf("round trip mismatch: got %+v want %+v", out, in) + } +} + +func TestStore_PublishCreate_HappyPath(t *testing.T) { + r := &fakeRedis{} + s := NewStore(r) + + meta := &SandboxLifecycleMeta{ + SandboxID: "sbx-42", + TimeoutSeconds: 60, + AutoPause: true, + } + s.PublishCreate(context.Background(), meta) + + calls := r.snapshot() + if len(calls) != 2 { + t.Fatalf("want 2 calls (HSET + XADD), got %d: %+v", len(calls), calls) + } + if calls[0].cmd != "HSET" || calls[0].args[0] != MetaKey || calls[0].args[1] != "sbx-42" { + t.Fatalf("HSET args wrong: %+v", calls[0]) + } + if calls[1].cmd != "XADD" || calls[1].args[0] != EventStreamKey { + t.Fatalf("XADD args wrong: %+v", calls[1]) + } + // XADD args layout: stream, MAXLEN, ~, N, *, op, OpCreate, sandbox_id, id, ts, ms, payload, bytes + if calls[1].args[5] != FieldOp || calls[1].args[6] != OpCreate { + t.Fatalf("XADD op field wrong: %+v", calls[1].args) + } + if calls[1].args[7] != FieldSandboxID || calls[1].args[8] != "sbx-42" { + t.Fatalf("XADD sandbox_id field wrong: %+v", calls[1].args) + } + // payload must round-trip through JSON + payloadBytes, ok := calls[0].args[2].([]byte) + if !ok { + t.Fatalf("HSET payload not bytes: %T", calls[0].args[2]) + } + var got SandboxLifecycleMeta + if err := json.Unmarshal(payloadBytes, &got); err != nil { + t.Fatalf("payload json: %v", err) + } + if got.SandboxID != "sbx-42" || !got.AutoPause || got.TimeoutSeconds != 60 { + t.Fatalf("payload wrong: %+v", got) + } +} + +func TestStore_PublishCreate_HSETFailureStillEmitsXADD(t *testing.T) { + r := &fakeRedis{failHSET: true} + s := NewStore(r) + + s.PublishCreate(context.Background(), &SandboxLifecycleMeta{SandboxID: "sbx-1"}) + + calls := r.snapshot() + if len(calls) != 2 { + t.Fatalf("want 2 calls even when HSET fails, got %d", len(calls)) + } + if calls[1].cmd != "XADD" { + t.Fatalf("expected XADD as second call, got %s", calls[1].cmd) + } +} + +func TestStore_PublishDelete(t *testing.T) { + r := &fakeRedis{} + s := NewStore(r) + + s.PublishDelete(context.Background(), "sbx-9") + + calls := r.snapshot() + if len(calls) != 2 { + t.Fatalf("want HDEL + XADD, got %d", len(calls)) + } + if calls[0].cmd != "HDEL" || calls[0].args[1] != "sbx-9" { + t.Fatalf("HDEL wrong: %+v", calls[0]) + } + if calls[1].cmd != "XADD" || calls[1].args[6] != OpDelete { + t.Fatalf("XADD op should be %q, got %+v", OpDelete, calls[1].args) + } + // OpDelete carries no payload field. + for _, a := range calls[1].args { + if s, ok := a.(string); ok && s == FieldPayload { + t.Fatalf("delete event should not include payload field: %+v", calls[1].args) + } + } +} + +func TestStore_DisabledIsNoOp(t *testing.T) { + r := &fakeRedis{} + s := NewStore(r) + s.SetEnabled(false) + + s.PublishCreate(context.Background(), &SandboxLifecycleMeta{SandboxID: "sbx-1"}) + s.PublishDelete(context.Background(), "sbx-1") + + if got := len(r.snapshot()); got != 0 { + t.Fatalf("disabled store should make zero calls, got %d", got) + } +} + +func TestStore_NilGuards(t *testing.T) { + // nil store, nil doer, nil meta, empty id — all must be safe. + var s *Store + s.PublishCreate(context.Background(), &SandboxLifecycleMeta{SandboxID: "x"}) + s.PublishDelete(context.Background(), "x") + + s2 := NewStore(nil) + s2.PublishCreate(context.Background(), &SandboxLifecycleMeta{SandboxID: "x"}) + s2.PublishDelete(context.Background(), "x") + + r := &fakeRedis{} + s3 := NewStore(r) + s3.PublishCreate(context.Background(), nil) + s3.PublishCreate(context.Background(), &SandboxLifecycleMeta{}) + s3.PublishDelete(context.Background(), "") + if got := len(r.snapshot()); got != 0 { + t.Fatalf("nil/empty inputs must not reach Redis, got %d calls", got) + } +} diff --git a/CubeMaster/pkg/service/sandbox/runtime_ref_hook.go b/CubeMaster/pkg/service/sandbox/runtime_ref_hook.go index d7e79e78c..3e848861d 100644 --- a/CubeMaster/pkg/service/sandbox/runtime_ref_hook.go +++ b/CubeMaster/pkg/service/sandbox/runtime_ref_hook.go @@ -6,21 +6,65 @@ package sandbox import ( "context" + "sync" + "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/base/log" "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/service/sandbox/types" ) -var afterDestroySandboxSuccess func(context.Context, string) error +// destroyHooks holds every after-destroy callback registered at startup. We +// fan out to all of them on success so multiple subsystems (templatecenter, +// lifecycle metadata, ...) can react without stepping on each other. +var ( + destroyHooksMu sync.RWMutex + destroyHooks []func(context.Context, string) error +) + +// RegisterAfterDestroySandboxSuccessHook appends a hook to the destroy chain. +// Hooks run sequentially in registration order; an individual hook's error is +// returned to the caller of runAfterDestroySandboxSuccessHook (joined when +// multiple hooks fail) but does NOT short-circuit later hooks. +func RegisterAfterDestroySandboxSuccessHook(hook func(context.Context, string) error) { + if hook == nil { + return + } + destroyHooksMu.Lock() + destroyHooks = append(destroyHooks, hook) + destroyHooksMu.Unlock() +} +// SetAfterDestroySandboxSuccessHook is retained for backward compatibility +// with single-registration callers (templatecenter). It now appends to the +// chain rather than replacing it; callers that genuinely need replacement +// semantics should use ResetAfterDestroySandboxSuccessHooks first. func SetAfterDestroySandboxSuccessHook(hook func(context.Context, string) error) { - afterDestroySandboxSuccess = hook + RegisterAfterDestroySandboxSuccessHook(hook) +} + +// ResetAfterDestroySandboxSuccessHooks clears every registered destroy hook. +// Test-only helper; production code never calls it. +func ResetAfterDestroySandboxSuccessHooks() { + destroyHooksMu.Lock() + destroyHooks = nil + destroyHooksMu.Unlock() } func runAfterDestroySandboxSuccessHook(ctx context.Context, sandboxID string) error { - if afterDestroySandboxSuccess == nil { - return nil + destroyHooksMu.RLock() + hooks := append([]func(context.Context, string) error(nil), destroyHooks...) + destroyHooksMu.RUnlock() + + var firstErr error + for _, h := range hooks { + if err := h(ctx, sandboxID); err != nil { + if firstErr == nil { + firstErr = err + } else { + log.G(ctx).Warnf("afterDestroySandboxSuccess hook chain error: %v", err) + } + } } - return afterDestroySandboxSuccess(ctx, sandboxID) + return firstErr } // CreateSandboxSuccessHook is invoked after a sandbox is successfully created @@ -29,17 +73,49 @@ func runAfterDestroySandboxSuccessHook(ctx context.Context, sandboxID string) er // error and continues. type CreateSandboxSuccessHook func(ctx context.Context, sandboxID, hostID, hostIP string, req *types.CreateCubeSandboxReq) error -var afterCreateSandboxSuccess CreateSandboxSuccessHook +var ( + createHooksMu sync.RWMutex + createHooks []CreateSandboxSuccessHook +) + +// RegisterAfterCreateSandboxSuccessHook appends a hook to the create chain. +func RegisterAfterCreateSandboxSuccessHook(hook CreateSandboxSuccessHook) { + if hook == nil { + return + } + createHooksMu.Lock() + createHooks = append(createHooks, hook) + createHooksMu.Unlock() +} -// SetAfterCreateSandboxSuccessHook registers a single hook to receive -// successful create events. Re-registering overwrites the previous hook. +// SetAfterCreateSandboxSuccessHook keeps the historical single-registration +// signature working: it appends rather than replaces. func SetAfterCreateSandboxSuccessHook(hook CreateSandboxSuccessHook) { - afterCreateSandboxSuccess = hook + RegisterAfterCreateSandboxSuccessHook(hook) +} + +// ResetAfterCreateSandboxSuccessHooks clears every registered create hook. +// Test-only helper. +func ResetAfterCreateSandboxSuccessHooks() { + createHooksMu.Lock() + createHooks = nil + createHooksMu.Unlock() } func runAfterCreateSandboxSuccessHook(ctx context.Context, sandboxID, hostID, hostIP string, req *types.CreateCubeSandboxReq) error { - if afterCreateSandboxSuccess == nil { - return nil + createHooksMu.RLock() + hooks := append([]CreateSandboxSuccessHook(nil), createHooks...) + createHooksMu.RUnlock() + + var firstErr error + for _, h := range hooks { + if err := h(ctx, sandboxID, hostID, hostIP, req); err != nil { + if firstErr == nil { + firstErr = err + } else { + log.G(ctx).Warnf("afterCreateSandboxSuccess hook chain error: %v", err) + } + } } - return afterCreateSandboxSuccess(ctx, sandboxID, hostID, hostIP, req) + return firstErr } diff --git a/CubeMaster/pkg/service/sandbox/runtime_ref_hook_test.go b/CubeMaster/pkg/service/sandbox/runtime_ref_hook_test.go new file mode 100644 index 000000000..8bfee791d --- /dev/null +++ b/CubeMaster/pkg/service/sandbox/runtime_ref_hook_test.go @@ -0,0 +1,74 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +package sandbox + +import ( + "context" + "errors" + "testing" + + "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/service/sandbox/types" +) + +func TestDestroyHookChain_FIFOAndContinueOnError(t *testing.T) { + ResetAfterDestroySandboxSuccessHooks() + defer ResetAfterDestroySandboxSuccessHooks() + + var order []string + wantErr := errors.New("h2 boom") + + RegisterAfterDestroySandboxSuccessHook(func(_ context.Context, id string) error { + order = append(order, "h1:"+id) + return nil + }) + RegisterAfterDestroySandboxSuccessHook(func(_ context.Context, id string) error { + order = append(order, "h2:"+id) + return wantErr + }) + RegisterAfterDestroySandboxSuccessHook(func(_ context.Context, id string) error { + order = append(order, "h3:"+id) + return nil + }) + + err := runAfterDestroySandboxSuccessHook(context.Background(), "sbx-x") + if !errors.Is(err, wantErr) { + t.Fatalf("first error must propagate, got %v", err) + } + if got := order; len(got) != 3 || got[0] != "h1:sbx-x" || got[1] != "h2:sbx-x" || got[2] != "h3:sbx-x" { + t.Fatalf("hooks ran out of order or stopped early: %v", got) + } +} + +func TestCreateHookChain_FIFO(t *testing.T) { + ResetAfterCreateSandboxSuccessHooks() + defer ResetAfterCreateSandboxSuccessHooks() + + var order []string + RegisterAfterCreateSandboxSuccessHook(func(_ context.Context, id, _, _ string, _ *types.CreateCubeSandboxReq) error { + order = append(order, "a:"+id) + return nil + }) + RegisterAfterCreateSandboxSuccessHook(func(_ context.Context, id, _, _ string, _ *types.CreateCubeSandboxReq) error { + order = append(order, "b:"+id) + return nil + }) + + if err := runAfterCreateSandboxSuccessHook(context.Background(), "sbx-y", "h-1", "1.2.3.4", &types.CreateCubeSandboxReq{}); err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(order) != 2 || order[0] != "a:sbx-y" || order[1] != "b:sbx-y" { + t.Fatalf("hooks order wrong: %v", order) + } +} + +func TestNilHookIgnored(t *testing.T) { + ResetAfterDestroySandboxSuccessHooks() + defer ResetAfterDestroySandboxSuccessHooks() + + RegisterAfterDestroySandboxSuccessHook(nil) + if err := runAfterDestroySandboxSuccessHook(context.Background(), "x"); err != nil { + t.Fatalf("nil hook must be skipped silently, got %v", err) + } +} diff --git a/CubeMaster/pkg/service/sandbox/types/types.go b/CubeMaster/pkg/service/sandbox/types/types.go index b4caa0ae8..93a3d3b15 100644 --- a/CubeMaster/pkg/service/sandbox/types/types.go +++ b/CubeMaster/pkg/service/sandbox/types/types.go @@ -52,6 +52,17 @@ type CreateCubeSandboxReq struct { RuntimeHandler string `json:"runtime_handler,omitempty"` Namespace string `json:"namespace,omitempty"` + + // AutoPause asks CubeMaster (via the lifecycle subsystem) to publish this + // sandbox to the auto-pause registry: once the proxy reports it has been + // idle for `Timeout` seconds the sidecar will pause it. Default false + // preserves the historical "never pause" behavior. + AutoPause bool `json:"auto_pause,omitempty"` + + // AutoResume signals that an incoming request hitting a paused sandbox + // should transparently resume it. Default false means a request hitting a + // paused sandbox returns an error instead. + AutoResume bool `json:"auto_resume,omitempty"` } func (r *CreateCubeSandboxReq) UnmarshalJSON(data []byte) error { diff --git a/CubeMaster/pkg/task/runtime_ref_hook.go b/CubeMaster/pkg/task/runtime_ref_hook.go index 29a9cdff4..77b155ec7 100644 --- a/CubeMaster/pkg/task/runtime_ref_hook.go +++ b/CubeMaster/pkg/task/runtime_ref_hook.go @@ -4,17 +4,55 @@ package task -import "context" +import ( + "context" + "sync" -var afterDestroyTaskSuccess func(context.Context, string) error + "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/base/log" +) +var ( + destroyTaskHooksMu sync.RWMutex + destroyTaskHooks []func(context.Context, string) error +) + +// RegisterAfterDestroyTaskSuccessHook appends a hook fired after the async +// destroy task confirms the sandbox is gone. Hooks run sequentially. +func RegisterAfterDestroyTaskSuccessHook(hook func(context.Context, string) error) { + if hook == nil { + return + } + destroyTaskHooksMu.Lock() + destroyTaskHooks = append(destroyTaskHooks, hook) + destroyTaskHooksMu.Unlock() +} + +// SetAfterDestroyTaskSuccessHook retained for backward compat: appends. func SetAfterDestroyTaskSuccessHook(hook func(context.Context, string) error) { - afterDestroyTaskSuccess = hook + RegisterAfterDestroyTaskSuccessHook(hook) +} + +// ResetAfterDestroyTaskSuccessHooks clears every registered hook (test-only). +func ResetAfterDestroyTaskSuccessHooks() { + destroyTaskHooksMu.Lock() + destroyTaskHooks = nil + destroyTaskHooksMu.Unlock() } func runAfterDestroyTaskSuccessHook(ctx context.Context, sandboxID string) error { - if afterDestroyTaskSuccess == nil { - return nil + destroyTaskHooksMu.RLock() + hooks := append([]func(context.Context, string) error(nil), destroyTaskHooks...) + destroyTaskHooksMu.RUnlock() + + var firstErr error + for _, h := range hooks { + if err := h(ctx, sandboxID); err != nil { + if firstErr == nil { + firstErr = err + } else { + log.G(ctx).Warnf("afterDestroyTaskSuccess hook chain error: %v", err) + } + } } - return afterDestroyTaskSuccess(ctx, sandboxID) + return firstErr } diff --git a/CubeProxy/.gitignore b/CubeProxy/.gitignore new file mode 100644 index 000000000..ae3c17260 --- /dev/null +++ b/CubeProxy/.gitignore @@ -0,0 +1 @@ +/bin/ diff --git a/CubeProxy/Dockerfile b/CubeProxy/Dockerfile index d83783bc9..c55b70082 100644 --- a/CubeProxy/Dockerfile +++ b/CubeProxy/Dockerfile @@ -16,6 +16,19 @@ COPY rotate_nginx_log.sh /usr/local/openresty/nginx/sbin/rotate_nginx_log.sh COPY root /etc/crontabs/root COPY start.sh /usr/local/openresty/nginx/sbin/start.sh +# The auto-pause sidecar binary is built outside Docker (so we don't have to +# carry a Go toolchain into the runtime image) and dropped at this exact path +# by either: +# - CubeProxy/Makefile prebuild-sidecar (in-tree dev path) +# - deploy/one-click/build-release-bundle.sh (release bundle path; pre-built +# binary is copied into build-context/bin/cube-proxy-sidecar) +# +# The binary is REQUIRED. Both build flows now refuse to ship a stub — a +# missing or non-statically-linked sidecar would either crash at container +# start or silently hide the lifecycle feature, both of which are worse +# failure modes than failing the docker build right here. +COPY bin/cube-proxy-sidecar /usr/local/openresty/nginx/sbin/cube-proxy-sidecar + EXPOSE 8080 8081 STOPSIGNAL SIGQUIT diff --git a/CubeProxy/Makefile b/CubeProxy/Makefile index ab929e240..457082505 100644 --- a/CubeProxy/Makefile +++ b/CubeProxy/Makefile @@ -5,6 +5,13 @@ IMAGE_TAG := $(VERSION)-$(DATE)-$(COMMIT) IMAGE_LOCAL := cube-proxy +# CubeProxy ships an auto-pause sidecar binary inside the same image; we +# pre-build it on the host (via a Go toolchain) before invoking docker build, +# so the runtime image does not have to carry a Go toolchain. +SIDECAR_SRC := $(CURDIR)/sidecar +SIDECAR_OUT_DIR := $(CURDIR)/bin +SIDECAR_OUT := $(SIDECAR_OUT_DIR)/cube-proxy-sidecar + ifeq ($(V),1) Q = msg = @@ -26,8 +33,45 @@ test: build release: build .PHONY: build -build: +build: prebuild-sidecar $(call msg,BUILD IMAGE $(IMAGE_LOCAL):$(IMAGE_TAG)) $(Q)docker build --rm $(QUIET) \ -t $(IMAGE_LOCAL):$(IMAGE_TAG) \ . + +# prebuild-sidecar produces a static linux/amd64 binary at +# bin/cube-proxy-sidecar so the Dockerfile can simply COPY it into the +# musl-based openresty:alpine-fat image. The binary MUST be statically +# linked — CGO_ENABLED=0 plus the netgo/osusergo build tags forces the +# pure-Go DNS resolver and pure-Go user lookups, removing the implicit +# dependency on glibc's ld-linux-x86-64.so.2. Without these the binary +# fails at exec with rc=127 / "required file not found" inside Alpine. +# +# Hard requirement: the sidecar source tree must be present and `go` must +# be on PATH. We refuse to fall back to a stub binary — silently shipping +# a placeholder hides real configuration errors and was the source of an +# embarrassing class of "lifecycle just doesn't work" reports. Operators +# building partial checkouts must explicitly opt out via a different +# image flow. +.PHONY: prebuild-sidecar +prebuild-sidecar: + $(Q)mkdir -p $(SIDECAR_OUT_DIR) + $(Q)if [ ! -f "$(SIDECAR_SRC)/go.mod" ]; then \ + echo "ERROR: cube-proxy-sidecar source not found at $(SIDECAR_SRC)" >&2; \ + echo " cube-proxy image cannot be built without it." >&2; \ + exit 1; \ + fi + $(Q)if ! command -v go >/dev/null 2>&1; then \ + echo "ERROR: 'go' toolchain not found on PATH" >&2; \ + echo " cube-proxy-sidecar must be built before docker build." >&2; \ + exit 1; \ + fi + $(call msg,BUILD,cube-proxy-sidecar) + $(Q)cd "$(SIDECAR_SRC)" && \ + CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \ + go build -trimpath -tags 'netgo osusergo' -ldflags '-s -w' \ + -o "$(SIDECAR_OUT)" ./cmd/sidecar + +.PHONY: clean +clean: + $(Q)rm -rf $(SIDECAR_OUT_DIR) diff --git a/CubeProxy/lua/admin_phase.lua b/CubeProxy/lua/admin_phase.lua new file mode 100644 index 000000000..583519499 --- /dev/null +++ b/CubeProxy/lua/admin_phase.lua @@ -0,0 +1,206 @@ +-- admin_phase.lua +-- +-- Loopback-only admin endpoints used by CubeProxy-sidecar to drive the +-- auto-pause / auto-resume coordination dicts. See nginx.conf admin server +-- block for routing. +-- +-- All requests are loopback-bound (the listen directive is 127.0.0.1:8082); +-- additionally, when $cube_admin_token is non-empty, requests must carry +-- a matching X-Cube-Admin-Token header. Mismatch → 403. +-- +-- Request bodies are JSON; responses are JSON. Errors carry HTTP status + +-- {"error": "..."} body. + +local cjson = require "cjson.safe" + +local META = ngx.shared.cube_sandbox_meta +local STATE = ngx.shared.cube_sandbox_state +local LAST = ngx.shared.cube_sandbox_last_active + +-- ── helpers ──────────────────────────────────────────────────────────────── + +local function reply(status, body) + ngx.status = status + ngx.header["Content-Type"] = "application/json" + if body ~= nil then + ngx.print(cjson.encode(body)) + end + ngx.exit(status) +end + +local function reply_error(status, msg) + reply(status, { error = msg }) +end + +local function check_token() + local expected = ngx.var.cube_admin_token + if not expected or expected == "" then + return + end + local got = ngx.var.http_x_cube_admin_token + if got ~= expected then + reply_error(ngx.HTTP_FORBIDDEN, "admin token mismatch") + end +end + +local function read_json_body() + ngx.req.read_body() + local raw = ngx.req.get_body_data() + if not raw or raw == "" then + return nil, "empty body" + end + local obj, err = cjson.decode(raw) + if not obj then + return nil, "invalid json: " .. tostring(err) + end + if type(obj) ~= "table" then + return nil, "body must be a JSON object" + end + return obj, nil +end + +local function require_string(obj, key) + local v = obj[key] + if type(v) ~= "string" or v == "" then + return nil, string.format("field %q must be a non-empty string", key) + end + return v, nil +end + +-- ── handlers ─────────────────────────────────────────────────────────────── + +-- POST /admin/meta/upsert +-- body: {"sandbox_id": "...", ...arbitrary metadata...} +-- semantics: stores the JSON-encoded metadata under sandbox_id. Sidecar +-- re-pushes the full snapshot on reconnect, so "set" is enough; +-- we don't merge. +local function handle_meta_upsert() + local obj, err = read_json_body() + if not obj then return reply_error(ngx.HTTP_BAD_REQUEST, err) end + local sid, e2 = require_string(obj, "sandbox_id") + if not sid then return reply_error(ngx.HTTP_BAD_REQUEST, e2) end + + local payload = cjson.encode(obj) + local ok, set_err, forcible = META:set(sid, payload) + if not ok then + return reply_error(ngx.HTTP_INTERNAL_SERVER_ERROR, + "meta set failed: " .. tostring(set_err)) + end + if forcible then + ngx.log(ngx.WARN, "LEVEL_WARN||", + "cube_sandbox_meta dict full, an entry was evicted") + end + return reply(ngx.HTTP_OK, { ok = true }) +end + +-- POST /admin/meta/delete +-- body: {"sandbox_id": "..."} +-- semantics: removes meta + state + last_active in one shot, matching the +-- "sandbox is gone" lifecycle event. +local function handle_meta_delete() + local obj, err = read_json_body() + if not obj then return reply_error(ngx.HTTP_BAD_REQUEST, err) end + local sid, e2 = require_string(obj, "sandbox_id") + if not sid then return reply_error(ngx.HTTP_BAD_REQUEST, e2) end + + META:delete(sid) + STATE:delete(sid) + LAST:delete(sid) + return reply(ngx.HTTP_OK, { ok = true }) +end + +-- POST /admin/state +-- body: {"sandbox_id": "...", "state": "running|pausing|paused"} +local function handle_state() + local obj, err = read_json_body() + if not obj then return reply_error(ngx.HTTP_BAD_REQUEST, err) end + local sid, e2 = require_string(obj, "sandbox_id") + if not sid then return reply_error(ngx.HTTP_BAD_REQUEST, e2) end + local st, e3 = require_string(obj, "state") + if not st then return reply_error(ngx.HTTP_BAD_REQUEST, e3) end + if st ~= "running" and st ~= "pausing" and st ~= "paused" then + return reply_error(ngx.HTTP_BAD_REQUEST, + "state must be one of running|pausing|paused") + end + + local ok, set_err, forcible = STATE:set(sid, st) + if not ok then + return reply_error(ngx.HTTP_INTERNAL_SERVER_ERROR, + "state set failed: " .. tostring(set_err)) + end + if forcible then + ngx.log(ngx.WARN, "LEVEL_WARN||", + "cube_sandbox_state dict full, an entry was evicted") + end + return reply(ngx.HTTP_OK, { ok = true }) +end + +-- GET /admin/last_active +-- GET /admin/last_active?since= +-- Returns {"now": , "entries": {"": , ...}}. +-- `since` filters: only entries with ts > since are included. Useful for +-- the sidecar's incremental polling loop. +local function handle_last_active() + local args = ngx.req.get_uri_args(2) + local since = tonumber(args.since) or 0 + + local entries = {} + -- get_keys(0) returns up to 1024 keys; we iterate until exhausted. + -- For the dict size we set (8m → ~10w entries) the sidecar should pull + -- often enough that any single response stays bounded; if not, the + -- sidecar must use ?since= incrementally. + local keys = LAST:get_keys(0) + for _, k in ipairs(keys) do + local v = LAST:get(k) + if v and v > since then + entries[k] = v + end + end + + return reply(ngx.HTTP_OK, { + now = math.floor(ngx.now() * 1000), + since = since, + count = #keys, + entries = entries, + }) +end + +local function handle_healthz() + return reply(ngx.HTTP_OK, { + ok = true, + meta = META and META:free_space() or nil, + state = STATE and STATE:free_space() or nil, + last = LAST and LAST:free_space() or nil, + }) +end + +-- ── dispatch ─────────────────────────────────────────────────────────────── + +local function dispatch() + if not META or not STATE or not LAST then + return reply_error(ngx.HTTP_INTERNAL_SERVER_ERROR, + "auto-pause shared dicts not configured; check nginx.conf") + end + + check_token() + + local uri = ngx.var.uri or "" + local method = ngx.req.get_method() + + if uri == "/admin/healthz" and method == "GET" then + return handle_healthz() + elseif uri == "/admin/meta/upsert" and method == "POST" then + return handle_meta_upsert() + elseif uri == "/admin/meta/delete" and method == "POST" then + return handle_meta_delete() + elseif uri == "/admin/state" and method == "POST" then + return handle_state() + elseif uri == "/admin/last_active" and method == "GET" then + return handle_last_active() + end + + return reply_error(ngx.HTTP_NOT_FOUND, + string.format("no route for %s %s", method, uri)) +end + +dispatch() diff --git a/CubeProxy/lua/log_phase.lua b/CubeProxy/lua/log_phase.lua index 48ddc83ab..a73ae414e 100644 --- a/CubeProxy/lua/log_phase.lua +++ b/CubeProxy/lua/log_phase.lua @@ -1,3 +1,16 @@ +-- log_phase.lua — runs after the response is sent. +-- +-- 1. Records the access time for the access log (preserved). +-- 2. Stamps the per-sandbox "last active" timestamp into a worker-shared +-- lua dict. The CubeProxy-sidecar polls /admin/last_active to learn +-- which sandboxes are still receiving traffic, so this is the entire +-- feed for the auto-pause decision. Failure modes: +-- - Empty ngx.var.ins_id (request bypassed sandbox routing): skip. +-- - Dict full (no_memory): we drop a key so other sandboxes +-- continue to register; auto-pause decisions about the dropped +-- sandbox simply slip by one sweep cycle. +-- All paths are non-blocking; this phase MUST NOT delay log emission. + local function get_currtime() -- ngx.var.msec = 1663839717.105 local current_time_seconds_with_ms = ngx.var.msec @@ -11,3 +24,35 @@ local function get_currtime() end ngx.var.access_time = get_currtime() + +-- Record per-sandbox activity for the auto-pause sidecar. Sandboxes that +-- haven't opted into auto_pause are still cheap to record here — the +-- sidecar simply ignores their entries when computing pause decisions. +local ins_id = ngx.var.ins_id +if ins_id and ins_id ~= "" then + local active = ngx.shared.cube_sandbox_last_active + if active then + -- Coalesce sub-second writes: a single sandbox handling 1k QPS would + -- otherwise issue 1k dict writes per second per worker. The dict is + -- already keyed per-sandbox, so we only need a "good enough" most- + -- recent-second timestamp for the sidecar's idle calculation. Skip + -- if our last write was less than 1s ago. + local now_ms = math.floor(ngx.now() * 1000) + local prev = active:get(ins_id) + if (not prev) or (now_ms - prev) >= 1000 then + local ok, err, forcible = active:set(ins_id, now_ms) + if not ok then + -- safe_set isn't worth the extra LOC here; on failure we just + -- log at warn — the next request retries. + ngx.log(ngx.WARN, "LEVEL_WARN||", + string.format("cube_sandbox_last_active set %s failed: %s", ins_id, tostring(err))) + elseif forcible then + -- Dict was full and the LRU eviction took out a different + -- sandbox. We still recorded ours — surface visibility so + -- the operator can grow lua_shared_dict. + ngx.log(ngx.WARN, "LEVEL_WARN||", + "cube_sandbox_last_active dict full, an entry was evicted; consider raising lua_shared_dict size") + end + end + end +end diff --git a/CubeProxy/lua/path_rewrite_phase.lua b/CubeProxy/lua/path_rewrite_phase.lua index 5b7d2ffdd..241e966a5 100644 --- a/CubeProxy/lua/path_rewrite_phase.lua +++ b/CubeProxy/lua/path_rewrite_phase.lua @@ -7,6 +7,7 @@ -- routing in rewrite_phase.lua. local sb = require "sandbox_backend" +local state = require "sandbox_state" local uri = ngx.var.uri or "" local ins_id, container_port, rest = uri:match("^/sandbox/([%w_%-]+)/(%d+)(/?.*)$") @@ -34,6 +35,11 @@ ngx.var.container_port = container_port -- continue to run as configured. ngx.req.set_uri(rest, false) +-- Auto-pause gate: if the sandbox is currently paused, ask the sidecar to +-- resume it before we attempt backend resolution. No-op when the lifecycle +-- feature is disabled or the sandbox isn't tracked. +state.gate(ins_id) + local host_ip, host_port = sb.resolve_backend(ins_id, container_port) ngx.var.backend_ip = host_ip ngx.var.backend_port = host_port diff --git a/CubeProxy/lua/rewrite_phase.lua b/CubeProxy/lua/rewrite_phase.lua index dfbdb5261..8fe9cd16d 100644 --- a/CubeProxy/lua/rewrite_phase.lua +++ b/CubeProxy/lua/rewrite_phase.lua @@ -1,5 +1,6 @@ local utils = require "utils" local sb = require "sandbox_backend" +local state = require "sandbox_state" -- Parse Host: -. e.g. 49983-7c8fbcd45ffe450fb8f7fb223ad45507.cube.app -- Returns container_port, ins_id (sandbox / instance id), or nil, nil on failure. @@ -27,6 +28,14 @@ if not container_port or not ins_id then ngx.exit(400) end +-- log_phase reads ngx.var.ins_id to record activity. The host-based location +-- doesn't otherwise set it (only the path-based one does, for proxy_redirect +-- purposes), so populate it here to make activity tracking work uniformly. +ngx.var.ins_id = ins_id + +-- Auto-pause gate. See sandbox_state.lua for failure-mode semantics. +state.gate(ins_id) + local host_ip, host_port = sb.resolve_backend(ins_id, container_port) ngx.var.backend_ip = host_ip ngx.var.backend_port = host_port diff --git a/CubeProxy/lua/sandbox_state.lua b/CubeProxy/lua/sandbox_state.lua new file mode 100644 index 000000000..5b98a3845 --- /dev/null +++ b/CubeProxy/lua/sandbox_state.lua @@ -0,0 +1,120 @@ +-- sandbox_state.lua +-- +-- Auto-pause / auto-resume gate run from the rewrite phase. +-- +-- Reads the per-sandbox state from cube_sandbox_state (a worker-shared dict +-- maintained by CubeProxy-sidecar over the admin server) and either: +-- - lets the request through (state == "running" or unknown / not paused), +-- - returns 503 Retry-After when the sandbox is mid-pause (avoid the race +-- where we'd resume something the sidecar is actively pausing), +-- - or fires an internal sub-request to the sidecar's /internal/resume, +-- blocking the dataplane request until the sandbox is alive again. +-- +-- Failure modes are biased towards availability: +-- * Lifecycle feature disabled -> always pass. +-- * State dict missing or empty entry -> always pass (sidecar hasn't +-- gotten to us yet, which is the +-- same as "not opted in"). +-- * Sidecar address unset -> log + pass (paused sandbox will +-- return 5xx on its own; the +-- operator gets a clear error). +-- * Sidecar resume fails / times out -> 503 with Retry-After to let the +-- client back off. + +local _M = { _VERSION = "0.01" } + +-- gate runs from rewrite phase. On a successful resume it returns to the +-- caller; everything else either passes through (running) or terminates the +-- request with ngx.exit(). +-- +-- The gate is always-on but cheap when the sandbox has no state entry: a +-- single shared-dict lookup and an early return. The sidecar only populates +-- cube_sandbox_state for sandboxes that opted into auto_pause, so for the +-- common (non-opted-in) case this is essentially free. +function _M.gate(ins_id) + if not ins_id or ins_id == "" then + return + end + + local states = ngx.shared.cube_sandbox_state + if not states then + return + end + + local state = states:get(ins_id) + if not state or state == "running" then + return + end + + if state == "pausing" then + -- Sidecar is mid-flight; better to bounce the client than race it. + ngx.log(ngx.WARN, "LEVEL_WARN||", + string.format("request %s sandbox %s is pausing; returning 503", + ngx.var.http_x_cube_request_id or "-", ins_id)) + ngx.var.cube_retcode = "310503" + ngx.header["Retry-After"] = "2" + ngx.exit(503) + end + + if state ~= "paused" then + -- Unknown state — log once and let the request through; resume logic + -- only fires for the well-known "paused" value to keep behaviour + -- predictable as new states are added. + ngx.log(ngx.WARN, "LEVEL_WARN||", + string.format("sandbox %s has unknown state %q; passing through", + ins_id, tostring(state))) + return + end + + -- state == "paused": ask the sidecar to resume, then continue. + if not ngx.var.cube_sidecar_addr or ngx.var.cube_sidecar_addr == "" then + ngx.log(ngx.ERR, "LEVEL_ERROR||", + string.format("sandbox %s is paused but cube_sidecar_addr is unset; dataplane will fail", + ins_id)) + return + end + + -- ngx.location.capture issues a sub-request to /_sidecar_resume which + -- proxy_passes to the sidecar. We pass sandbox_id and request_id as args + -- so the sidecar can correlate logs and dedupe. + local args = "sandbox_id=" .. ngx.escape_uri(ins_id) + local rid = ngx.var.http_x_cube_request_id + if rid and rid ~= "" then + args = args .. "&request_id=" .. ngx.escape_uri(rid) + end + + -- Explicitly set body="" so ngx.location.capture does NOT inherit the + -- parent request's body. Without this, capture reuses the parent's + -- Content-Length header (e.g. "112" from a POST /execute), then our + -- /_sidecar_resume location's `proxy_pass_request_body off` strips + -- the actual body but leaves the inherited Content-Length intact. + -- The sidecar's Go http.Server then blocks reading the promised 112 + -- bytes that never arrive, parking the connection until the keepalive + -- timeout (~15s) — even though the request itself is processed + -- successfully on the sidecar side. + local res = ngx.location.capture("/_sidecar_resume", { + method = ngx.HTTP_POST, + args = args, + body = "", + }) + + if not res or res.status ~= ngx.HTTP_OK then + local status = (res and res.status) or "nil" + ngx.log(ngx.ERR, "LEVEL_ERROR||", + string.format("request %s sidecar resume for sandbox %s failed: status=%s body=%s", + rid or "-", ins_id, tostring(status), + (res and res.body) and string.sub(res.body, 1, 200) or "-")) + ngx.var.cube_retcode = "310503" + ngx.header["Retry-After"] = "5" + ngx.exit(503) + end + + -- Resume succeeded. Optimistically mark the sandbox running locally so a + -- burst of concurrent requests that arrived during the pause don't all + -- launch their own resume sub-requests. The sidecar will push the + -- authoritative value via /admin/state shortly after, which simply + -- overwrites this. + states:set(ins_id, "running") +end + +return _M diff --git a/CubeProxy/nginx.conf b/CubeProxy/nginx.conf index b166b84ba..f2d6128b2 100644 --- a/CubeProxy/nginx.conf +++ b/CubeProxy/nginx.conf @@ -81,6 +81,23 @@ http { } lua_shared_dict local_cache 100m; + + # Auto-pause / auto-resume coordination dicts. Owned by CubeProxy-sidecar + # over the admin server (127.0.0.1:8082) for the first two; the third is + # written purely from log_by_lua on every request and pulled by sidecar. + # + # cube_sandbox_meta sandboxID -> JSON metadata snapshot + # cube_sandbox_state sandboxID -> "running" | "pausing" | "paused" + # cube_sandbox_last_active sandboxID -> unix ms of most recent request + # + # Sizing rationale: ~80B per entry on the active dict supports >>10w live + # sandboxes per CubeProxy worker pool, with healthy headroom for keys and + # bookkeeping. Meta entries are JSON, ~512B nominal. + lua_shared_dict cube_sandbox_meta 16m; + lua_shared_dict cube_sandbox_state 4m; + lua_shared_dict cube_sandbox_last_active 8m; + + init_worker_by_lua_file lua/init_worker_phase.lua; server { @@ -92,6 +109,30 @@ http { set $ins_id ""; set $container_port ""; + # Auto-pause feature wiring. Defaults are safe — the sidecar binds + # to 127.0.0.1:8083 by default, so an empty $cube_sidecar_addr falls + # back to that. Operators override in conf/global/global.conf: + # $cube_sidecar_addr -> "127.0.0.1:8083" (or wherever the sidecar listens) + # $cube_admin_token -> shared secret for admin requests (optional) + set $cube_sidecar_addr "127.0.0.1:8083"; + set $cube_admin_token ""; + + # Internal sub-location used by sandbox_state.lua to ask the sidecar to + # resume a paused sandbox. Not reachable from outside this nginx + # (`internal` directive enforces that). + location = /_sidecar_resume { + internal; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Content-Length 0; + proxy_pass_request_body off; + proxy_pass_request_headers off; + proxy_connect_timeout 2s; + proxy_send_timeout 30s; + proxy_read_timeout 30s; + proxy_pass http://$cube_sidecar_addr/internal/resume?$args; + } + # Path-based routing: /sandbox/// # Lets clients reach a sandbox via a plain IP+port without wildcard DNS or TLS. location ^~ /sandbox/ { @@ -167,9 +208,26 @@ http { # Declared at the server level so proxy_redirect / proxy_cookie_path can reference them. set $ins_id ""; set $container_port ""; + # Auto-pause feature wiring; mirrors the 8081 server. See that block for docs. + set $cube_sidecar_addr "127.0.0.1:8083"; + set $cube_admin_token ""; ssl_certificate /usr/local/openresty/nginx/certs/cube.app+3.pem; ssl_certificate_key /usr/local/openresty/nginx/certs/cube.app+3-key.pem; + # Mirror of the 8081 server's internal sub-location. + location = /_sidecar_resume { + internal; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Content-Length 0; + proxy_pass_request_body off; + proxy_pass_request_headers off; + proxy_connect_timeout 2s; + proxy_send_timeout 30s; + proxy_read_timeout 30s; + proxy_pass http://$cube_sidecar_addr/internal/resume?$args; + } + # Path-based routing: /sandbox/// # Lets clients reach a sandbox via a plain IP+port without wildcard DNS or TLS. location ^~ /sandbox/ { @@ -237,4 +295,35 @@ http { log_by_lua_file lua/log_phase.lua; } } + + # ── CubeProxy admin server (auto-pause coordination) ────────────────────── + # + # Bound to loopback only; carries the shared-dict mutation surface used by + # CubeProxy-sidecar to push lifecycle metadata + state, and to pull each + # request's last-active timestamp. Dataplane traffic must NEVER reach + # this server. + # + # Routes: + # POST /admin/meta/upsert body=JSON {sandbox_id, ...} → cube_sandbox_meta + # POST /admin/meta/delete body=JSON {sandbox_id} → drops meta+state+last_active + # POST /admin/state body=JSON {sandbox_id, state} → cube_sandbox_state + # GET /admin/last_active → JSON snapshot of dict + # GET /admin/last_active?since= → only entries newer than `since` + # GET /admin/healthz + server { + listen 127.0.0.1:8082; + server_name _; + + # Operator may override in global.conf to require a shared token. When + # left empty the admin endpoints accept any loopback request. + set $cube_admin_token ""; + + # Admin payloads are tiny JSON; cap to keep a misbehaving client from + # eating the worker's memory. + client_max_body_size 1m; + + location /admin/ { + content_by_lua_file lua/admin_phase.lua; + } + } } diff --git a/CubeProxy/sidecar/cmd/sidecar/main.go b/CubeProxy/sidecar/cmd/sidecar/main.go new file mode 100644 index 000000000..e4c286fd4 --- /dev/null +++ b/CubeProxy/sidecar/cmd/sidecar/main.go @@ -0,0 +1,272 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +// cube-proxy-sidecar drives the auto-pause / auto-resume loop that sits +// between CubeMaster, CubeProxy, and Redis. +package main + +import ( + "context" + "errors" + "os/signal" + "syscall" + "time" + + "github.com/redis/go-redis/v9" + "go.uber.org/zap" + + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/config" + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/cubemasterclient" + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/httpapi" + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/lifecycle" + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/proxypush" + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/redisstream" + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/registry" + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/resumer" + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/sweeper" +) + +func main() { + if err := run(); err != nil && !errors.Is(err, context.Canceled) { + zap.L().Fatal("sidecar exit", zap.Error(err)) + } +} + +func run() error { + logger, err := zap.NewProduction() + if err != nil { + return err + } + defer func() { _ = logger.Sync() }() + zap.ReplaceGlobals(logger) + + cfg, err := config.Load() + if err != nil { + return err + } + if err := cfg.Validate(); err != nil { + return err + } + + logger.Info("sidecar starting", + zap.String("redis_addr", cfg.RedisAddr), + zap.Strings("cube_proxy_admin_urls", cfg.CubeProxyAdminURLs), + zap.String("cubemaster_url", cfg.CubeMasterURL), + zap.String("listen_addr", cfg.ListenAddr), + zap.String("consumer_group", cfg.ConsumerGroup), + zap.String("consumer_name", cfg.ConsumerName)) + + rdb := redis.NewClient(&redis.Options{ + Addr: cfg.RedisAddr, + Password: cfg.RedisPassword, + DB: cfg.RedisDB, + }) + defer func() { _ = rdb.Close() }() + + stream := redisstream.New(rdb, logger.Named("redis")) + pushClient := proxypush.New(cfg.CubeProxyAdminURLs, cfg.CubeAdminToken, cfg.HTTPTimeout, logger.Named("proxypush")) + masterClient := cubemasterclient.New(cfg.CubeMasterURL, cfg.HTTPTimeout) + reg := registry.New() + + rootCtx, cancel := signalContext() + defer cancel() + + // startupTs marks the boundary between "bootstrap entries (HGETALL)" + // and "stream entries (XREADGROUP)" for the sweeper's warmup logic. + startupTs := time.Now() + + // 1. Bootstrap registry from HSet, push it all to CubeProxy. After this + // the proxy has the full meta map even before any new events arrive. + if err := bootstrap(rootCtx, stream, pushClient, reg, startupTs, logger); err != nil { + return err + } + + // 2. Ensure the consumer group exists. + if err := stream.EnsureGroup(rootCtx, cfg.ConsumerGroup); err != nil { + return err + } + + resumeImpl := resumer.New(resumer.Options{ + Registry: reg, + Redis: stream, + CubeMaster: masterClient, + ProxyPush: pushClient, + StateLockTTL: cfg.StateLockTTL, + Log: logger.Named("resumer"), + }) + + sweep := sweeper.New(sweeper.Options{ + Registry: reg, + Redis: stream, + CubeMaster: masterClient, + ProxyPush: pushClient, + DefaultIdleTimeout: cfg.DefaultIdleTimeout, + BootstrapWarmup: cfg.BootstrapWarmup, + StateLockTTL: cfg.StateLockTTL, + Interval: cfg.IdleSweepInterval, + StartedAt: startupTs, + Log: logger.Named("sweeper"), + }) + + apiSrv := httpapi.New(cfg.ListenAddr, resumeImpl, reg, logger.Named("http")) + + // 3. Run all background loops concurrently. First error cancels the rest. + errs := make(chan error, 4) + go func() { errs <- consumeStream(rootCtx, stream, pushClient, reg, cfg, logger.Named("stream")) }() + go func() { errs <- pollLastActive(rootCtx, pushClient, reg, cfg.LastActivePoll, logger.Named("active")) }() + go func() { errs <- sweep.Run(rootCtx) }() + go func() { errs <- apiSrv.Run(rootCtx) }() + + // First loop to return wins; we cancel siblings via context and drain. + first := <-errs + cancel() + for i := 0; i < 3; i++ { + <-errs + } + return first +} + +func signalContext() (context.Context, context.CancelFunc) { + ctx, cancel := signal.NotifyContext(context.Background(), + syscall.SIGINT, syscall.SIGTERM) + return ctx, cancel +} + +// bootstrap reads the meta HSet, replaces the in-memory registry, and pushes +// every entry to CubeProxy so the proxy starts up with a complete view. +// +// Bootstrap entries get their FirstSeenAt backdated to a fixed startup +// timestamp so the sweeper's BootstrapWarmup gate can distinguish "loaded +// from HGETALL at process start" (FirstSeenAt == startupTs) from "arrived +// later via stream" (FirstSeenAt > startupTs). +func bootstrap(ctx context.Context, stream *redisstream.Client, push *proxypush.Client, + reg *registry.Registry, startupTs time.Time, log *zap.Logger) error { + + metas, err := stream.Bootstrap(ctx) + if err != nil { + return err + } + reg.Reset() + for _, m := range metas { + reg.Upsert(m) + // Pin FirstSeenAt to the recorded startup time. Without this, + // every bootstrap entry would have FirstSeenAt = time.Now() at + // the moment Upsert ran, which is a moving target and trips the + // sweeper's "is this a fresh stream event?" check (it compares + // FirstSeenAt against startedAt with .After() semantics). + reg.SetFirstSeenAt(m.SandboxID, startupTs) + if err := push.UpsertMeta(ctx, m); err != nil { + // Continue: the proxy will receive entries via the stream consumer + // loop too, so a partial bootstrap isn't fatal. The next periodic + // reconcile (if/when added) will close the gap. + log.Warn("bootstrap push failed", + zap.String("sandbox_id", m.SandboxID), zap.Error(err)) + } + } + log.Info("bootstrap complete", zap.Int("entries", len(metas))) + return nil +} + +// consumeStream is the increment-side of the lifecycle channel. It maintains +// the registry + pushes deltas to CubeProxy as create / delete events arrive. +func consumeStream(ctx context.Context, stream *redisstream.Client, push *proxypush.Client, + reg *registry.Registry, cfg *config.Config, log *zap.Logger) error { + + for { + if ctx.Err() != nil { + return ctx.Err() + } + events, err := stream.ReadGroup(ctx, cfg.ConsumerGroup, cfg.ConsumerName, + cfg.StreamReadBlock, 100) + if err != nil { + log.Warn("xreadgroup failed; backing off", zap.Error(err)) + select { + case <-ctx.Done(): + return ctx.Err() + case <-time.After(time.Second): + } + continue + } + for _, ev := range events { + handleEvent(ctx, ev, push, reg, log) + if err := stream.Ack(ctx, cfg.ConsumerGroup, ev.StreamID); err != nil { + log.Warn("ack failed", + zap.String("id", ev.StreamID), zap.Error(err)) + } + } + } +} + +func handleEvent(ctx context.Context, ev redisstream.Event, push *proxypush.Client, + reg *registry.Registry, log *zap.Logger) { + + switch ev.Op { + case lifecycle.OpCreate: + if ev.Meta == nil { + log.Warn("create event missing payload", + zap.String("sandbox_id", ev.SandboxID)) + return + } + reg.Upsert(*ev.Meta) + // Log every create at info level: this is the heartbeat that + // proves CubeMaster -> Redis -> sidecar is wired correctly. The + // volume is bounded by sandbox creation rate (≪ QPS) so this is + // not a noise concern. + log.Info("create event applied", + zap.String("sandbox_id", ev.SandboxID), + zap.Bool("auto_pause", ev.Meta.AutoPause), + zap.Bool("auto_resume", ev.Meta.AutoResume), + zap.Int("timeout_seconds", ev.Meta.TimeoutSeconds), + zap.Int("registry_size", reg.Len())) + if err := push.UpsertMeta(ctx, *ev.Meta); err != nil { + log.Warn("create event push failed", + zap.String("sandbox_id", ev.SandboxID), zap.Error(err)) + } + case lifecycle.OpDelete: + reg.Delete(ev.SandboxID) + log.Info("delete event applied", + zap.String("sandbox_id", ev.SandboxID), + zap.Int("registry_size", reg.Len())) + if err := push.DeleteMeta(ctx, ev.SandboxID); err != nil { + log.Warn("delete event push failed", + zap.String("sandbox_id", ev.SandboxID), zap.Error(err)) + } + default: + log.Warn("unknown event op", + zap.String("op", ev.Op), + zap.String("sandbox_id", ev.SandboxID)) + } +} + +// pollLastActive pulls /admin/last_active from every CubeProxy and merges +// the timestamps into the registry. The sweeper consumes the merged view. +func pollLastActive(ctx context.Context, push *proxypush.Client, reg *registry.Registry, + interval time.Duration, log *zap.Logger) error { + + t := time.NewTicker(interval) + defer t.Stop() + + var since int64 + for { + select { + case <-ctx.Done(): + return ctx.Err() + case <-t.C: + } + entries, minNow, err := push.PullLastActive(ctx, since) + if err != nil { + log.Warn("pull last_active failed", zap.Error(err)) + continue + } + for sid, ts := range entries { + reg.MergeLastActive(sid, ts) + } + // Bump the watermark so the next pull is incremental. Using the + // minimum `now` across responses guarantees no entry can fall into + // the (since, next_since] gap if one CubeProxy clock is behind. + if minNow > since { + since = minNow + } + } +} diff --git a/CubeProxy/sidecar/go.mod b/CubeProxy/sidecar/go.mod new file mode 100644 index 000000000..bd41582c6 --- /dev/null +++ b/CubeProxy/sidecar/go.mod @@ -0,0 +1,15 @@ +module github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar + +go 1.25.7 + +require ( + github.com/google/uuid v1.6.0 + github.com/redis/go-redis/v9 v9.20.0 + go.uber.org/zap v1.28.0 +) + +require ( + github.com/cespare/xxhash/v2 v2.3.0 // indirect + go.uber.org/atomic v1.11.0 // indirect + go.uber.org/multierr v1.10.0 // indirect +) diff --git a/CubeProxy/sidecar/go.sum b/CubeProxy/sidecar/go.sum new file mode 100644 index 000000000..1d150d28c --- /dev/null +++ b/CubeProxy/sidecar/go.sum @@ -0,0 +1,34 @@ +github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs= +github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c= +github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA= +github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/klauspost/cpuid/v2 v2.2.10 h1:tBs3QSyvjDyFTq3uoc/9xFpCuOsJQFNPiAhYdw2skhE= +github.com/klauspost/cpuid/v2 v2.2.10/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/redis/go-redis/v9 v9.20.0 h1:WnQYxLkgO2xiXTCJY0ldIiI8dNqCDlQAG+AtaH7a2a0= +github.com/redis/go-redis/v9 v9.20.0/go.mod h1:v/M13XI1PVCDcm01VtPFOADfZtHf8YW3baQf57KlIkA= +github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= +github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= +github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs= +github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s= +go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE= +go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0= +go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= +go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= +go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ= +go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= +go.uber.org/zap v1.28.0 h1:IZzaP1Fv73/T/pBMLk4VutPl36uNC+OSUh3JLG3FIjo= +go.uber.org/zap v1.28.0/go.mod h1:rDLpOi171uODNm/mxFcuYWxDsqWSAVkFdX4XojSKg/Q= +go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc= +go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg= +golang.org/x/sys v0.30.0 h1:QjkSwP/36a20jFYWkSue1YwXzLmsV5Gfq7Eiy72C1uc= +golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/CubeProxy/sidecar/internal/config/config.go b/CubeProxy/sidecar/internal/config/config.go new file mode 100644 index 000000000..bafd8472a --- /dev/null +++ b/CubeProxy/sidecar/internal/config/config.go @@ -0,0 +1,216 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +package config + +import ( + "errors" + "fmt" + "os" + "strconv" + "strings" + "time" +) + +// Config holds the sidecar's runtime parameters. The shape is intentionally +// flat — every field maps to a single env var — so the operator can wire it +// up via systemd EnvironmentFile= without a YAML parser dependency. +type Config struct { + // Redis (the same instance CubeMaster writes to). + RedisAddr string + RedisPassword string + RedisDB int + + // CubeProxy admin endpoints to push to and pull from. Multiple endpoints + // are supported even though the recommended deployment is one sidecar + // per CubeProxy: future operators may consolidate. + CubeProxyAdminURLs []string + CubeAdminToken string // optional shared secret; sent as X-Cube-Admin-Token + + // CubeMaster internal HTTP for pause/resume. Sidecar calls + // POST /cube/sandbox/update with action=pause|resume. + CubeMasterURL string + + // HTTP listener for /internal/resume (called by CubeProxy via the + // internal sub-location). Bind to loopback in same-host deployments. + ListenAddr string + + // Defaults applied when a sandbox's lifecycle meta omits TimeoutSeconds. + DefaultIdleTimeout time.Duration + + // Loop intervals. + StreamReadBlock time.Duration // XREADGROUP BLOCK arg + LastActivePoll time.Duration // GET /admin/last_active cadence + IdleSweepInterval time.Duration // sweeper cadence + // BootstrapWarmup: after sidecar restart, wait this long before pausing + // any sandbox that was loaded via HGETALL bootstrap. Lets the + // last_active poller backfill activity timestamps first. New sandboxes + // that arrive AFTER startup are not affected by this delay. + BootstrapWarmup time.Duration + + // Pause/resume locks (SETNX TTL). Long enough to outlive a slow + // CubeMaster RPC, short enough that a crashed sidecar releases the lock. + StateLockTTL time.Duration + + // Consumer group identity. Group name is fixed; consumer name defaults + // to the host's name so multiple sidecars in a cluster get independent + // pending-entries lists. + ConsumerGroup string + ConsumerName string // empty → derived from os.Hostname() + + // HTTP client timeouts (for outbound calls to CubeMaster + CubeProxy). + HTTPTimeout time.Duration +} + +// Default returns a config populated with safe defaults; callers then override +// via Load(env) or direct field writes (tests). +func Default() *Config { + return &Config{ + RedisAddr: "127.0.0.1:6379", + RedisDB: 0, + CubeProxyAdminURLs: []string{"http://127.0.0.1:8082"}, + // CubeMaster's HTTP listener defaults to :8089 (config key + // `common.http_port`). Override via CUBE_SIDECAR_CUBEMASTER_URL. + CubeMasterURL: "http://127.0.0.1:8089", + ListenAddr: "127.0.0.1:8083", + DefaultIdleTimeout: 5 * time.Minute, + StreamReadBlock: 5 * time.Second, + LastActivePoll: 5 * time.Second, + IdleSweepInterval: 5 * time.Second, + BootstrapWarmup: 30 * time.Second, + StateLockTTL: 60 * time.Second, + ConsumerGroup: "cube-proxy-sidecar", + HTTPTimeout: 10 * time.Second, + } +} + +// Load builds a Config from environment variables, falling back to Default() +// for unset fields. Returns an error only when an env var is set to a value +// that cannot be parsed — missing values are not an error. +func Load() (*Config, error) { + c := Default() + + var errs []string + addErr := func(name string, err error) { + errs = append(errs, fmt.Sprintf("%s: %v", name, err)) + } + + if v := os.Getenv("CUBE_SIDECAR_REDIS_ADDR"); v != "" { + c.RedisAddr = v + } + if v := os.Getenv("CUBE_SIDECAR_REDIS_PASSWORD"); v != "" { + c.RedisPassword = v + } + if v := os.Getenv("CUBE_SIDECAR_REDIS_DB"); v != "" { + n, err := strconv.Atoi(v) + if err != nil { + addErr("CUBE_SIDECAR_REDIS_DB", err) + } else { + c.RedisDB = n + } + } + if v := os.Getenv("CUBE_SIDECAR_PROXY_ADMIN_URLS"); v != "" { + c.CubeProxyAdminURLs = splitAndTrim(v) + } + if v := os.Getenv("CUBE_SIDECAR_ADMIN_TOKEN"); v != "" { + c.CubeAdminToken = v + } + if v := os.Getenv("CUBE_SIDECAR_CUBEMASTER_URL"); v != "" { + c.CubeMasterURL = v + } + if v := os.Getenv("CUBE_SIDECAR_LISTEN_ADDR"); v != "" { + c.ListenAddr = v + } + if v := os.Getenv("CUBE_SIDECAR_DEFAULT_IDLE_TIMEOUT"); v != "" { + if d, err := time.ParseDuration(v); err != nil { + addErr("CUBE_SIDECAR_DEFAULT_IDLE_TIMEOUT", err) + } else { + c.DefaultIdleTimeout = d + } + } + if v := os.Getenv("CUBE_SIDECAR_LAST_ACTIVE_POLL"); v != "" { + if d, err := time.ParseDuration(v); err != nil { + addErr("CUBE_SIDECAR_LAST_ACTIVE_POLL", err) + } else { + c.LastActivePoll = d + } + } + if v := os.Getenv("CUBE_SIDECAR_IDLE_SWEEP_INTERVAL"); v != "" { + if d, err := time.ParseDuration(v); err != nil { + addErr("CUBE_SIDECAR_IDLE_SWEEP_INTERVAL", err) + } else { + c.IdleSweepInterval = d + } + } + if v := os.Getenv("CUBE_SIDECAR_BOOTSTRAP_WARMUP"); v != "" { + if d, err := time.ParseDuration(v); err != nil { + addErr("CUBE_SIDECAR_BOOTSTRAP_WARMUP", err) + } else { + c.BootstrapWarmup = d + } + } + if v := os.Getenv("CUBE_SIDECAR_STATE_LOCK_TTL"); v != "" { + if d, err := time.ParseDuration(v); err != nil { + addErr("CUBE_SIDECAR_STATE_LOCK_TTL", err) + } else { + c.StateLockTTL = d + } + } + if v := os.Getenv("CUBE_SIDECAR_CONSUMER_NAME"); v != "" { + c.ConsumerName = v + } + + if c.ConsumerName == "" { + host, err := os.Hostname() + if err != nil { + addErr("hostname", err) + } else { + c.ConsumerName = host + } + } + + if len(errs) > 0 { + return nil, errors.New("config load: " + strings.Join(errs, "; ")) + } + return c, nil +} + +// Validate returns an error if the config has any field combination that the +// sidecar can't proceed with. +func (c *Config) Validate() error { + if c.RedisAddr == "" { + return errors.New("redis addr is empty") + } + if len(c.CubeProxyAdminURLs) == 0 { + return errors.New("cube proxy admin urls is empty") + } + if c.CubeMasterURL == "" { + return errors.New("cube master url is empty") + } + if c.ListenAddr == "" { + return errors.New("listen addr is empty") + } + if c.ConsumerName == "" { + return errors.New("consumer name is empty") + } + if c.IdleSweepInterval <= 0 { + return errors.New("idle sweep interval must be > 0") + } + if c.LastActivePoll <= 0 { + return errors.New("last active poll must be > 0") + } + return nil +} + +func splitAndTrim(s string) []string { + parts := strings.Split(s, ",") + out := parts[:0] + for _, p := range parts { + p = strings.TrimSpace(p) + if p != "" { + out = append(out, p) + } + } + return out +} diff --git a/CubeProxy/sidecar/internal/cubemasterclient/client.go b/CubeProxy/sidecar/internal/cubemasterclient/client.go new file mode 100644 index 000000000..aa4a1aebc --- /dev/null +++ b/CubeProxy/sidecar/internal/cubemasterclient/client.go @@ -0,0 +1,156 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +// Package cubemasterclient is the sidecar's tiny HTTP client for CubeMaster. +// It calls the same /cube/sandbox/update endpoint that CubeAPI uses; we go +// directly here to avoid the sidecar → CubeAPI → CubeMaster round-trip. +package cubemasterclient + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "time" + + "github.com/google/uuid" +) + +// CubeMaster ret_code constants the sidecar reasons about. The full set +// lives in CubeMaster/api/services/errorcode/v1/errorcode.proto; we mirror +// only the codes we need to react to here, keeping the sidecar free of a +// build-time dependency on the master proto. +const ( + // RetCodeSuccess is CubeMaster's "operation succeeded" code. + RetCodeSuccess = 200 + + // RetCodeInvalidParamFormat is reused by CubeMaster's pause/resume path + // for "sandbox does not exist" — the meta lookup misses, surfaced as + // ret_msg "key not found". Treat as a hard NotFound for the caller. + RetCodeInvalidParamFormat = 130483 + + // RetCodeTaskStateInvalid is returned when the requested transition is + // a no-op (e.g. pause on an already-paused sandbox, or resume on a + // running one). Idempotent from the caller's POV. + RetCodeTaskStateInvalid = 130490 +) + +// APIError is returned by the client whenever CubeMaster replies with a +// non-success ret_code. Callers can errors.As-extract it to react to +// specific conditions (e.g. "sandbox already paused" → treat as success). +type APIError struct { + RetCode int + RetMsg string +} + +func (e *APIError) Error() string { + return fmt.Sprintf("cubemaster returned ret_code=%d msg=%q", e.RetCode, e.RetMsg) +} + +// IsNotFound reports whether the master replied with the "sandbox does not +// exist" ret_code. Sidecar uses this to evict stale registry entries instead +// of retrying a doomed pause/resume forever. +func (e *APIError) IsNotFound() bool { + return e != nil && e.RetCode == RetCodeInvalidParamFormat +} + +// IsAlreadyInState reports whether the master refused the transition because +// the sandbox is already in the desired state. From the sidecar's POV this +// is success: the sandbox is already where we wanted it, no retry needed. +func (e *APIError) IsAlreadyInState() bool { + return e != nil && e.RetCode == RetCodeTaskStateInvalid +} + +// Client is a thin wrapper around http.Client + base URL. Concurrency-safe. +type Client struct { + baseURL string + httpc *http.Client +} + +func New(baseURL string, timeout time.Duration) *Client { + return &Client{ + baseURL: baseURL, + httpc: &http.Client{Timeout: timeout}, + } +} + +// updateRequest mirrors CubeMaster pkg/service/sandbox/types.UpdateRequest. +type updateRequest struct { + RequestID string `json:"requestID"` + SandboxID string `json:"sandbox_id"` + InstanceType string `json:"instance_type"` + Action string `json:"action"` // "pause" | "resume" +} + +type updateResponse struct { + Ret struct { + RetCode int `json:"ret_code"` + RetMsg string `json:"ret_msg"` + } `json:"ret"` +} + +// Pause asks CubeMaster to pause the given sandbox. instanceType is required +// by the master; for the cubebox runtime that's "cubebox". +// +// Returns nil on success or when the sandbox is already paused. Returns an +// *APIError for any non-success ret_code; use APIError.IsNotFound / +// IsAlreadyInState to classify. +func (c *Client) Pause(ctx context.Context, sandboxID, instanceType string) error { + return c.update(ctx, sandboxID, instanceType, "pause") +} + +// Resume asks CubeMaster to resume the given sandbox. Same error semantics +// as Pause. +func (c *Client) Resume(ctx context.Context, sandboxID, instanceType string) error { + return c.update(ctx, sandboxID, instanceType, "resume") +} + +func (c *Client) update(ctx context.Context, sandboxID, instanceType, action string) error { + if sandboxID == "" || instanceType == "" { + return errors.New("sandbox_id and instance_type are required") + } + + body, err := json.Marshal(updateRequest{ + RequestID: uuid.NewString(), + SandboxID: sandboxID, + InstanceType: instanceType, + Action: action, + }) + if err != nil { + return fmt.Errorf("marshal: %w", err) + } + + req, err := http.NewRequestWithContext(ctx, http.MethodPost, + c.baseURL+"/cube/sandbox/update", bytes.NewReader(body)) + if err != nil { + return err + } + req.Header.Set("Content-Type", "application/json") + + resp, err := c.httpc.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + + raw, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) + if err != nil { + return fmt.Errorf("read body: %w", err) + } + if resp.StatusCode/100 != 2 { + return fmt.Errorf("http %d: %s", resp.StatusCode, raw) + } + + var ur updateResponse + if err := json.Unmarshal(raw, &ur); err != nil { + return fmt.Errorf("decode response: %w (body=%q)", err, raw) + } + if ur.Ret.RetCode == RetCodeSuccess { + return nil + } + return &APIError{RetCode: ur.Ret.RetCode, RetMsg: ur.Ret.RetMsg} +} diff --git a/CubeProxy/sidecar/internal/httpapi/server.go b/CubeProxy/sidecar/internal/httpapi/server.go new file mode 100644 index 000000000..084b71e7c --- /dev/null +++ b/CubeProxy/sidecar/internal/httpapi/server.go @@ -0,0 +1,136 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +// Package httpapi exposes the sidecar's internal HTTP surface used by the +// CubeProxy /_sidecar_resume internal location. Routes: +// +// POST /internal/resume?sandbox_id=...&request_id=... +// GET /healthz +// GET /readyz +package httpapi + +import ( + "context" + "encoding/json" + "errors" + "net" + "net/http" + "time" + + "go.uber.org/zap" + + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/registry" + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/resumer" +) + +// Server wires resume/healthz handlers and runs a *http.Server. +type Server struct { + addr string + resumer *resumer.Resumer + registry *registry.Registry + log *zap.Logger + srv *http.Server +} + +func New(addr string, r *resumer.Resumer, reg *registry.Registry, log *zap.Logger) *Server { + return &Server{addr: addr, resumer: r, registry: reg, log: log} +} + +// Run blocks until ctx is cancelled or ListenAndServe returns. +func (s *Server) Run(ctx context.Context) error { + mux := http.NewServeMux() + mux.HandleFunc("/internal/resume", s.handleResume) + mux.HandleFunc("/healthz", s.handleHealthz) + mux.HandleFunc("/readyz", s.handleReadyz) + + s.srv = &http.Server{ + Addr: s.addr, + Handler: mux, + ReadHeaderTimeout: 5 * time.Second, + // Resume RPCs to CubeMaster can be slow; make sure the sub-request + // from CubeProxy isn't cut off by us. + WriteTimeout: 35 * time.Second, + } + + ln, err := net.Listen("tcp", s.addr) + if err != nil { + return err + } + + errCh := make(chan error, 1) + go func() { + s.log.Info("sidecar http server listening", zap.String("addr", s.addr)) + errCh <- s.srv.Serve(ln) + }() + + select { + case <-ctx.Done(): + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + _ = s.srv.Shutdown(shutdownCtx) + return ctx.Err() + case err := <-errCh: + if errors.Is(err, http.ErrServerClosed) { + return nil + } + return err + } +} + +func (s *Server) handleResume(w http.ResponseWriter, req *http.Request) { + if req.Method != http.MethodPost { + http.Error(w, "method not allowed", http.StatusMethodNotAllowed) + return + } + sid := req.URL.Query().Get("sandbox_id") + if sid == "" { + http.Error(w, "sandbox_id query param is required", http.StatusBadRequest) + return + } + rid := req.URL.Query().Get("request_id") + + // CubeProxy's proxy_read_timeout is 30s. Cap our own work at 25s so we + // always have time to flush a 5xx response before nginx gives up on us + // — otherwise nginx returns "504 upstream timed out" with no body and + // we lose the ability to report what actually went wrong. + ctx, cancel := context.WithTimeout(req.Context(), 25*time.Second) + defer cancel() + + start := time.Now() + s.log.Info("resume request received", + zap.String("sandbox_id", sid), + zap.String("request_id", rid)) + + if err := s.resumer.Resume(ctx, sid); err != nil { + s.log.Warn("resume failed", + zap.String("sandbox_id", sid), + zap.String("request_id", rid), + zap.Duration("elapsed", time.Since(start)), + zap.Error(err)) + http.Error(w, err.Error(), http.StatusServiceUnavailable) + return + } + s.log.Info("resume request completed", + zap.String("sandbox_id", sid), + zap.String("request_id", rid), + zap.Duration("elapsed", time.Since(start))) + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "ok": true, + "sandbox_id": sid, + }) +} + +func (s *Server) handleHealthz(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte("ok")) +} + +func (s *Server) handleReadyz(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "ok": true, + "registry_len": s.registry.Len(), + }) +} diff --git a/CubeProxy/sidecar/internal/httpapi/server_test.go b/CubeProxy/sidecar/internal/httpapi/server_test.go new file mode 100644 index 000000000..038a729ea --- /dev/null +++ b/CubeProxy/sidecar/internal/httpapi/server_test.go @@ -0,0 +1,192 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +package httpapi + +import ( + "context" + "errors" + "io" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" + "time" + + "go.uber.org/zap" + + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/lifecycle" + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/registry" + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/resumer" +) + +// ------ resumer test doubles (re-used pattern from resumer_test.go) ------- + +type fakeStore struct { + states map[string]string +} + +func newFakeStore() *fakeStore { return &fakeStore{states: map[string]string{}} } +func (f *fakeStore) AcquireState(_ context.Context, sid, state string, _ time.Duration) (bool, error) { + if _, ok := f.states[sid]; ok { + return false, nil + } + f.states[sid] = state + return true, nil +} +func (f *fakeStore) SetState(_ context.Context, sid, state string, _ time.Duration) error { + f.states[sid] = state + return nil +} +func (f *fakeStore) ClearState(_ context.Context, sid string) error { + delete(f.states, sid) + return nil +} +func (f *fakeStore) GetState(_ context.Context, sid string) (string, bool, error) { + v, ok := f.states[sid] + return v, ok, nil +} + +type fakeMaster struct { + calls int32 + failNext bool +} + +func (f *fakeMaster) Resume(_ context.Context, _, _ string) error { + atomic.AddInt32(&f.calls, 1) + if f.failNext { + return errors.New("master failed") + } + return nil +} + +type fakePush struct{} + +func (fakePush) SetState(_ context.Context, _, _ string) error { return nil } +func (fakePush) DeleteMeta(_ context.Context, _ string) error { return nil } + +// ------ tests ------------------------------------------------------------- + +// helper wires up the same handlers Run() registers, so we can use httptest +// without binding a real port. +func newTestHandler(reg *registry.Registry, store *fakeStore, master *fakeMaster) http.Handler { + r := resumer.New(resumer.Options{ + Registry: reg, + Redis: store, + CubeMaster: master, + ProxyPush: fakePush{}, + StateLockTTL: time.Minute, + Log: zap.NewNop(), + }) + s := New(":0", r, reg, zap.NewNop()) + mux := http.NewServeMux() + mux.HandleFunc("/internal/resume", s.handleResume) + mux.HandleFunc("/healthz", s.handleHealthz) + mux.HandleFunc("/readyz", s.handleReadyz) + return mux +} + +func TestResumeEndpoint_HappyPath(t *testing.T) { + reg := registry.New() + reg.Upsert(lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx", InstanceType: "cubebox", AutoResume: true, + }) + master := &fakeMaster{} + srv := httptest.NewServer(newTestHandler(reg, newFakeStore(), master)) + defer srv.Close() + + resp, err := http.Post(srv.URL+"/internal/resume?sandbox_id=sbx", "", nil) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + body, _ := io.ReadAll(resp.Body) + t.Fatalf("expected 200, got %d: %s", resp.StatusCode, body) + } + if got := atomic.LoadInt32(&master.calls); got != 1 { + t.Fatalf("expected 1 master.Resume call, got %d", got) + } +} + +func TestResumeEndpoint_RejectsGet(t *testing.T) { + srv := httptest.NewServer(newTestHandler(registry.New(), newFakeStore(), &fakeMaster{})) + defer srv.Close() + + resp, err := http.Get(srv.URL + "/internal/resume?sandbox_id=sbx") + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusMethodNotAllowed { + t.Fatalf("expected 405, got %d", resp.StatusCode) + } +} + +func TestResumeEndpoint_BadRequestWithoutSandboxID(t *testing.T) { + srv := httptest.NewServer(newTestHandler(registry.New(), newFakeStore(), &fakeMaster{})) + defer srv.Close() + + resp, err := http.Post(srv.URL+"/internal/resume", "", nil) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusBadRequest { + body, _ := io.ReadAll(resp.Body) + t.Fatalf("expected 400, got %d: %s", resp.StatusCode, body) + } +} + +func TestResumeEndpoint_503OnResumerError(t *testing.T) { + reg := registry.New() + reg.Upsert(lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx", InstanceType: "cubebox", AutoResume: true, + }) + master := &fakeMaster{failNext: true} + srv := httptest.NewServer(newTestHandler(reg, newFakeStore(), master)) + defer srv.Close() + + resp, err := http.Post(srv.URL+"/internal/resume?sandbox_id=sbx", "", nil) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusServiceUnavailable { + body, _ := io.ReadAll(resp.Body) + t.Fatalf("expected 503, got %d: %s", resp.StatusCode, body) + } +} + +func TestHealthzAndReadyz(t *testing.T) { + reg := registry.New() + reg.Upsert(lifecycle.SandboxLifecycleMeta{SandboxID: "sbx"}) + + srv := httptest.NewServer(newTestHandler(reg, newFakeStore(), &fakeMaster{})) + defer srv.Close() + + resp, err := http.Get(srv.URL + "/healthz") + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + body, _ := io.ReadAll(resp.Body) + if resp.StatusCode != http.StatusOK || string(body) != "ok" { + t.Fatalf("/healthz wrong: status=%d body=%q", resp.StatusCode, body) + } + + resp2, err := http.Get(srv.URL + "/readyz") + if err != nil { + t.Fatal(err) + } + defer resp2.Body.Close() + body2, _ := io.ReadAll(resp2.Body) + if resp2.StatusCode != http.StatusOK { + t.Fatalf("/readyz wrong status: %d", resp2.StatusCode) + } + if !strings.Contains(string(body2), `"registry_len":1`) { + t.Fatalf("/readyz body should mention registry_len=1: %s", body2) + } +} diff --git a/CubeProxy/sidecar/internal/lifecycle/parity_test.go b/CubeProxy/sidecar/internal/lifecycle/parity_test.go new file mode 100644 index 000000000..341ac18cb --- /dev/null +++ b/CubeProxy/sidecar/internal/lifecycle/parity_test.go @@ -0,0 +1,40 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +package lifecycle + +import "testing" + +// TestSchemaConstants is a fence: if anyone changes one of these values +// without updating CubeMaster/pkg/lifecycle/schema.go in lockstep, the test +// fails and the diff makes the divergence obvious. +// +// Source of truth lives in CubeMaster; we hardcode the same values here so +// the sidecar can be built without taking a build-time dependency on the +// CubeMaster module. Whenever you touch the constants, update both files in +// the same commit. +func TestSchemaConstants(t *testing.T) { + cases := []struct { + name, got, want string + }{ + {"MetaKey", MetaKey, "cube:sandbox:meta"}, + {"EventStreamKey", EventStreamKey, "cube:sandbox:events"}, + {"StateKeyPrefix", StateKeyPrefix, "cube:sandbox:state:"}, + {"OpCreate", OpCreate, "create"}, + {"OpDelete", OpDelete, "delete"}, + {"FieldOp", FieldOp, "op"}, + {"FieldSandboxID", FieldSandboxID, "sandbox_id"}, + {"FieldPayload", FieldPayload, "payload"}, + {"FieldTimestamp", FieldTimestamp, "ts"}, + } + for _, c := range cases { + if c.got != c.want { + t.Errorf("%s: %q != %q (CubeMaster schema drifted?)", + c.name, c.got, c.want) + } + } + if EventStreamMaxLen != 100000 { + t.Errorf("EventStreamMaxLen = %d; CubeMaster expects 100000", EventStreamMaxLen) + } +} diff --git a/CubeProxy/sidecar/internal/lifecycle/schema.go b/CubeProxy/sidecar/internal/lifecycle/schema.go new file mode 100644 index 000000000..5c7fa6454 --- /dev/null +++ b/CubeProxy/sidecar/internal/lifecycle/schema.go @@ -0,0 +1,62 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +// Package lifecycle is the sidecar-local mirror of +// CubeMaster/pkg/lifecycle. The two MUST stay byte-compatible: CubeMaster is +// the single writer, the sidecar is a pure consumer. +// +// We do not import the CubeMaster module directly because it would drag in +// MySQL, gRPC, scheduler, and a host of other heavy dependencies that have no +// place in the sidecar. The schema is small enough that copying it (with a +// pointer to the canonical definition) is cheaper than the cross-module wire. +// +// Source of truth: +// /data/cube-opensource-dev/CubeSandbox/CubeMaster/pkg/lifecycle/schema.go +// +// Whenever you change one side, change the other in the same commit. +package lifecycle + +const ( + // MetaKey is the HSet snapshot of every live sandbox. + MetaKey = "cube:sandbox:meta" + + // EventStreamKey is the append-only stream of create/delete events. + EventStreamKey = "cube:sandbox:events" + + // EventStreamMaxLen caps the stream so an offline sidecar cannot drive + // unbounded Redis growth. + EventStreamMaxLen = 100000 + + // StateKeyPrefix + sandboxID stores "running" | "pausing" | "paused" | + // "resuming". The sidecar uses these as cross-process locks (SETNX with + // TTL) to coordinate concurrent pause/resume. + StateKeyPrefix = "cube:sandbox:state:" +) + +// Op codes carried in stream entries. +const ( + OpCreate = "create" + OpDelete = "delete" +) + +// Stream entry field names. +const ( + FieldOp = "op" + FieldSandboxID = "sandbox_id" + FieldPayload = "payload" + FieldTimestamp = "ts" +) + +// SandboxLifecycleMeta mirrors CubeMaster/pkg/lifecycle.SandboxLifecycleMeta. +type SandboxLifecycleMeta struct { + SandboxID string `json:"sandbox_id"` + TemplateID string `json:"template_id,omitempty"` + HostID string `json:"host_id,omitempty"` + HostIP string `json:"host_ip,omitempty"` + InstanceType string `json:"instance_type,omitempty"` + TimeoutSeconds int `json:"timeout_seconds,omitempty"` + AutoPause bool `json:"auto_pause,omitempty"` + AutoResume bool `json:"auto_resume,omitempty"` + CreatedAt int64 `json:"created_at,omitempty"` +} diff --git a/CubeProxy/sidecar/internal/proxypush/client.go b/CubeProxy/sidecar/internal/proxypush/client.go new file mode 100644 index 000000000..3775734f4 --- /dev/null +++ b/CubeProxy/sidecar/internal/proxypush/client.go @@ -0,0 +1,205 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +// Package proxypush is the HTTP client the sidecar uses to push lifecycle +// metadata + state to one or more CubeProxy admin endpoints, and to pull the +// per-request last_active timestamps back. The protocol is documented in +// CubeProxy/lua/admin_phase.lua — this file is the canonical Go peer. +package proxypush + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "strconv" + "sync" + "time" + + "go.uber.org/zap" + + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/lifecycle" +) + +// Client fans pushes out to every configured CubeProxy admin URL and merges +// pulls (last_active) by max(timestamp). +type Client struct { + endpoints []string + token string + httpc *http.Client + log *zap.Logger +} + +func New(endpoints []string, token string, timeout time.Duration, log *zap.Logger) *Client { + return &Client{ + endpoints: endpoints, + token: token, + httpc: &http.Client{Timeout: timeout}, + log: log, + } +} + +// LastActiveResponse mirrors the JSON body returned by GET /admin/last_active. +type LastActiveResponse struct { + Now int64 `json:"now"` + Since int64 `json:"since"` + Count int `json:"count"` + Entries map[string]int64 `json:"entries"` +} + +// UpsertMeta pushes one sandbox's metadata to every CubeProxy. +func (c *Client) UpsertMeta(ctx context.Context, meta lifecycle.SandboxLifecycleMeta) error { + body, err := json.Marshal(meta) + if err != nil { + return fmt.Errorf("marshal meta: %w", err) + } + return c.broadcast(ctx, http.MethodPost, "/admin/meta/upsert", body) +} + +// DeleteMeta drops a sandbox from every CubeProxy. +func (c *Client) DeleteMeta(ctx context.Context, sandboxID string) error { + body, err := json.Marshal(map[string]string{"sandbox_id": sandboxID}) + if err != nil { + return fmt.Errorf("marshal delete: %w", err) + } + return c.broadcast(ctx, http.MethodPost, "/admin/meta/delete", body) +} + +// SetState pushes a state transition to every CubeProxy. +// state must be one of "running" | "pausing" | "paused". +func (c *Client) SetState(ctx context.Context, sandboxID, state string) error { + body, err := json.Marshal(map[string]string{ + "sandbox_id": sandboxID, + "state": state, + }) + if err != nil { + return fmt.Errorf("marshal state: %w", err) + } + return c.broadcast(ctx, http.MethodPost, "/admin/state", body) +} + +// PullLastActive queries every CubeProxy for entries newer than `since` and +// merges them by max(ts). Returns merged entries plus the *minimum* `now` +// across responses (callers use that as the next `since`, ensuring no gap). +// +// Endpoint failures are logged at warn and skipped: a single CubeProxy being +// unreachable shouldn't blind the sweeper to entries on the others. +func (c *Client) PullLastActive(ctx context.Context, since int64) (map[string]int64, int64, error) { + merged := make(map[string]int64) + minNow := int64(0) + first := true + var ( + successes int + lastErr error + ) + + for _, url := range c.endpoints { + path := "/admin/last_active?since=" + strconv.FormatInt(since, 10) + raw, err := c.do(ctx, http.MethodGet, url, path, nil) + if err != nil { + c.log.Warn("pull last_active failed", zap.String("url", url), zap.Error(err)) + lastErr = err + continue + } + var resp LastActiveResponse + if err := json.Unmarshal(raw, &resp); err != nil { + c.log.Warn("pull last_active: bad json", zap.String("url", url), zap.Error(err)) + lastErr = err + continue + } + for sid, ts := range resp.Entries { + if cur, ok := merged[sid]; !ok || ts > cur { + merged[sid] = ts + } + } + if first || resp.Now < minNow { + minNow = resp.Now + first = false + } + successes++ + } + + if successes == 0 { + if lastErr == nil { + lastErr = errors.New("no admin endpoints succeeded") + } + return nil, 0, lastErr + } + return merged, minNow, nil +} + +// broadcast fans out a write to every endpoint. Returns an error only when +// every endpoint failed; partial success returns nil but logs the failures +// (CubeProxy is the consumer and will eventually reconverge from the next +// stream replay). +func (c *Client) broadcast(ctx context.Context, method, path string, body []byte) error { + var ( + wg sync.WaitGroup + mu sync.Mutex + failures []error + ok int + ) + + for _, url := range c.endpoints { + wg.Add(1) + url := url + go func() { + defer wg.Done() + if _, err := c.do(ctx, method, url, path, body); err != nil { + c.log.Warn("admin push failed", + zap.String("url", url), + zap.String("path", path), + zap.Error(err)) + mu.Lock() + failures = append(failures, fmt.Errorf("%s: %w", url, err)) + mu.Unlock() + return + } + mu.Lock() + ok++ + mu.Unlock() + }() + } + wg.Wait() + + if ok == 0 && len(failures) > 0 { + return errors.Join(failures...) + } + return nil +} + +func (c *Client) do(ctx context.Context, method, base, path string, body []byte) ([]byte, error) { + var rdr io.Reader + if body != nil { + rdr = bytes.NewReader(body) + } + req, err := http.NewRequestWithContext(ctx, method, base+path, rdr) + if err != nil { + return nil, err + } + if body != nil { + req.Header.Set("Content-Type", "application/json") + } + if c.token != "" { + req.Header.Set("X-Cube-Admin-Token", c.token) + } + + resp, err := c.httpc.Do(req) + if err != nil { + return nil, err + } + defer resp.Body.Close() + + respBody, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) + if err != nil { + return nil, err + } + if resp.StatusCode/100 != 2 { + return nil, fmt.Errorf("status=%d body=%q", resp.StatusCode, respBody) + } + return respBody, nil +} diff --git a/CubeProxy/sidecar/internal/proxypush/client_test.go b/CubeProxy/sidecar/internal/proxypush/client_test.go new file mode 100644 index 000000000..48c465c41 --- /dev/null +++ b/CubeProxy/sidecar/internal/proxypush/client_test.go @@ -0,0 +1,209 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +package proxypush + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "sync" + "testing" + "time" + + "go.uber.org/zap" + + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/lifecycle" +) + +// fakeAdmin is a tiny stand-in for CubeProxy's admin server. It records every +// request it sees and serves /admin/last_active from an in-memory map. +type fakeAdmin struct { + mu sync.Mutex + lastActive map[string]int64 + now int64 + tokenWanted string + + upserts []map[string]any + deletes []map[string]any + states []map[string]any + missingT int // count of requests missing the expected token +} + +func (f *fakeAdmin) handler() http.Handler { + mux := http.NewServeMux() + + check := func(w http.ResponseWriter, r *http.Request) bool { + if f.tokenWanted == "" { + return true + } + if r.Header.Get("X-Cube-Admin-Token") != f.tokenWanted { + f.mu.Lock() + f.missingT++ + f.mu.Unlock() + http.Error(w, "forbidden", http.StatusForbidden) + return false + } + return true + } + + mux.HandleFunc("/admin/meta/upsert", func(w http.ResponseWriter, r *http.Request) { + if !check(w, r) { + return + } + body, _ := io.ReadAll(r.Body) + var obj map[string]any + _ = json.Unmarshal(body, &obj) + f.mu.Lock() + f.upserts = append(f.upserts, obj) + f.mu.Unlock() + _, _ = w.Write([]byte(`{"ok":true}`)) + }) + mux.HandleFunc("/admin/meta/delete", func(w http.ResponseWriter, r *http.Request) { + if !check(w, r) { + return + } + body, _ := io.ReadAll(r.Body) + var obj map[string]any + _ = json.Unmarshal(body, &obj) + f.mu.Lock() + f.deletes = append(f.deletes, obj) + f.mu.Unlock() + _, _ = w.Write([]byte(`{"ok":true}`)) + }) + mux.HandleFunc("/admin/state", func(w http.ResponseWriter, r *http.Request) { + if !check(w, r) { + return + } + body, _ := io.ReadAll(r.Body) + var obj map[string]any + _ = json.Unmarshal(body, &obj) + f.mu.Lock() + f.states = append(f.states, obj) + f.mu.Unlock() + _, _ = w.Write([]byte(`{"ok":true}`)) + }) + mux.HandleFunc("/admin/last_active", func(w http.ResponseWriter, r *http.Request) { + if !check(w, r) { + return + } + f.mu.Lock() + entries := make(map[string]int64, len(f.lastActive)) + for k, v := range f.lastActive { + entries[k] = v + } + now := f.now + f.mu.Unlock() + _ = json.NewEncoder(w).Encode(map[string]any{ + "now": now, + "since": 0, + "count": len(entries), + "entries": entries, + }) + }) + return mux +} + +func TestUpsertMeta_RoundTrip(t *testing.T) { + fa := &fakeAdmin{lastActive: map[string]int64{}} + srv := httptest.NewServer(fa.handler()) + defer srv.Close() + + c := New([]string{srv.URL}, "", time.Second, zap.NewNop()) + meta := lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx-1", AutoPause: true, TimeoutSeconds: 60, + } + if err := c.UpsertMeta(context.Background(), meta); err != nil { + t.Fatalf("UpsertMeta failed: %v", err) + } + if len(fa.upserts) != 1 { + t.Fatalf("expected 1 upsert, got %d", len(fa.upserts)) + } + if got, _ := fa.upserts[0]["sandbox_id"].(string); got != "sbx-1" { + t.Fatalf("upsert sandbox_id wrong: %v", fa.upserts[0]) + } +} + +func TestSetState_AndDeleteMeta(t *testing.T) { + fa := &fakeAdmin{lastActive: map[string]int64{}} + srv := httptest.NewServer(fa.handler()) + defer srv.Close() + c := New([]string{srv.URL}, "", time.Second, zap.NewNop()) + + if err := c.SetState(context.Background(), "sbx-1", "paused"); err != nil { + t.Fatal(err) + } + if err := c.DeleteMeta(context.Background(), "sbx-1"); err != nil { + t.Fatal(err) + } + if len(fa.states) != 1 || fa.states[0]["state"] != "paused" { + t.Fatalf("states wrong: %+v", fa.states) + } + if len(fa.deletes) != 1 || fa.deletes[0]["sandbox_id"] != "sbx-1" { + t.Fatalf("deletes wrong: %+v", fa.deletes) + } +} + +func TestPullLastActive_MergesAcrossEndpoints(t *testing.T) { + a := &fakeAdmin{lastActive: map[string]int64{"sbx-1": 100, "sbx-2": 50}, now: 1000} + b := &fakeAdmin{lastActive: map[string]int64{"sbx-1": 200, "sbx-3": 75}, now: 1100} + sa := httptest.NewServer(a.handler()) + defer sa.Close() + sb := httptest.NewServer(b.handler()) + defer sb.Close() + + c := New([]string{sa.URL, sb.URL}, "", time.Second, zap.NewNop()) + merged, minNow, err := c.PullLastActive(context.Background(), 0) + if err != nil { + t.Fatalf("pull failed: %v", err) + } + if merged["sbx-1"] != 200 { + t.Fatalf("expected merged sbx-1=200 (max across endpoints), got %d", merged["sbx-1"]) + } + if merged["sbx-2"] != 50 || merged["sbx-3"] != 75 { + t.Fatalf("merged map wrong: %+v", merged) + } + if minNow != 1000 { + t.Fatalf("minNow should be 1000 (the smaller of the two clocks), got %d", minNow) + } +} + +func TestPullLastActive_TolerantToOneEndpointDown(t *testing.T) { + a := &fakeAdmin{lastActive: map[string]int64{"sbx-1": 100}, now: 500} + sa := httptest.NewServer(a.handler()) + defer sa.Close() + + // "sb" deliberately points at an unused port to force a connection error. + c := New([]string{sa.URL, "http://127.0.0.1:1"}, "", 200*time.Millisecond, zap.NewNop()) + merged, _, err := c.PullLastActive(context.Background(), 0) + if err != nil { + t.Fatalf("partial-success pull should not error: %v", err) + } + if merged["sbx-1"] != 100 { + t.Fatalf("expected merged sbx-1=100, got %+v", merged) + } +} + +func TestUpsertMeta_TokenHeader(t *testing.T) { + fa := &fakeAdmin{lastActive: map[string]int64{}, tokenWanted: "secret"} + srv := httptest.NewServer(fa.handler()) + defer srv.Close() + + withTok := New([]string{srv.URL}, "secret", time.Second, zap.NewNop()) + if err := withTok.UpsertMeta(context.Background(), + lifecycle.SandboxLifecycleMeta{SandboxID: "ok"}); err != nil { + t.Fatalf("token-bearing call should succeed: %v", err) + } + + noTok := New([]string{srv.URL}, "", time.Second, zap.NewNop()) + if err := noTok.UpsertMeta(context.Background(), + lifecycle.SandboxLifecycleMeta{SandboxID: "fail"}); err == nil { + t.Fatal("token-less call should error") + } + if fa.missingT != 1 { + t.Fatalf("expected exactly 1 missing-token rejection, got %d", fa.missingT) + } +} diff --git a/CubeProxy/sidecar/internal/redisstream/stream.go b/CubeProxy/sidecar/internal/redisstream/stream.go new file mode 100644 index 000000000..9aee4b878 --- /dev/null +++ b/CubeProxy/sidecar/internal/redisstream/stream.go @@ -0,0 +1,203 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +// Package redisstream owns every interaction with the lifecycle Redis schema: +// the meta HSet bootstrap, the events stream consumer, and the per-sandbox +// state locks used to serialize pause/resume across sidecar instances. +package redisstream + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "time" + + "github.com/redis/go-redis/v9" + "go.uber.org/zap" + + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/lifecycle" +) + +// Client wraps a go-redis client with lifecycle-shaped methods. +type Client struct { + rdb *redis.Client + log *zap.Logger +} + +func New(rdb *redis.Client, log *zap.Logger) *Client { + return &Client{rdb: rdb, log: log} +} + +// Bootstrap returns every sandbox in cube:sandbox:meta. Empty result is fine — +// it just means CubeMaster hasn't published anything yet. +func (c *Client) Bootstrap(ctx context.Context) (map[string]lifecycle.SandboxLifecycleMeta, error) { + raw, err := c.rdb.HGetAll(ctx, lifecycle.MetaKey).Result() + if err != nil { + return nil, fmt.Errorf("hgetall %s: %w", lifecycle.MetaKey, err) + } + out := make(map[string]lifecycle.SandboxLifecycleMeta, len(raw)) + for sid, payload := range raw { + var meta lifecycle.SandboxLifecycleMeta + if err := json.Unmarshal([]byte(payload), &meta); err != nil { + c.log.Warn("bootstrap: skipping bad meta entry", + zap.String("sandbox_id", sid), zap.Error(err)) + continue + } + // Defensive: ensure sandbox_id matches the hash field even if the + // payload happens to have a different value — we trust the field. + meta.SandboxID = sid + out[sid] = meta + } + return out, nil +} + +// EnsureGroup creates the consumer group on the events stream, ignoring +// "BUSYGROUP" (group already exists) errors. MKSTREAM lets the group be +// created before any events have been published. +func (c *Client) EnsureGroup(ctx context.Context, group string) error { + err := c.rdb.XGroupCreateMkStream(ctx, lifecycle.EventStreamKey, group, "$").Err() + if err == nil { + return nil + } + // go-redis surfaces BUSYGROUP as a generic error with a known message. + if isBusyGroup(err) { + return nil + } + return fmt.Errorf("xgroup create mkstream: %w", err) +} + +// Event is a decoded entry from the events stream. +type Event struct { + StreamID string + Op string // create | delete + SandboxID string + Meta *lifecycle.SandboxLifecycleMeta // populated only on create + Timestamp int64 +} + +// ReadGroup blocks for up to `block` waiting for new events on the stream. +// Returns when at least one entry arrives, when the context is cancelled, or +// when the block timeout expires (in which case it returns an empty slice and +// nil error — the caller loops). +func (c *Client) ReadGroup(ctx context.Context, group, consumer string, block time.Duration, count int) ([]Event, error) { + res, err := c.rdb.XReadGroup(ctx, &redis.XReadGroupArgs{ + Group: group, + Consumer: consumer, + Streams: []string{lifecycle.EventStreamKey, ">"}, + Count: int64(count), + Block: block, + }).Result() + + if errors.Is(err, redis.Nil) { + return nil, nil + } + if err != nil { + // Block-timeout shows up as a context-deadline-ish error from + // go-redis when no entries arrive and BLOCK > 0; treat as empty. + if errors.Is(err, context.DeadlineExceeded) { + return nil, nil + } + return nil, fmt.Errorf("xreadgroup: %w", err) + } + + var out []Event + for _, stream := range res { + for _, msg := range stream.Messages { + ev := decodeEvent(msg) + if ev != nil { + out = append(out, *ev) + } else { + c.log.Warn("redisstream: dropping unparseable event", + zap.String("id", msg.ID), zap.Any("values", msg.Values)) + // Still ack so we don't loop on it. + _ = c.Ack(ctx, group, msg.ID) + } + } + } + return out, nil +} + +// Ack marks the event as processed so it leaves the consumer's pending list. +func (c *Client) Ack(ctx context.Context, group, id string) error { + return c.rdb.XAck(ctx, lifecycle.EventStreamKey, group, id).Err() +} + +// AcquireState performs a SET NX EX on cube:sandbox:state: with the +// supplied desired state. Returns true on success. Used to coordinate concurrent +// pause/resume across sidecars: whoever wins the SETNX owns the transition. +func (c *Client) AcquireState(ctx context.Context, sandboxID, state string, ttl time.Duration) (bool, error) { + key := lifecycle.StateKeyPrefix + sandboxID + ok, err := c.rdb.SetNX(ctx, key, state, ttl).Result() + if err != nil { + return false, fmt.Errorf("setnx %s: %w", key, err) + } + return ok, nil +} + +// SetState forces the state value (overwriting any existing). Used to +// transition pausing → paused or resuming → running once the underlying +// operation has actually completed. +func (c *Client) SetState(ctx context.Context, sandboxID, state string, ttl time.Duration) error { + key := lifecycle.StateKeyPrefix + sandboxID + return c.rdb.Set(ctx, key, state, ttl).Err() +} + +// ClearState drops the key altogether. Used on rollback (operation failed) +// and on sandbox delete. +func (c *Client) ClearState(ctx context.Context, sandboxID string) error { + key := lifecycle.StateKeyPrefix + sandboxID + return c.rdb.Del(ctx, key).Err() +} + +// GetState returns the current state and whether the key exists. +func (c *Client) GetState(ctx context.Context, sandboxID string) (string, bool, error) { + key := lifecycle.StateKeyPrefix + sandboxID + v, err := c.rdb.Get(ctx, key).Result() + if errors.Is(err, redis.Nil) { + return "", false, nil + } + if err != nil { + return "", false, err + } + return v, true, nil +} + +func decodeEvent(msg redis.XMessage) *Event { + op, _ := msg.Values[lifecycle.FieldOp].(string) + sid, _ := msg.Values[lifecycle.FieldSandboxID].(string) + if op == "" || sid == "" { + return nil + } + ev := &Event{ + StreamID: msg.ID, + Op: op, + SandboxID: sid, + } + if ts, ok := msg.Values[lifecycle.FieldTimestamp].(string); ok { + // CubeMaster writes the millisecond unix timestamp; tolerate both + // string and numeric forms. + var t int64 + if _, err := fmt.Sscanf(ts, "%d", &t); err == nil { + ev.Timestamp = t + } + } + if op == lifecycle.OpCreate { + if payload, ok := msg.Values[lifecycle.FieldPayload].(string); ok && payload != "" { + var meta lifecycle.SandboxLifecycleMeta + if err := json.Unmarshal([]byte(payload), &meta); err == nil { + meta.SandboxID = sid + ev.Meta = &meta + } + } + } + return ev +} + +func isBusyGroup(err error) bool { + if err == nil { + return false + } + return err.Error() == "BUSYGROUP Consumer Group name already exists" +} diff --git a/CubeProxy/sidecar/internal/registry/registry.go b/CubeProxy/sidecar/internal/registry/registry.go new file mode 100644 index 000000000..37663d8ba --- /dev/null +++ b/CubeProxy/sidecar/internal/registry/registry.go @@ -0,0 +1,150 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +// Package registry holds the in-memory map of every sandbox the sidecar is +// tracking. It is the single source of truth that the sweeper reads to make +// pause decisions and that the resume HTTP handler consults to know whether +// auto-resume is enabled for a given sandbox. +package registry + +import ( + "sort" + "sync" + "time" + + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/lifecycle" +) + +// Entry is one row in the registry. LastActiveMs is updated by the +// last-active poller; everything else comes from CubeMaster events. +type Entry struct { + Meta lifecycle.SandboxLifecycleMeta + + // LastActiveMs is the most recent activity timestamp seen across all + // CubeProxy instances (sidecar takes max() over instances). Zero means + // "never observed" — the sweeper falls back to Meta.CreatedAt for the + // idle calculation. + LastActiveMs int64 + + // FirstSeenAt is when the sidecar registered the sandbox locally. The + // sweeper compares this against config.GracePeriod so a freshly-restarted + // sidecar doesn't pause everything in its first sweep before it has a + // chance to receive any activity reports. + FirstSeenAt time.Time +} + +// Registry is a goroutine-safe map. Reads happen on every sweep + resume +// request; writes happen on every stream event + last_active poll. We pick +// sync.RWMutex over sync.Map because the sweep wants a deterministic snapshot. +type Registry struct { + mu sync.RWMutex + entries map[string]*Entry +} + +func New() *Registry { + return &Registry{entries: make(map[string]*Entry)} +} + +// Upsert installs (or replaces) the lifecycle meta for a sandbox. Existing +// LastActiveMs is preserved so a stream-replay event doesn't roll back our +// activity view. FirstSeenAt is set on first insert and not overwritten. +func (r *Registry) Upsert(meta lifecycle.SandboxLifecycleMeta) { + r.mu.Lock() + defer r.mu.Unlock() + + cur, ok := r.entries[meta.SandboxID] + if !ok { + r.entries[meta.SandboxID] = &Entry{ + Meta: meta, + FirstSeenAt: time.Now(), + } + return + } + cur.Meta = meta +} + +// Delete drops a sandbox; no-op if absent. +func (r *Registry) Delete(sandboxID string) { + r.mu.Lock() + delete(r.entries, sandboxID) + r.mu.Unlock() +} + +// MergeLastActive bumps LastActiveMs to max(current, ts). Returns true when +// the timestamp moved forward, so callers can avoid spurious work. +// Sandboxes that aren't in the registry are ignored — last_active is only +// meaningful for sandboxes CubeMaster has told us about. +func (r *Registry) MergeLastActive(sandboxID string, tsMs int64) bool { + r.mu.Lock() + defer r.mu.Unlock() + + e, ok := r.entries[sandboxID] + if !ok { + return false + } + if tsMs > e.LastActiveMs { + e.LastActiveMs = tsMs + return true + } + return false +} + +// Get returns a copy of the entry for inspection. Returns nil when absent. +func (r *Registry) Get(sandboxID string) *Entry { + r.mu.RLock() + defer r.mu.RUnlock() + e, ok := r.entries[sandboxID] + if !ok { + return nil + } + cp := *e + return &cp +} + +// Snapshot returns every entry in a deterministic (sandbox-ID-sorted) order. +// Used by the sweeper. Each returned Entry is a copy — safe to mutate. +func (r *Registry) Snapshot() []Entry { + r.mu.RLock() + defer r.mu.RUnlock() + + ids := make([]string, 0, len(r.entries)) + for k := range r.entries { + ids = append(ids, k) + } + sort.Strings(ids) + + out := make([]Entry, 0, len(ids)) + for _, id := range ids { + out = append(out, *r.entries[id]) + } + return out +} + +// Len returns the entry count. Cheap; used by the /metrics endpoint. +func (r *Registry) Len() int { + r.mu.RLock() + defer r.mu.RUnlock() + return len(r.entries) +} + +// Reset removes every entry. Bootstrap uses it before re-applying a fresh +// HGETALL to avoid leaking stale entries across sidecar restarts within the +// same in-memory state (e.g. tests). +func (r *Registry) Reset() { + r.mu.Lock() + r.entries = make(map[string]*Entry) + r.mu.Unlock() +} + +// SetFirstSeenAt overrides the FirstSeenAt for a sandbox. Production code +// must not call this — FirstSeenAt is meant to be set exactly once on the +// initial Upsert. It exists so tests can backdate the registry past the +// sweeper's grace period without sleeping. +func (r *Registry) SetFirstSeenAt(sandboxID string, t time.Time) { + r.mu.Lock() + if e, ok := r.entries[sandboxID]; ok { + e.FirstSeenAt = t + } + r.mu.Unlock() +} diff --git a/CubeProxy/sidecar/internal/registry/registry_test.go b/CubeProxy/sidecar/internal/registry/registry_test.go new file mode 100644 index 000000000..d7ab7727f --- /dev/null +++ b/CubeProxy/sidecar/internal/registry/registry_test.go @@ -0,0 +1,98 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +package registry + +import ( + "testing" + "time" + + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/lifecycle" +) + +func TestUpsert_PreservesLastActiveOnReplace(t *testing.T) { + r := New() + r.Upsert(lifecycle.SandboxLifecycleMeta{SandboxID: "sbx", AutoPause: true}) + if !r.MergeLastActive("sbx", 1000) { + t.Fatal("first MergeLastActive should advance the watermark") + } + // Replace meta with a fresh value (e.g. stream replay). + r.Upsert(lifecycle.SandboxLifecycleMeta{SandboxID: "sbx", AutoPause: false}) + got := r.Get("sbx") + if got == nil { + t.Fatal("entry vanished after re-upsert") + } + if got.LastActiveMs != 1000 { + t.Fatalf("LastActiveMs reset to %d; should have been preserved at 1000", got.LastActiveMs) + } + if got.Meta.AutoPause { + t.Fatal("Meta.AutoPause should reflect latest upsert") + } +} + +func TestMergeLastActive_TakesMaxAndIgnoresUnknown(t *testing.T) { + r := New() + r.Upsert(lifecycle.SandboxLifecycleMeta{SandboxID: "sbx"}) + + if !r.MergeLastActive("sbx", 500) { + t.Fatal("expected first merge to advance") + } + if r.MergeLastActive("sbx", 400) { + t.Fatal("merge with smaller ts must not advance") + } + if !r.MergeLastActive("sbx", 600) { + t.Fatal("merge with larger ts must advance") + } + if got := r.Get("sbx").LastActiveMs; got != 600 { + t.Fatalf("expected 600, got %d", got) + } + // Unknown sandbox: ignored, returns false (not an error). + if r.MergeLastActive("nope", 9999) { + t.Fatal("merge for unknown sandbox should return false") + } +} + +func TestSnapshot_StableOrderAndCopy(t *testing.T) { + r := New() + r.Upsert(lifecycle.SandboxLifecycleMeta{SandboxID: "b"}) + r.Upsert(lifecycle.SandboxLifecycleMeta{SandboxID: "a"}) + r.Upsert(lifecycle.SandboxLifecycleMeta{SandboxID: "c"}) + + snap := r.Snapshot() + if len(snap) != 3 || snap[0].Meta.SandboxID != "a" || snap[1].Meta.SandboxID != "b" || snap[2].Meta.SandboxID != "c" { + t.Fatalf("snapshot order wrong: %+v", snap) + } + + // Snapshot entries are copies — mutating must not bleed into the registry. + snap[0].LastActiveMs = 7777 + if r.Get("a").LastActiveMs != 0 { + t.Fatal("snapshot mutation leaked back into registry") + } +} + +func TestFirstSeenAt_StableAcrossUpserts(t *testing.T) { + r := New() + r.Upsert(lifecycle.SandboxLifecycleMeta{SandboxID: "sbx"}) + first := r.Get("sbx").FirstSeenAt + if first.IsZero() { + t.Fatal("FirstSeenAt should be set on first upsert") + } + time.Sleep(2 * time.Millisecond) + r.Upsert(lifecycle.SandboxLifecycleMeta{SandboxID: "sbx", AutoPause: true}) + if got := r.Get("sbx").FirstSeenAt; !got.Equal(first) { + t.Fatalf("FirstSeenAt changed across re-upsert: was %s now %s", first, got) + } +} + +func TestDelete(t *testing.T) { + r := New() + r.Upsert(lifecycle.SandboxLifecycleMeta{SandboxID: "sbx"}) + r.Delete("sbx") + if r.Get("sbx") != nil { + t.Fatal("entry should be gone after Delete") + } + // Delete on absent must not panic. + r.Delete("sbx") + r.Delete("never-existed") +} diff --git a/CubeProxy/sidecar/internal/resumer/iface.go b/CubeProxy/sidecar/internal/resumer/iface.go new file mode 100644 index 000000000..c329a38ba --- /dev/null +++ b/CubeProxy/sidecar/internal/resumer/iface.go @@ -0,0 +1,32 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +package resumer + +import ( + "context" + "time" +) + +// stateStore is the subset of redisstream.Client we use. Tests substitute an +// in-memory fake so we don't depend on a live Redis. +type stateStore interface { + AcquireState(ctx context.Context, sandboxID, state string, ttl time.Duration) (bool, error) + SetState(ctx context.Context, sandboxID, state string, ttl time.Duration) error + ClearState(ctx context.Context, sandboxID string) error + GetState(ctx context.Context, sandboxID string) (string, bool, error) +} + +// resumePauser describes the slice of CubeMaster client we need. +type resumePauser interface { + Resume(ctx context.Context, sandboxID, instanceType string) error +} + +// stateNotifier is the slice of proxypush.Client we use. DeleteMeta is +// invoked when CubeMaster reports the sandbox no longer exists so we evict +// the local proxy entry alongside the shared registry one. +type stateNotifier interface { + SetState(ctx context.Context, sandboxID, state string) error + DeleteMeta(ctx context.Context, sandboxID string) error +} diff --git a/CubeProxy/sidecar/internal/resumer/resumer.go b/CubeProxy/sidecar/internal/resumer/resumer.go new file mode 100644 index 000000000..695ad5a8f --- /dev/null +++ b/CubeProxy/sidecar/internal/resumer/resumer.go @@ -0,0 +1,306 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +// Package resumer is the request-side counterpart to sweeper. CubeProxy +// posts to /internal/resume when it sees a paused sandbox; this package +// serializes the work so concurrent dataplane requests for the same sandbox +// share a single resume RPC. +package resumer + +import ( + "context" + "errors" + "sync" + "time" + + "go.uber.org/zap" + + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/cubemasterclient" + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/registry" +) + +// Options bundles dependencies for the Resumer. Concrete production wiring +// lives in cmd/sidecar/main.go; the interface types defined in iface.go let +// tests substitute fakes for Redis / CubeMaster / CubeProxy. +type Options struct { + Registry *registry.Registry + Redis stateStore + CubeMaster resumePauser + ProxyPush stateNotifier + StateLockTTL time.Duration + Log *zap.Logger +} + +// Resumer coalesces concurrent resume requests for the same sandbox into a +// single in-flight call to CubeMaster. It does NOT cache outcomes — every +// caller that arrives after a resume completes will see the registry/state +// already updated and return immediately, but a fresh paused→resume cycle +// must be allowed to fire a new RPC. +type Resumer struct { + o Options + mu sync.Mutex + calls map[string]*call +} + +// call represents one in-flight resume operation. Every goroutine waiting on +// the same sandbox blocks on done; the first arrival drives the work. +type call struct { + done chan struct{} + err error +} + +// New constructs a Resumer. +func New(o Options) *Resumer { + return &Resumer{ + o: o, + calls: make(map[string]*call), + } +} + +// Resume drives the sandbox back to running. Safe to call concurrently with +// the same sandboxID; only one CubeMaster RPC fires per outstanding paused +// state. +func (r *Resumer) Resume(ctx context.Context, sandboxID string) error { + if sandboxID == "" { + return errors.New("empty sandbox_id") + } + + r.mu.Lock() + if c, ok := r.calls[sandboxID]; ok { + r.mu.Unlock() + select { + case <-c.done: + return c.err + case <-ctx.Done(): + return ctx.Err() + } + } + c := &call{done: make(chan struct{})} + r.calls[sandboxID] = c + r.mu.Unlock() + + defer func() { + r.mu.Lock() + delete(r.calls, sandboxID) + r.mu.Unlock() + close(c.done) + }() + + c.err = r.doResume(ctx, sandboxID) + return c.err +} + +func (r *Resumer) doResume(ctx context.Context, sandboxID string) error { + entry := r.o.Registry.Get(sandboxID) + if entry == nil { + return errors.New("sandbox not in registry") + } + if !entry.Meta.AutoResume { + return errors.New("auto_resume not enabled for sandbox") + } + + // cube:sandbox:state: is dual-purpose: terminal markers (paused / + // running) AND in-flight transition locks (pausing / resuming) live in + // the same key. SETNX alone can't distinguish "I'm resuming this" from + // "this sandbox is parked at terminal-paused waiting for someone to + // resume it" — we need to peek first. + switch ownErr := r.acquireResumeOwnership(ctx, sandboxID); { + case ownErr == nil: + // We own the resume; call CubeMaster. + if err := r.callCubeMasterResume(ctx, sandboxID, entry.Meta.InstanceType); err != nil { + return err + } + case errors.Is(ownErr, errAlreadyRunning): + // Sandbox is already running per Redis. Skip the RPC and run the + // success bookkeeping below so we re-assert state to the proxy. + // (This is the case where a peer resume already completed but the + // proxy's local dict still says "paused".) + default: + return ownErr + } + + // Success bookkeeping. Three writes, all best-effort: + // + // 1. Redis state → "running" so the next request from any sidecar + // instance sees the right state. + // 2. CubeProxy local state dict → "running" so the rewrite_phase gate + // stops triggering resumes for this sandbox. + // 3. In-memory registry LastActiveMs → now. Without (3) the sweeper + // sees a stale baseline (LastActiveMs=0, CreatedAt from minutes + // ago) and immediately on the next 5s tick logs "idle threshold + // exceeded; pausing" for the sandbox we *just* woke up. The log + // is mostly cosmetic — tryPause will SETNX-fail against + // state=running and silently return — but the noise is + // misleading. The proxy's log_phase will eventually overwrite + // this via the periodic last_active poll, but we want the right + // answer immediately, not 5–10 seconds later. + if err := r.o.Redis.SetState(ctx, sandboxID, "running", r.o.StateLockTTL); err != nil { + r.o.Log.Warn("write running state failed", + zap.String("sandbox_id", sandboxID), zap.Error(err)) + } + if err := r.o.ProxyPush.SetState(ctx, sandboxID, "running"); err != nil { + // Best-effort; CubeProxy locally also flips state to running on a + // successful resume sub-request, so this is a safety net. + r.o.Log.Warn("push running state failed", + zap.String("sandbox_id", sandboxID), zap.Error(err)) + } + r.o.Registry.MergeLastActive(sandboxID, time.Now().UnixMilli()) + + r.o.Log.Info("auto-resumed sandbox", zap.String("sandbox_id", sandboxID)) + return nil +} + +// callCubeMasterResume issues the resume RPC and maps the three classes +// of CubeMaster response (success / not-found / already-running / real +// failure) onto the appropriate caller-side cleanup. Returns nil when the +// caller should proceed to the success-bookkeeping path; non-nil when the +// caller should bail with an error. +func (r *Resumer) callCubeMasterResume(ctx context.Context, sandboxID, instanceType string) error { + resumeErr := r.o.CubeMaster.Resume(ctx, sandboxID, instanceType) + if resumeErr == nil { + return nil + } + var apiErr *cubemasterclient.APIError + switch { + case errors.As(resumeErr, &apiErr) && apiErr.IsNotFound(): + // CubeMaster doesn't know this sandbox anymore — deleted out + // from under us. Evict everywhere and surface as an error to + // the HTTP caller so CubeProxy returns 5xx (the dataplane + // request can't be served either way). + _ = r.o.Redis.ClearState(ctx, sandboxID) + _ = r.o.ProxyPush.DeleteMeta(ctx, sandboxID) + r.o.Registry.Delete(sandboxID) + r.o.Log.Info("sandbox not found on cubemaster during resume; evicted", + zap.String("sandbox_id", sandboxID), + zap.Int("ret_code", apiErr.RetCode), + zap.String("ret_msg", apiErr.RetMsg)) + return errors.New("sandbox no longer exists") + case errors.As(resumeErr, &apiErr) && apiErr.IsAlreadyInState(): + // Already running. Reconcile state and return success — the + // dataplane request can proceed. + r.o.Log.Info("sandbox already running on cubemaster; reconciling state", + zap.String("sandbox_id", sandboxID), + zap.Int("ret_code", apiErr.RetCode)) + return nil + default: + // Real failure: clear the resuming key so a future request can + // retry, and surface the error. + _ = r.o.Redis.ClearState(ctx, sandboxID) + return errors.New("cubemaster resume: " + resumeErr.Error()) + } +} + +// acquireResumeOwnership decides whether the current call should drive +// the resume RPC, wait for a peer to finish, or return immediately. It +// returns nil when the caller owns the resume (i.e. has the lock written +// as "resuming"); a non-nil error when the caller should NOT proceed +// (peer in flight resolved, sandbox already running, or real failure). +// +// The state-key conflict (terminal markers vs. transition locks share +// the key) is resolved by GET-ing the current value: +// +// - "paused" or expired: we own the resume — write "resuming" +// - "running": nothing to do, return nil-and-success +// via a sentinel (the caller's success +// bookkeeping then runs and re-asserts +// state, which is the right behaviour +// for race-recovery). +// - "pausing" or "resuming": a peer is in flight → waitForRunning +// +// This is intentionally racy: between GET and SET another sidecar could +// claim the key. That's fine because the worst case is two resumers both +// calling CubeMaster.Resume — which CubeMaster handles idempotently +// (returns "already running" the second time, which we already map to +// success in the caller). +func (r *Resumer) acquireResumeOwnership(ctx context.Context, sandboxID string) error { + cur, ok, err := r.o.Redis.GetState(ctx, sandboxID) + if err != nil { + return err + } + + switch { + case !ok, cur == "paused": + // Either no lock at all (most common after sweeper's TTL expired) + // or terminal "paused" left by a successful sweep. Either way we + // claim ownership by SET-ing "resuming". + if err := r.o.Redis.SetState(ctx, sandboxID, "resuming", r.o.StateLockTTL); err != nil { + return err + } + return nil + case cur == "running": + // Sandbox is already running on Redis's view. No-op resume; the + // caller's success path will re-push running to the proxy in case + // the local dict drifted. + r.o.Log.Info("resume requested but sandbox already running; reconciling", + zap.String("sandbox_id", sandboxID)) + return errAlreadyRunning + case cur == "pausing" || cur == "resuming": + // Active transition by a peer → wait it out. waitForRunning + // returning nil means the peer transitioned to "running"; treat + // that as a no-op resume from our perspective so we DON'T issue + // our own duplicate RPC. Any other return value (peer-paused, + // lock expired, ctx done) propagates as an error. + if err := r.waitForRunning(ctx, sandboxID); err != nil { + return err + } + return errAlreadyRunning + default: + // Unknown state — fall back to wait, same translation rule. + r.o.Log.Warn("unknown state during resume ownership probe", + zap.String("sandbox_id", sandboxID), + zap.String("state", cur)) + if err := r.waitForRunning(ctx, sandboxID); err != nil { + return err + } + return errAlreadyRunning + } +} + +// errAlreadyRunning is returned by acquireResumeOwnership when GetState +// reports the sandbox is already running. The doResume caller treats it +// as a successful no-op (state will be re-asserted into the proxy dict). +var errAlreadyRunning = errors.New("sandbox already running") + +// waitForRunning is invoked when AcquireState lost the SETNX race — i.e. +// some other key/holder occupies cube:sandbox:state:. We poll that key +// for one of three terminal outcomes: +// +// - state == "running" → peer succeeded, request can proceed. +// - state == "paused" → peer gave up; bail with an error so +// CubeProxy returns 503 and the next request +// gets a fresh resume attempt. +// - key expired (!ok) → peer crashed mid-flight; do NOT treat this +// as success — return a clear error so the +// caller can retry. Without this guard we +// would silently let through a request to a +// still-paused sandbox. +func (r *Resumer) waitForRunning(ctx context.Context, sandboxID string) error { + const pollEvery = 200 * time.Millisecond + t := time.NewTicker(pollEvery) + defer t.Stop() + for { + state, ok, err := r.o.Redis.GetState(ctx, sandboxID) + if err != nil { + return err + } + switch { + case !ok: + // Peer's lock expired without writing a terminal state. Don't + // pretend everything is fine — the sandbox is in an unknown + // state. The caller will surface a 503 and the next request + // re-enters Resume() and re-acquires the lock cleanly. + return errors.New("peer resume lock expired without resolution") + case state == "running": + return nil + case state == "paused": + return errors.New("peer resume left sandbox paused") + // pausing / resuming → keep polling + } + select { + case <-ctx.Done(): + return ctx.Err() + case <-t.C: + } + } +} diff --git a/CubeProxy/sidecar/internal/resumer/resumer_test.go b/CubeProxy/sidecar/internal/resumer/resumer_test.go new file mode 100644 index 000000000..2c6d0c399 --- /dev/null +++ b/CubeProxy/sidecar/internal/resumer/resumer_test.go @@ -0,0 +1,320 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +package resumer + +import ( + "context" + "errors" + "sync" + "sync/atomic" + "testing" + "time" + + "go.uber.org/zap" + + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/lifecycle" + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/registry" +) + +// fakeStore is the resumer-side test double for the redisstream client. +type fakeStore struct { + mu sync.Mutex + states map[string]string + // allowAcquire controls whether AcquireState succeeds. When the second + // element is non-empty, AcquireState seeds that state value into the + // map (simulating a peer holding the lock) and returns false. + preLocked map[string]string +} + +func newFakeStore() *fakeStore { + return &fakeStore{states: make(map[string]string), preLocked: make(map[string]string)} +} + +func (f *fakeStore) AcquireState(_ context.Context, sid, state string, _ time.Duration) (bool, error) { + f.mu.Lock() + defer f.mu.Unlock() + if v, ok := f.preLocked[sid]; ok { + f.states[sid] = v + return false, nil + } + if _, ok := f.states[sid]; ok { + return false, nil + } + f.states[sid] = state + return true, nil +} + +func (f *fakeStore) SetState(_ context.Context, sid, state string, _ time.Duration) error { + f.mu.Lock() + defer f.mu.Unlock() + f.states[sid] = state + return nil +} + +func (f *fakeStore) ClearState(_ context.Context, sid string) error { + f.mu.Lock() + defer f.mu.Unlock() + delete(f.states, sid) + return nil +} + +func (f *fakeStore) GetState(_ context.Context, sid string) (string, bool, error) { + f.mu.Lock() + defer f.mu.Unlock() + v, ok := f.states[sid] + return v, ok, nil +} + +func (f *fakeStore) state(sid string) string { + f.mu.Lock() + defer f.mu.Unlock() + return f.states[sid] +} + +// fakeMaster captures Resume calls; supports artificial latency to exercise +// the in-flight de-duplication path. +type fakeMaster struct { + mu sync.Mutex + calls int32 + latency time.Duration + failNext bool + failError error +} + +func (f *fakeMaster) Resume(ctx context.Context, _, _ string) error { + atomic.AddInt32(&f.calls, 1) + if f.latency > 0 { + select { + case <-time.After(f.latency): + case <-ctx.Done(): + return ctx.Err() + } + } + f.mu.Lock() + defer f.mu.Unlock() + if f.failNext { + f.failNext = false + return f.failError + } + return nil +} + +type fakePush struct { + mu sync.Mutex + pushed []string + deleted []string +} + +func (f *fakePush) SetState(_ context.Context, _, state string) error { + f.mu.Lock() + defer f.mu.Unlock() + f.pushed = append(f.pushed, state) + return nil +} + +func (f *fakePush) DeleteMeta(_ context.Context, sid string) error { + f.mu.Lock() + defer f.mu.Unlock() + f.deleted = append(f.deleted, sid) + return nil +} + +func newTestResumer(reg *registry.Registry, store *fakeStore, master *fakeMaster, push *fakePush) *Resumer { + return New(Options{ + Registry: reg, + Redis: store, + CubeMaster: master, + ProxyPush: push, + StateLockTTL: 30 * time.Second, + Log: zap.NewNop(), + }) +} + +func TestResumer_HappyPath(t *testing.T) { + reg := registry.New() + reg.Upsert(lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx", InstanceType: "cubebox", AutoResume: true, + CreatedAt: time.Now().Add(-1 * time.Hour).UnixMilli(), // ancient + }) + store := newFakeStore() + master := &fakeMaster{} + push := &fakePush{} + + beforeResume := time.Now().UnixMilli() + r := newTestResumer(reg, store, master, push) + if err := r.Resume(context.Background(), "sbx"); err != nil { + t.Fatalf("resume failed: %v", err) + } + if got := atomic.LoadInt32(&master.calls); got != 1 { + t.Fatalf("expected 1 master.Resume call, got %d", got) + } + if got := store.state("sbx"); got != "running" { + t.Fatalf("expected redis state=running, got %q", got) + } + // Regression: a successful resume must reset the registry's + // LastActiveMs to "now" — otherwise the sweeper, on its next 5s + // tick, computes idle = now - CreatedAt (hours ago) and noisily + // logs "idle threshold exceeded" for a sandbox we just woke up. + entry := reg.Get("sbx") + if entry == nil { + t.Fatal("registry entry missing after resume") + } + if entry.LastActiveMs < beforeResume { + t.Fatalf("LastActiveMs not refreshed: got %d, expected >= %d", + entry.LastActiveMs, beforeResume) + } +} + +func TestResumer_RejectsAutoResumeDisabled(t *testing.T) { + reg := registry.New() + reg.Upsert(lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx", InstanceType: "cubebox", AutoResume: false, + }) + r := newTestResumer(reg, newFakeStore(), &fakeMaster{}, &fakePush{}) + + err := r.Resume(context.Background(), "sbx") + if err == nil || err.Error() != "auto_resume not enabled for sandbox" { + t.Fatalf("expected auto_resume disabled error, got %v", err) + } +} + +func TestResumer_RejectsUnknownSandbox(t *testing.T) { + r := newTestResumer(registry.New(), newFakeStore(), &fakeMaster{}, &fakePush{}) + if err := r.Resume(context.Background(), "ghost"); err == nil { + t.Fatal("resume of unknown sandbox should error") + } +} + +func TestResumer_DedupesConcurrentResumes(t *testing.T) { + reg := registry.New() + reg.Upsert(lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx", InstanceType: "cubebox", AutoResume: true, + }) + store := newFakeStore() + master := &fakeMaster{latency: 100 * time.Millisecond} + push := &fakePush{} + r := newTestResumer(reg, store, master, push) + + const N = 20 + var wg sync.WaitGroup + errs := make([]error, N) + for i := 0; i < N; i++ { + wg.Add(1) + i := i + go func() { + defer wg.Done() + errs[i] = r.Resume(context.Background(), "sbx") + }() + } + wg.Wait() + + for i, e := range errs { + if e != nil { + t.Fatalf("goroutine %d failed: %v", i, e) + } + } + if got := atomic.LoadInt32(&master.calls); got != 1 { + t.Fatalf("concurrent resumes must coalesce into 1 RPC, got %d", got) + } +} + +func TestResumer_RollsBackOnRPCFailure(t *testing.T) { + reg := registry.New() + reg.Upsert(lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx", InstanceType: "cubebox", AutoResume: true, + }) + store := newFakeStore() + master := &fakeMaster{failNext: true, failError: errors.New("master 500")} + push := &fakePush{} + r := newTestResumer(reg, store, master, push) + + if err := r.Resume(context.Background(), "sbx"); err == nil { + t.Fatal("expected error from RPC failure") + } + if got := store.state("sbx"); got != "" { + t.Fatalf("state must be cleared on rollback, got %q", got) + } +} + +func TestResumer_WaitsWhenPeerHoldsLock(t *testing.T) { + // Pre-seed the state key with "resuming" — simulates a peer sidecar + // that's already mid-flight on this sandbox. acquireResumeOwnership + // should observe it via GetState and route to waitForRunning instead + // of issuing a duplicate CubeMaster RPC. + reg := registry.New() + reg.Upsert(lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx", InstanceType: "cubebox", AutoResume: true, + }) + store := newFakeStore() + store.states["sbx"] = "resuming" // peer's in-flight lock, GetState-visible + master := &fakeMaster{} + push := &fakePush{} + r := newTestResumer(reg, store, master, push) + + // Have the peer flip state to running after a moment, simulating their + // RPC completing. + go func() { + time.Sleep(100 * time.Millisecond) + _ = store.SetState(context.Background(), "sbx", "running", time.Minute) + }() + + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + if err := r.Resume(ctx, "sbx"); err != nil { + t.Fatalf("waiter should have observed running, got %v", err) + } + if got := atomic.LoadInt32(&master.calls); got != 0 { + t.Fatalf("waiter must not call master.Resume, got %d", got) + } +} + +func TestResumer_OwnsResumeWhenStateIsTerminalPaused(t *testing.T) { + // The race we hit in production: sweeper successfully paused the + // sandbox, leaving cube:sandbox:state:="paused". A subsequent + // request hits the gate, which calls /_sidecar_resume. The resumer + // must NOT mistake the terminal "paused" for a peer's in-flight lock + // — it must claim ownership and drive the resume RPC. + reg := registry.New() + reg.Upsert(lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx-paused", InstanceType: "cubebox", AutoResume: true, + }) + store := newFakeStore() + store.states["sbx-paused"] = "paused" // terminal marker from sweeper + master := &fakeMaster{} + push := &fakePush{} + r := newTestResumer(reg, store, master, push) + + if err := r.Resume(context.Background(), "sbx-paused"); err != nil { + t.Fatalf("resume should succeed, got %v", err) + } + if got := atomic.LoadInt32(&master.calls); got != 1 { + t.Fatalf("resumer must call master.Resume exactly once, got %d", got) + } + if got := store.state("sbx-paused"); got != "running" { + t.Fatalf("state should be running after successful resume, got %q", got) + } +} + +func TestResumer_NoOpWhenStateIsAlreadyRunning(t *testing.T) { + // If a peer already finished resume but the proxy's local dict still + // says paused (e.g. push delay), we must not re-call CubeMaster.Resume + // — just re-assert the running state into the proxy. + reg := registry.New() + reg.Upsert(lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx-run", InstanceType: "cubebox", AutoResume: true, + }) + store := newFakeStore() + store.states["sbx-run"] = "running" + master := &fakeMaster{} + push := &fakePush{} + r := newTestResumer(reg, store, master, push) + + if err := r.Resume(context.Background(), "sbx-run"); err != nil { + t.Fatalf("resume should be a no-op success, got %v", err) + } + if got := atomic.LoadInt32(&master.calls); got != 0 { + t.Fatalf("master.Resume must NOT be called when state=running, got %d", got) + } +} diff --git a/CubeProxy/sidecar/internal/sweeper/iface.go b/CubeProxy/sidecar/internal/sweeper/iface.go new file mode 100644 index 000000000..a955a47d1 --- /dev/null +++ b/CubeProxy/sidecar/internal/sweeper/iface.go @@ -0,0 +1,36 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +package sweeper + +import ( + "context" + "time" +) + +// stateStore is the subset of redisstream.Client that the sweeper needs. +// Defining it as an interface here lets tests substitute an in-memory fake +// without spinning up a real Redis. The concrete *redisstream.Client +// satisfies this interface implicitly. +type stateStore interface { + AcquireState(ctx context.Context, sandboxID, state string, ttl time.Duration) (bool, error) + SetState(ctx context.Context, sandboxID, state string, ttl time.Duration) error + ClearState(ctx context.Context, sandboxID string) error + GetState(ctx context.Context, sandboxID string) (string, bool, error) +} + +// pauser is the subset of cubemasterclient.Client that the sweeper needs. +type pauser interface { + Pause(ctx context.Context, sandboxID, instanceType string) error +} + +// stateNotifier is the subset of proxypush.Client that the sweeper needs. +// SetState pushes a transition (running/pausing/paused). DeleteMeta is +// invoked when CubeMaster reports a sandbox no longer exists, so we evict +// the corresponding entry from CubeProxy's local meta dict in the same +// step. +type stateNotifier interface { + SetState(ctx context.Context, sandboxID, state string) error + DeleteMeta(ctx context.Context, sandboxID string) error +} diff --git a/CubeProxy/sidecar/internal/sweeper/sweeper.go b/CubeProxy/sidecar/internal/sweeper/sweeper.go new file mode 100644 index 000000000..cdc61aa52 --- /dev/null +++ b/CubeProxy/sidecar/internal/sweeper/sweeper.go @@ -0,0 +1,263 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +// Package sweeper periodically scans the registry for sandboxes that have +// exceeded their idle timeout and triggers a pause via CubeMaster. It is the +// auto-pause half of the system; resume runs on demand from the HTTP server. +package sweeper + +import ( + "context" + "errors" + "sync/atomic" + "time" + + "go.uber.org/zap" + + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/cubemasterclient" + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/registry" +) + +// Options is the dependency injection bundle for Sweeper. Pulling everything +// through here keeps the sweep logic itself a pure function of inputs and +// lets tests substitute fakes for the Redis / RPC dependencies. +type Options struct { + Registry *registry.Registry + Redis stateStore + CubeMaster pauser + ProxyPush stateNotifier + DefaultIdleTimeout time.Duration + + // BootstrapWarmup delays the first sweep after the sidecar starts so the + // last_active poller has a chance to populate activity timestamps for + // sandboxes that were already running before this sidecar instance came + // up. Without this delay, a fresh sidecar would observe LastActiveMs=0 + // for every bootstrap entry and immediately try to pause anything past + // its idle deadline — even if the sandbox has been actively serving + // traffic on the proxy. + // + // Set to 0 in tests that drive sweepOnce manually. + BootstrapWarmup time.Duration + + StateLockTTL time.Duration + Interval time.Duration + + // StartedAt is the sidecar's process start time. Used as the boundary + // between "bootstrap" and "stream" entries for the warmup gate. When + // zero, defaults to Now() at construction time. + StartedAt time.Time + + Now func() time.Time // injectable for tests + Log *zap.Logger +} + +// Sweeper iterates the registry on a fixed interval. It is intended to run as +// its own goroutine; Run returns when ctx is cancelled. +type Sweeper struct { + o Options + // metrics — exposed for testing rather than via Prometheus for now. + pauseTriggered atomic.Int64 + pauseFailed atomic.Int64 +} + +func New(o Options) *Sweeper { + if o.Now == nil { + o.Now = time.Now + } + if o.StartedAt.IsZero() { + o.StartedAt = o.Now() + } + return &Sweeper{o: o} +} + +// Run blocks until ctx is cancelled, sweeping every Interval. +func (s *Sweeper) Run(ctx context.Context) error { + t := time.NewTicker(s.o.Interval) + defer t.Stop() + // Don't fire on the first tick; let the registry warm up via Bootstrap + + // the initial last_active poll. The next tick is one Interval away. + for { + select { + case <-ctx.Done(): + return ctx.Err() + case <-t.C: + s.sweepOnce(ctx) + } + } +} + +// sweepOnce is exported (lowercase but called via tests in the same package) +// so the test can drive a single iteration deterministically. +func (s *Sweeper) sweepOnce(ctx context.Context) { + now := s.o.Now() + nowMs := now.UnixMilli() + + // Bootstrap-warmup gate: when the sidecar just started, hold off on + // pausing entries that came in via HGETALL bootstrap (FirstSeenAt ≈ + // startedAt). Two reasons: + // * LastActiveMs hasn't been backfilled yet — first last_active + // poll lands ~LastActivePoll seconds in. + // * If the proxy is healthy, those sandboxes very likely received + // traffic in the last few seconds; we just don't know it yet. + // New entries that arrive AFTER startup (FirstSeenAt > startedAt) do + // not need this protection: their CreatedAt is recent and they are + // immediately tracked by log_phase, so we can act on the standard + // `idle = now - max(LastActiveMs, CreatedAt)` rule. + withinWarmup := now.Sub(s.o.StartedAt) < s.o.BootstrapWarmup + + for _, e := range s.o.Registry.Snapshot() { + if !e.Meta.AutoPause { + continue + } + // Bootstrap entries during warmup → skip. `FirstSeenAt <= StartedAt` + // (we backdate FirstSeenAt during bootstrap, so equality means + // "loaded from HGETALL", inequality means "new event"). + if withinWarmup && !e.FirstSeenAt.After(s.o.StartedAt) { + continue + } + + // Baseline = the most recent of (LastActiveMs, CreatedAt). For a + // sandbox the sidecar has just observed via stream, LastActiveMs + // is 0 until the next request arrives — fall through to CreatedAt + // which always carries a real timestamp from CubeMaster. + baseline := e.LastActiveMs + if e.Meta.CreatedAt > baseline { + baseline = e.Meta.CreatedAt + } + if baseline == 0 { + // No CreatedAt either (legacy entry?) — fall back to FirstSeenAt + // so we don't blindly pause on zero. + baseline = e.FirstSeenAt.UnixMilli() + } + + timeout := time.Duration(e.Meta.TimeoutSeconds) * time.Second + if timeout <= 0 { + timeout = s.o.DefaultIdleTimeout + } + + idleFor := time.Duration(nowMs-baseline) * time.Millisecond + if idleFor < timeout { + continue + } + + // Already-paused fast path: if Redis says the sandbox is parked + // at terminal "paused", there is nothing for us to do — the + // dataplane will resume it on demand. Without this guard the + // sweeper logs "idle threshold exceeded" every Interval and the + // state-key TTL (StateLockTTL=60s) expires periodically, causing + // a pointless RPC churn against CubeMaster every minute. + curState, _, stateErr := s.o.Redis.GetState(ctx, e.Meta.SandboxID) + if stateErr != nil { + s.o.Log.Warn("get state failed; will attempt pause anyway", + zap.String("sandbox_id", e.Meta.SandboxID), + zap.Error(stateErr)) + } else if curState == "paused" || curState == "pausing" { + // Nothing to do. "pausing" means a peer (or our own previous + // invocation) is mid-flight; let it finish. + continue + } + + s.o.Log.Info("idle threshold exceeded; pausing", + zap.String("sandbox_id", e.Meta.SandboxID), + zap.Duration("idle_for", idleFor), + zap.Int("timeout_seconds", e.Meta.TimeoutSeconds)) + + if err := s.tryPause(ctx, e); err != nil { + s.pauseFailed.Add(1) + s.o.Log.Warn("auto-pause failed", + zap.String("sandbox_id", e.Meta.SandboxID), + zap.Duration("idle_for", idleFor), + zap.Error(err)) + } + } +} + +// tryPause acquires the state lock, calls CubeMaster, and pushes the new +// state out to CubeProxy. It is idempotent — a lost SETNX race is treated as +// success (someone else is pausing the same sandbox). +// +// Two CubeMaster ret_codes are not real failures and are mapped to +// terminal-success behaviour: +// +// - InvalidParamFormat / "key not found" → the sandbox is gone (deleted +// out from under us, e.g. CubeMaster's own cleanup raced). We evict it +// from the registry so the next sweep doesn't keep retrying forever +// and don't leave the proxy seeing a stale "pausing" state. +// - TaskStateInvalid / "sandbox is already paused" → the sandbox is +// already where we wanted it (peer sidecar / earlier failed-but-applied +// attempt). Treat exactly like a fresh successful pause. +func (s *Sweeper) tryPause(ctx context.Context, e registry.Entry) error { + sid := e.Meta.SandboxID + got, err := s.o.Redis.AcquireState(ctx, sid, "pausing", s.o.StateLockTTL) + if err != nil { + return err + } + if !got { + // Another sidecar (or our own resume handler) holds the state. Skip. + return nil + } + + // Tell CubeProxy first that the sandbox is pausing, so any new requests + // hit the 503 retry path immediately and don't race the rpc. + if err := s.o.ProxyPush.SetState(ctx, sid, "pausing"); err != nil { + s.o.Log.Warn("push pausing state failed", + zap.String("sandbox_id", sid), zap.Error(err)) + // Continue anyway — the rpc and final state push are still useful. + } + + pauseErr := s.o.CubeMaster.Pause(ctx, sid, e.Meta.InstanceType) + if pauseErr != nil { + var apiErr *cubemasterclient.APIError + switch { + case errors.As(pauseErr, &apiErr) && apiErr.IsNotFound(): + // Sandbox doesn't exist on CubeMaster anymore. Clean up local + // state and stop chasing it. + _ = s.o.Redis.ClearState(ctx, sid) + _ = s.o.ProxyPush.DeleteMeta(ctx, sid) + s.o.Registry.Delete(sid) + s.o.Log.Info("sandbox not found on cubemaster; evicting from registry", + zap.String("sandbox_id", sid), + zap.Int("ret_code", apiErr.RetCode), + zap.String("ret_msg", apiErr.RetMsg)) + return nil + case errors.As(pauseErr, &apiErr) && apiErr.IsAlreadyInState(): + // CubeMaster says the sandbox is already paused. Fall through + // to the success path so we still write `paused` state to + // Redis + push it to CubeProxy (in case a peer sidecar paused + // it but didn't push, or our previous attempt failed only on + // the post-RPC bookkeeping). + s.o.Log.Info("sandbox already paused on cubemaster; reconciling state", + zap.String("sandbox_id", sid), + zap.Int("ret_code", apiErr.RetCode)) + // no return — proceed to success bookkeeping below + default: + // Real failure. Roll back: clear the pausing state so a future + // sweep can retry, and tell CubeProxy the sandbox is back to + // running (it never actually paused). + _ = s.o.Redis.ClearState(ctx, sid) + _ = s.o.ProxyPush.SetState(ctx, sid, "running") + return errors.New("cubemaster pause: " + pauseErr.Error()) + } + } + + if err := s.o.Redis.SetState(ctx, sid, "paused", s.o.StateLockTTL); err != nil { + s.o.Log.Warn("write paused state failed", + zap.String("sandbox_id", sid), zap.Error(err)) + } + if err := s.o.ProxyPush.SetState(ctx, sid, "paused"); err != nil { + s.o.Log.Warn("push paused state failed", + zap.String("sandbox_id", sid), zap.Error(err)) + } + + s.pauseTriggered.Add(1) + s.o.Log.Info("auto-paused sandbox", + zap.String("sandbox_id", sid), + zap.Int("timeout_seconds", e.Meta.TimeoutSeconds)) + return nil +} + +// Stats returns the running counters, useful for tests and /metrics. +func (s *Sweeper) Stats() (triggered, failed int64) { + return s.pauseTriggered.Load(), s.pauseFailed.Load() +} diff --git a/CubeProxy/sidecar/internal/sweeper/sweeper_test.go b/CubeProxy/sidecar/internal/sweeper/sweeper_test.go new file mode 100644 index 000000000..76332d7cf --- /dev/null +++ b/CubeProxy/sidecar/internal/sweeper/sweeper_test.go @@ -0,0 +1,500 @@ +// Copyright (c) 2026 Tencent Inc. +// SPDX-License-Identifier: Apache-2.0 +// + +package sweeper + +import ( + "context" + "errors" + "sync" + "testing" + "time" + + "go.uber.org/zap" + + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/cubemasterclient" + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/lifecycle" + "github.com/tencentcloud/CubeSandbox/CubeProxy/sidecar/internal/registry" +) + +// fakeStore implements stateStore. It uses simple maps; lock contention isn't +// the focus of these tests, atomicity of state transitions is. +type fakeStore struct { + mu sync.Mutex + states map[string]string + + failAcquire bool + acquireBy func(sid, state string) bool // when set, controls AcquireState success +} + +func newFakeStore() *fakeStore { + return &fakeStore{states: make(map[string]string)} +} + +func (f *fakeStore) AcquireState(_ context.Context, sid, state string, _ time.Duration) (bool, error) { + if f.failAcquire { + return false, errors.New("redis down") + } + f.mu.Lock() + defer f.mu.Unlock() + if f.acquireBy != nil && !f.acquireBy(sid, state) { + return false, nil + } + if _, ok := f.states[sid]; ok { + return false, nil // already held + } + f.states[sid] = state + return true, nil +} + +func (f *fakeStore) SetState(_ context.Context, sid, state string, _ time.Duration) error { + f.mu.Lock() + defer f.mu.Unlock() + f.states[sid] = state + return nil +} + +func (f *fakeStore) GetState(_ context.Context, sid string) (string, bool, error) { + f.mu.Lock() + defer f.mu.Unlock() + v, ok := f.states[sid] + return v, ok, nil +} + +func (f *fakeStore) ClearState(_ context.Context, sid string) error { + f.mu.Lock() + defer f.mu.Unlock() + delete(f.states, sid) + return nil +} + +func (f *fakeStore) state(sid string) string { + f.mu.Lock() + defer f.mu.Unlock() + return f.states[sid] +} + +// fakeMaster captures every Pause call and lets tests inject an error. +type fakeMaster struct { + mu sync.Mutex + calls []string + failNext bool + failError error +} + +func (f *fakeMaster) Pause(_ context.Context, sid, _ string) error { + f.mu.Lock() + defer f.mu.Unlock() + f.calls = append(f.calls, sid) + if f.failNext { + f.failNext = false + return f.failError + } + return nil +} + +// fakePush records every state pushed to CubeProxy. It also tracks +// DeleteMeta calls so tests can assert that the not-found eviction path +// fires when expected. +type fakePush struct { + mu sync.Mutex + pushed map[string][]string // sandbox_id -> ordered list of states + deleted []string // sandbox_ids passed to DeleteMeta +} + +func newFakePush() *fakePush { + return &fakePush{pushed: make(map[string][]string)} +} + +func (f *fakePush) SetState(_ context.Context, sid, state string) error { + f.mu.Lock() + defer f.mu.Unlock() + f.pushed[sid] = append(f.pushed[sid], state) + return nil +} + +func (f *fakePush) DeleteMeta(_ context.Context, sid string) error { + f.mu.Lock() + defer f.mu.Unlock() + f.deleted = append(f.deleted, sid) + return nil +} + +func (f *fakePush) states(sid string) []string { + f.mu.Lock() + defer f.mu.Unlock() + out := make([]string, len(f.pushed[sid])) + copy(out, f.pushed[sid]) + return out +} + +func (f *fakePush) deletedIDs() []string { + f.mu.Lock() + defer f.mu.Unlock() + out := make([]string, len(f.deleted)) + copy(out, f.deleted) + return out +} + +// build a sweeper wired to fakes. `at` is the wall clock the sweeper sees. +func newTestSweeper(reg *registry.Registry, store *fakeStore, master *fakeMaster, push *fakePush, at time.Time) *Sweeper { + // BootstrapWarmup=0 means "act on every entry immediately" — tests + // drive sweepOnce manually and don't need the warmup gate. + return New(Options{ + Registry: reg, + Redis: store, + CubeMaster: master, + ProxyPush: push, + DefaultIdleTimeout: 5 * time.Minute, + BootstrapWarmup: 0, + StateLockTTL: 30 * time.Second, + Interval: time.Second, + Now: func() time.Time { return at }, + Log: zap.NewNop(), + }) +} + +// seedEntry inserts a registry entry. Unlike the previous grace-period +// design, the sweeper now bases its idle decision on max(LastActiveMs, +// CreatedAt), so test cases just need to set those fields appropriately. +func seedEntry(t *testing.T, r *registry.Registry, meta lifecycle.SandboxLifecycleMeta, lastActiveMs int64) { + t.Helper() + r.Upsert(meta) + if lastActiveMs > 0 { + if !r.MergeLastActive(meta.SandboxID, lastActiveMs) { + t.Fatalf("seed: MergeLastActive(%s, %d) didn't advance", meta.SandboxID, lastActiveMs) + } + } +} + +func TestSweeper_PausesIdleSandbox(t *testing.T) { + reg := registry.New() + store := newFakeStore() + master := &fakeMaster{} + push := newFakePush() + + // last_active was 10 minutes ago, timeout is 5 minutes → past due. + now := time.Now() + seedEntry(t, reg, lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx-1", InstanceType: "cubebox", + AutoPause: true, TimeoutSeconds: 300, + }, now.Add(-10*time.Minute).UnixMilli()) + + s := newTestSweeper(reg, store, master, push, now) + s.sweepOnce(context.Background()) + + if len(master.calls) != 1 || master.calls[0] != "sbx-1" { + t.Fatalf("expected single Pause call for sbx-1, got %v", master.calls) + } + if got := store.state("sbx-1"); got != "paused" { + t.Fatalf("expected redis state=paused, got %q", got) + } + pushed := push.states("sbx-1") + if len(pushed) < 2 || pushed[0] != "pausing" || pushed[len(pushed)-1] != "paused" { + t.Fatalf("expected push pausing→paused, got %v", pushed) + } + triggered, failed := s.Stats() + if triggered != 1 || failed != 0 { + t.Fatalf("stats: triggered=%d failed=%d", triggered, failed) + } +} + +func TestSweeper_SkipsSandboxWithoutAutoPause(t *testing.T) { + reg := registry.New() + store := newFakeStore() + master := &fakeMaster{} + push := newFakePush() + now := time.Now() + + seedEntry(t, reg, lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx-2", InstanceType: "cubebox", + AutoPause: false, TimeoutSeconds: 60, + }, now.Add(-1*time.Hour).UnixMilli()) + + s := newTestSweeper(reg, store, master, push, now) + s.sweepOnce(context.Background()) + + if len(master.calls) != 0 { + t.Fatalf("Pause must NOT fire when AutoPause=false: %v", master.calls) + } +} + +func TestSweeper_BootstrapWarmupSkipsBootstrapEntries(t *testing.T) { + // Verifies the bootstrap-warmup gate: while the sidecar is still in + // its warmup window, sandboxes whose FirstSeenAt is at-or-before the + // sweeper's StartedAt (i.e. loaded from HGETALL) are skipped, even if + // their CreatedAt is hours old. After the warmup elapses, the sweeper + // must act on them. + reg := registry.New() + store := newFakeStore() + master := &fakeMaster{} + push := newFakePush() + + startedAt := time.Now() + reg.Upsert(lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx-bootstrap", InstanceType: "cubebox", + AutoPause: true, TimeoutSeconds: 60, + CreatedAt: startedAt.Add(-1 * time.Hour).UnixMilli(), // ancient + }) + // Pin FirstSeenAt to startedAt — that's exactly what main.go's + // bootstrap() does for HGETALL entries. + reg.SetFirstSeenAt("sbx-bootstrap", startedAt) + + mkSweeper := func(now time.Time) *Sweeper { + return New(Options{ + Registry: reg, + Redis: store, + CubeMaster: master, + ProxyPush: push, + DefaultIdleTimeout: 5 * time.Minute, + BootstrapWarmup: 30 * time.Second, + StateLockTTL: 30 * time.Second, + Interval: time.Second, + StartedAt: startedAt, + Now: func() time.Time { return now }, + Log: zap.NewNop(), + }) + } + + // During warmup → skipped. + mkSweeper(startedAt).sweepOnce(context.Background()) + if len(master.calls) != 0 { + t.Fatalf("Pause must NOT fire on bootstrap entry during warmup: %v", master.calls) + } + + // 45s later, well past the 30s warmup → sweeper acts. + mkSweeper(startedAt.Add(45 * time.Second)).sweepOnce(context.Background()) + if len(master.calls) != 1 { + t.Fatalf("after warmup, sweeper should pause the idle bootstrap entry: %v", master.calls) + } +} + +func TestSweeper_RollsBackOnPauseFailure(t *testing.T) { + reg := registry.New() + store := newFakeStore() + master := &fakeMaster{failNext: true, failError: errors.New("master 500")} + push := newFakePush() + + now := time.Now() + seedEntry(t, reg, lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx-4", InstanceType: "cubebox", + AutoPause: true, TimeoutSeconds: 60, + }, now.Add(-10*time.Minute).UnixMilli()) + + s := newTestSweeper(reg, store, master, push, now) + s.sweepOnce(context.Background()) + + if got := store.state("sbx-4"); got != "" { + t.Fatalf("redis state should be cleared after rollback, got %q", got) + } + pushed := push.states("sbx-4") + if len(pushed) == 0 || pushed[len(pushed)-1] != "running" { + t.Fatalf("rollback should leave proxy at running, got %v", pushed) + } + _, failed := s.Stats() + if failed != 1 { + t.Fatalf("expected failed=1, got %d", failed) + } +} + +func TestSweeper_SkipsWhenLockHeldElsewhere(t *testing.T) { + reg := registry.New() + store := newFakeStore() + // Pre-seed the state map → AcquireState returns false (someone else owns). + store.states["sbx-5"] = "pausing" + + master := &fakeMaster{} + push := newFakePush() + + now := time.Now() + seedEntry(t, reg, lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx-5", InstanceType: "cubebox", + AutoPause: true, TimeoutSeconds: 1, + }, now.Add(-1*time.Hour).UnixMilli()) + + s := newTestSweeper(reg, store, master, push, now) + s.sweepOnce(context.Background()) + + if len(master.calls) != 0 { + t.Fatalf("Pause must not fire when lock is held: %v", master.calls) + } + triggered, _ := s.Stats() + if triggered != 0 { + t.Fatalf("triggered should be 0, got %d", triggered) + } +} + +func TestSweeper_FallsBackToCreatedAtWhenNoActivityRecorded(t *testing.T) { + reg := registry.New() + store := newFakeStore() + master := &fakeMaster{} + push := newFakePush() + + now := time.Now() + // No MergeLastActive call: LastActiveMs stays 0; CreatedAt is recent so + // the sandbox should NOT be paused. + reg.Upsert(lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx-recent", InstanceType: "cubebox", + AutoPause: true, TimeoutSeconds: 300, + CreatedAt: now.Add(-1 * time.Minute).UnixMilli(), + }) + // Pretend the entry was added long enough ago to clear the grace window. + at := now.Add(2 * time.Minute) + + s := newTestSweeper(reg, store, master, push, at) + s.sweepOnce(context.Background()) + + if len(master.calls) != 0 { + t.Fatalf("recent CreatedAt should keep sandbox alive: %v", master.calls) + } + + // Now CreatedAt is 10 minutes old → past timeout → expect pause. + reg.Upsert(lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx-old", InstanceType: "cubebox", + AutoPause: true, TimeoutSeconds: 300, + CreatedAt: now.Add(-10 * time.Minute).UnixMilli(), + }) + at2 := now.Add(2 * time.Minute) + s2 := newTestSweeper(reg, store, master, push, at2) + s2.sweepOnce(context.Background()) + + if len(master.calls) != 1 || master.calls[0] != "sbx-old" { + t.Fatalf("old CreatedAt must trigger pause: %v", master.calls) + } +} + +func TestSweeper_NotFoundEvictsRegistryEntry(t *testing.T) { + // CubeMaster returns "key not found" (RetCodeInvalidParamFormat) when the + // sandbox has been deleted out from under us. The sweeper must drop the + // entry from the registry, push a delete to CubeProxy, and NOT record + // the failure as a retry-able error. + reg := registry.New() + store := newFakeStore() + master := &fakeMaster{ + failNext: true, + failError: &cubemasterclient.APIError{ + RetCode: cubemasterclient.RetCodeInvalidParamFormat, + RetMsg: "key not found", + }, + } + push := newFakePush() + + now := time.Now() + seedEntry(t, reg, lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx-gone", InstanceType: "cubebox", + AutoPause: true, TimeoutSeconds: 60, + }, now.Add(-10*time.Minute).UnixMilli()) + + s := newTestSweeper(reg, store, master, push, now) + s.sweepOnce(context.Background()) + + if reg.Get("sbx-gone") != nil { + t.Fatal("registry entry should have been evicted") + } + if got := push.deletedIDs(); len(got) != 1 || got[0] != "sbx-gone" { + t.Fatalf("expected DeleteMeta(sbx-gone), got %v", got) + } + if got := store.state("sbx-gone"); got != "" { + t.Fatalf("redis state should be cleared, got %q", got) + } + _, failed := s.Stats() + if failed != 0 { + t.Fatalf("not-found is not a real failure; failed counter should be 0, got %d", failed) + } +} + +func TestSweeper_SkipsAlreadyPausedSandbox(t *testing.T) { + // Regression: when a previous sweep successfully paused the sandbox, + // Redis carries `state:="paused"`. Subsequent sweeps must NOT + // re-attempt — the sandbox is already at the desired terminal state + // and there is no resource to reclaim. Without this guard the sweeper + // hammers SETNX every Interval and issues a pointless RPC against + // CubeMaster every StateLockTTL seconds. + reg := registry.New() + store := newFakeStore() + store.states["sbx-zzz"] = "paused" // terminal marker from previous sweep + + master := &fakeMaster{} + push := newFakePush() + + now := time.Now() + seedEntry(t, reg, lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx-zzz", InstanceType: "cubebox", + AutoPause: true, TimeoutSeconds: 60, + }, now.Add(-1*time.Hour).UnixMilli()) + + s := newTestSweeper(reg, store, master, push, now) + s.sweepOnce(context.Background()) + + if len(master.calls) != 0 { + t.Fatalf("sweeper must NOT call Pause when state is already paused: %v", master.calls) + } + triggered, _ := s.Stats() + if triggered != 0 { + t.Fatalf("triggered should be 0, got %d", triggered) + } +} + +func TestSweeper_SkipsPausingSandbox(t *testing.T) { + // Same idea but for "pausing" — peer is mid-flight, don't pile on. + reg := registry.New() + store := newFakeStore() + store.states["sbx-mid"] = "pausing" + + master := &fakeMaster{} + push := newFakePush() + + now := time.Now() + seedEntry(t, reg, lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx-mid", InstanceType: "cubebox", + AutoPause: true, TimeoutSeconds: 60, + }, now.Add(-1*time.Hour).UnixMilli()) + + s := newTestSweeper(reg, store, master, push, now) + s.sweepOnce(context.Background()) + + if len(master.calls) != 0 { + t.Fatalf("sweeper must NOT call Pause when peer is mid-flight: %v", master.calls) + } +} + +func TestSweeper_AlreadyPausedReconcilesAsSuccess(t *testing.T) { + // CubeMaster returns "sandbox is already paused" (RetCodeTaskStateInvalid) + // when a peer sidecar already paused this sandbox. The sweeper should NOT + // roll back: it should write `paused` state to Redis + push to CubeProxy + // just like a fresh successful pause, so all callers converge on the + // same view. + reg := registry.New() + store := newFakeStore() + master := &fakeMaster{ + failNext: true, + failError: &cubemasterclient.APIError{ + RetCode: cubemasterclient.RetCodeTaskStateInvalid, + RetMsg: "sandbox is already paused", + }, + } + push := newFakePush() + + now := time.Now() + seedEntry(t, reg, lifecycle.SandboxLifecycleMeta{ + SandboxID: "sbx-already", InstanceType: "cubebox", + AutoPause: true, TimeoutSeconds: 60, + }, now.Add(-10*time.Minute).UnixMilli()) + + s := newTestSweeper(reg, store, master, push, now) + s.sweepOnce(context.Background()) + + if got := store.state("sbx-already"); got != "paused" { + t.Fatalf("expected redis state=paused (reconciliation), got %q", got) + } + pushed := push.states("sbx-already") + if len(pushed) == 0 || pushed[len(pushed)-1] != "paused" { + t.Fatalf("expected last push state=paused, got %v", pushed) + } + triggered, failed := s.Stats() + if triggered != 1 || failed != 0 { + t.Fatalf("already-paused should count as triggered, not failed: triggered=%d failed=%d", + triggered, failed) + } +} diff --git a/CubeProxy/start.sh b/CubeProxy/start.sh index 9f66d10b5..e6a80fdce 100755 --- a/CubeProxy/start.sh +++ b/CubeProxy/start.sh @@ -1,4 +1,55 @@ #!/bin/bash +# CubeProxy container entrypoint. +# +# Layout: +# - Foreground: openresty/nginx (PID 1's main duty after exec) +# - Background: cube-proxy-sidecar, lifecycle coordination loop +# - Background: crond, log rotation +# +# The sidecar binary is shipped inside this image rather than as a separate +# container so the lifecycle (auto-pause / auto-resume) feature is always +# co-resident with nginx. The binary is REQUIRED — if it is missing or +# unreadable we abort the entrypoint instead of starting a half-functional +# container that quietly drops the auto-pause feature. The Dockerfile +# performs the same sanity checks at build time; this is the runtime +# belt-and-braces. + +set -u + +SIDECAR_BIN="${SIDECAR_BIN:-/usr/local/openresty/nginx/sbin/cube-proxy-sidecar}" +SIDECAR_LOG="${SIDECAR_LOG:-/data/log/cube-proxy/sidecar.log}" + +start_sidecar() { + if [[ ! -x "${SIDECAR_BIN}" ]]; then + echo "$(date -Iseconds) FATAL: cube-proxy-sidecar binary missing or not executable at ${SIDECAR_BIN}" >&2 + echo "$(date -Iseconds) rebuild the cube-proxy image (CubeProxy/Makefile prebuild-sidecar)" >&2 + return 1 + fi + + # Loop in the background so a sidecar crash auto-restarts without taking + # nginx down with it. Exponential-ish backoff bounded at 30s. + ( + backoff=1 + while true; do + "${SIDECAR_BIN}" >>"${SIDECAR_LOG}" 2>&1 & + sidecar_pid=$! + wait "${sidecar_pid}" + rc=$? + echo "$(date -Iseconds) cube-proxy-sidecar exited rc=${rc}; restarting in ${backoff}s" >>"${SIDECAR_LOG}" + sleep "${backoff}" + if [[ "${backoff}" -lt 30 ]]; then + backoff=$((backoff * 2)) + [[ "${backoff}" -gt 30 ]] && backoff=30 + fi + done + ) & + echo "$(date -Iseconds) cube-proxy-sidecar supervisor started (logs: ${SIDECAR_LOG})" >&2 +} + +mkdir -p "$(dirname "${SIDECAR_LOG}")" /usr/sbin/crond -/usr/local/openresty/nginx/sbin/nginx +# Abort the entrypoint if the sidecar can't be brought up — nginx alone +# would silently mishandle paused sandboxes (returning 503 forever). +start_sidecar || exit 1 +exec /usr/local/openresty/nginx/sbin/nginx diff --git a/Makefile b/Makefile index be465e0aa..ef6baaf1b 100644 --- a/Makefile +++ b/Makefile @@ -70,6 +70,7 @@ help: @printf " cubecow-smoke Build cubecow smoke test CLI in Docker\n" @printf " cubecow-test-native Build SDK artifacts and run native tests in Docker\n" @printf " network-agent Build network-agent in Docker\n" + @printf " cube-proxy-sidecar Build cube-proxy-sidecar (developer-only; not in 'all')\n" @printf " agent Build cube-agent in Docker\n" @printf " cubeapi Build CubeAPI (cube-api) in Docker\n" @printf " cube-api Alias of cubeapi\n" @@ -206,6 +207,11 @@ network-agent: builder-image @mkdir -p "$(OUTPUT_DIR)" $(MAKE) builder-run BUILDER_CMD='mkdir -p /workspace/_output/bin && cd /workspace/network-agent && make proto && make build && cp bin/network-agent /workspace/_output/bin/network-agent' +.PHONY: cube-proxy-sidecar +cube-proxy-sidecar: builder-image + @mkdir -p "$(OUTPUT_DIR)" + $(MAKE) builder-run BUILDER_CMD='mkdir -p /workspace/_output/bin && cd /workspace/CubeProxy/sidecar && go mod download && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -tags "netgo osusergo" -ldflags "-s -w" -o /workspace/_output/bin/cube-proxy-sidecar ./cmd/sidecar' + .PHONY: agent agent: builder-image @mkdir -p "$(OUTPUT_DIR)" diff --git a/deploy/one-click/build-release-bundle.sh b/deploy/one-click/build-release-bundle.sh index c48ed28ad..6e6cfecc0 100755 --- a/deploy/one-click/build-release-bundle.sh +++ b/deploy/one-click/build-release-bundle.sh @@ -48,6 +48,7 @@ CUBELET_BUILD_MODE="${ONE_CLICK_CUBELET_BUILD_MODE:-local}" API_BUILD_MODE="${ONE_CLICK_CUBE_API_BUILD_MODE:-local}" NETWORK_AGENT_BUILD_MODE="${ONE_CLICK_NETWORK_AGENT_BUILD_MODE:-local}" CUBEVSMAPDUMP_BUILD_MODE="${ONE_CLICK_CUBEVSMAPDUMP_BUILD_MODE:-local}" +CUBE_PROXY_SIDECAR_BUILD_MODE="${ONE_CLICK_CUBE_PROXY_SIDECAR_BUILD_MODE:-local}" CUBEMASTER_BIN_OVERRIDE="${ONE_CLICK_CUBEMASTER_BIN:-}" CUBEMASTERCLI_BIN_OVERRIDE="${ONE_CLICK_CUBEMASTERCLI_BIN:-}" @@ -56,6 +57,7 @@ CUBECLI_BIN_OVERRIDE="${ONE_CLICK_CUBECLI_BIN:-}" API_BIN_OVERRIDE="${ONE_CLICK_CUBE_API_BIN:-}" NETWORK_AGENT_BIN_OVERRIDE="${ONE_CLICK_NETWORK_AGENT_BIN:-}" CUBEVSMAPDUMP_BIN_OVERRIDE="${ONE_CLICK_CUBEVSMAPDUMP_BIN:-}" +CUBE_PROXY_SIDECAR_BIN_OVERRIDE="${ONE_CLICK_CUBE_PROXY_SIDECAR_BIN:-}" go_version_ldflags() { local version_pkg="$1" @@ -484,6 +486,33 @@ build_or_copy_go_binary \ "cubevsmapdump" "${CUBEVSMAPDUMP_BIN_OVERRIDE}" \ "${ROOT_DIR}/CubeNet/cubevs" "${CUBEVSMAPDUMP_BUILD_MODE}" \ "${CORE_BIN_DIR}/cubevsmapdump" ./cmd/cubevsmapdump +# Auto-pause sidecar ships embedded inside the cube-proxy container image +# (CubeProxy/Dockerfile COPY bin/cube-proxy-sidecar). The cube-proxy image +# is openresty:alpine-fat (musl libc), so the binary MUST be statically +# linked — a default `go build` produces a glibc-linked binary that fails +# at exec with rc=127 / "required file not found" on musl. Force +# CGO_ENABLED=0 + -tags netgo,osusergo to get a pure-Go static binary. +# Skip the generic build_or_copy_go_binary helper (which doesn't expose +# these knobs) and call build directly here. +if [[ -n "${CUBE_PROXY_SIDECAR_BIN_OVERRIDE}" ]]; then + log "using prebuilt cube-proxy-sidecar: ${CUBE_PROXY_SIDECAR_BIN_OVERRIDE}" + copy_file "${CUBE_PROXY_SIDECAR_BIN_OVERRIDE}" "${CORE_BIN_DIR}/cube-proxy-sidecar" +else + log "building cube-proxy-sidecar (static, CGO_ENABLED=0)" + case "${CUBE_PROXY_SIDECAR_BUILD_MODE}" in + local) + require_cmd go + (cd "${ROOT_DIR}/CubeProxy/sidecar" && \ + go mod download && \ + CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \ + go build -trimpath -tags 'netgo osusergo' -ldflags '-s -w' \ + -o "${CORE_BIN_DIR}/cube-proxy-sidecar" ./cmd/sidecar) >&2 + ;; + *) + die "unsupported cube-proxy-sidecar build mode: ${CUBE_PROXY_SIDECAR_BUILD_MODE}" + ;; + esac +fi mkdir -p \ "${PACKAGE_ROOT}/network-agent/bin" \ @@ -537,6 +566,20 @@ copy_dir_contents "${CUBE_WEBUI_TEMPLATE_DIR}" "${PACKAGE_ROOT}/webui" copy_dir_contents "${CUBE_SYSTEMD_TEMPLATE_DIR}" "${PACKAGE_ROOT}/systemd" copy_dir_contents "${CUBE_PROXY_SOURCE_DIR}" "${PACKAGE_ROOT}/cubeproxy/build-context" rm -f "${PACKAGE_ROOT}/cubeproxy/build-context/Makefile" +# The build-context only needs the prebuilt sidecar binary, not the Go +# source: the runtime image uses `COPY bin/cube-proxy-sidecar` (no Go +# toolchain in the runtime stage). Strip the source tree to keep the +# release bundle lean. +rm -rf "${PACKAGE_ROOT}/cubeproxy/build-context/sidecar" +# Drop the prebuilt sidecar binary into the build context so the Dockerfile's +# COPY bin/cube-proxy-sidecar step has something to grab. Path matches the +# in-tree CubeProxy/Makefile prebuild-sidecar layout (CubeProxy/bin/) so the +# Dockerfile is identical for both build flows. +mkdir -p "${PACKAGE_ROOT}/cubeproxy/build-context/bin" +copy_file \ + "${CORE_BIN_DIR}/cube-proxy-sidecar" \ + "${PACKAGE_ROOT}/cubeproxy/build-context/bin/cube-proxy-sidecar" +chmod +x "${PACKAGE_ROOT}/cubeproxy/build-context/bin/cube-proxy-sidecar" generate_cube_proxy_nginx_template \ "${CUBE_PROXY_SOURCE_DIR}/nginx.conf" \ "${PACKAGE_ROOT}/cubeproxy/nginx.conf.template" diff --git a/deploy/one-click/cubeproxy/docker-compose.yaml.template b/deploy/one-click/cubeproxy/docker-compose.yaml.template index 9b5ab8d66..9eb3c4200 100644 --- a/deploy/one-click/cubeproxy/docker-compose.yaml.template +++ b/deploy/one-click/cubeproxy/docker-compose.yaml.template @@ -8,6 +8,22 @@ services: dockerfile: Dockerfile restart: unless-stopped network_mode: host + environment: + # cube-proxy-sidecar runs alongside nginx inside this container (started + # by /usr/local/openresty/nginx/sbin/start.sh). It needs to reach the + # same Redis CubeMaster writes lifecycle events to, plus the loopback + # admin server on the co-resident nginx. + CUBE_SIDECAR_REDIS_ADDR: "__CUBE_SIDECAR_REDIS_ADDR__" + CUBE_SIDECAR_REDIS_PASSWORD: "__CUBE_SIDECAR_REDIS_PASSWORD__" + CUBE_SIDECAR_REDIS_DB: "__CUBE_SIDECAR_REDIS_DB__" + CUBE_SIDECAR_PROXY_ADMIN_URLS: "__CUBE_SIDECAR_PROXY_ADMIN_URLS__" + CUBE_SIDECAR_CUBEMASTER_URL: "__CUBE_SIDECAR_CUBEMASTER_URL__" + CUBE_SIDECAR_LISTEN_ADDR: "__CUBE_SIDECAR_LISTEN_ADDR__" + CUBE_SIDECAR_DEFAULT_IDLE_TIMEOUT: "__CUBE_SIDECAR_DEFAULT_IDLE_TIMEOUT__" + # Shared secret sent as X-Cube-Admin-Token on every /admin/* push the + # sidecar makes to its co-resident nginx. Must match the value Lua + # global.conf substitutes into $cube_admin_token. + CUBE_SIDECAR_ADMIN_TOKEN: "__CUBE_ADMIN_TOKEN__" volumes: - __CUBE_PROXY_CERT_DIR__:/usr/local/openresty/nginx/certs:ro - __CUBE_PROXY_GLOBAL_CONF__:/usr/local/openresty/nginx/conf/global/global.conf:ro diff --git a/deploy/one-click/cubeproxy/global.conf.template b/deploy/one-click/cubeproxy/global.conf.template index 5189da656..1a74342d3 100644 --- a/deploy/one-click/cubeproxy/global.conf.template +++ b/deploy/one-click/cubeproxy/global.conf.template @@ -5,3 +5,11 @@ set $redis_index 0; set $timeout_min 500; set $timeout_max 700; set $cube_proxy_host_ip "__CUBE_PROXY_HOST_IP__"; + +# Auto-pause / auto-resume wiring. $cube_sidecar_addr points at the +# co-resident cube-proxy-sidecar (loopback by default). $cube_admin_token +# is an optional shared secret enforced by the sidecar admin server; leave +# empty to disable auth (safe because the admin server listens on 127.0.0.1 +# only). +set $cube_sidecar_addr "__CUBE_SIDECAR_NGX_ADDR__"; +set $cube_admin_token "__CUBE_ADMIN_TOKEN__"; diff --git a/deploy/one-click/scripts/one-click/up-cube-proxy.sh b/deploy/one-click/scripts/one-click/up-cube-proxy.sh index f6e906184..037df15a1 100644 --- a/deploy/one-click/scripts/one-click/up-cube-proxy.sh +++ b/deploy/one-click/scripts/one-click/up-cube-proxy.sh @@ -34,6 +34,25 @@ CUBE_PROXY_HTTPS_PORT="${CUBE_PROXY_HTTPS_PORT:-443}" CUBE_PROXY_HTTP_PORT="${CUBE_PROXY_HTTP_PORT:-80}" CUBE_PROXY_SSL_CERT="${CUBE_PROXY_SSL_CERT:-cube.app+3.pem}" CUBE_PROXY_SSL_KEY="${CUBE_PROXY_SSL_KEY:-cube.app+3-key.pem}" +# Auto-pause sidecar runs in the same container as nginx; it is always +# brought up alongside it and consumes the same Redis CubeMaster writes +# lifecycle events to. +CUBE_SIDECAR_REDIS_ADDR="${CUBE_SIDECAR_REDIS_ADDR:-${CUBE_PROXY_REDIS_IP}:${CUBE_PROXY_REDIS_PORT}}" +CUBE_SIDECAR_REDIS_PASSWORD="${CUBE_SIDECAR_REDIS_PASSWORD:-${CUBE_PROXY_REDIS_PASSWORD}}" +CUBE_SIDECAR_REDIS_DB="${CUBE_SIDECAR_REDIS_DB:-0}" +CUBE_SIDECAR_PROXY_ADMIN_URLS="${CUBE_SIDECAR_PROXY_ADMIN_URLS:-http://127.0.0.1:8082}" +# CubeMaster's HTTP listener defaults to :8089 (see configs/single-node/ +# cubemaster.yaml `common.http_port`). Override CUBE_SIDECAR_CUBEMASTER_URL +# at deploy time if your CubeMaster is on a different host:port. +CUBE_SIDECAR_CUBEMASTER_URL="${CUBE_SIDECAR_CUBEMASTER_URL:-http://${CUBE_SANDBOX_NODE_IP}:8089}" +CUBE_SIDECAR_LISTEN_ADDR="${CUBE_SIDECAR_LISTEN_ADDR:-127.0.0.1:8083}" +CUBE_SIDECAR_DEFAULT_IDLE_TIMEOUT="${CUBE_SIDECAR_DEFAULT_IDLE_TIMEOUT:-5m}" +# Lua $cube_sidecar_addr is consumed by the /_sidecar_resume internal proxy. +# Default to the same loopback address the sidecar binds to. +CUBE_SIDECAR_NGX_ADDR="${CUBE_SIDECAR_NGX_ADDR:-${CUBE_SIDECAR_LISTEN_ADDR}}" +# Optional shared secret accepted by CubeProxy's /admin/* endpoints. Empty +# means "no auth"; safe because the admin server listens on 127.0.0.1 only. +CUBE_ADMIN_TOKEN="${CUBE_ADMIN_TOKEN:-}" COMPOSE_DETACH="${ONE_CLICK_COMPOSE_DETACH:-1}" MKCERT_BUNDLED_BIN="${TOOLBOX_ROOT}/support/bin/mkcert" PREPARE_ONLY="${ONE_CLICK_PREPARE_ONLY:-0}" @@ -94,7 +113,9 @@ render_template_atomic \ -e "s/__CUBE_PROXY_REDIS_IP__/$(escape_sed "${CUBE_PROXY_REDIS_IP}")/g" \ -e "s/__CUBE_PROXY_REDIS_PORT__/$(escape_sed "${CUBE_PROXY_REDIS_PORT}")/g" \ -e "s/__CUBE_PROXY_REDIS_PASSWORD__/$(escape_sed "${CUBE_PROXY_REDIS_PASSWORD}")/g" \ - -e "s/__CUBE_PROXY_HOST_IP__/$(escape_sed "${CUBE_SANDBOX_NODE_IP}")/g" + -e "s/__CUBE_PROXY_HOST_IP__/$(escape_sed "${CUBE_SANDBOX_NODE_IP}")/g" \ + -e "s#__CUBE_SIDECAR_NGX_ADDR__#$(escape_sed "${CUBE_SIDECAR_NGX_ADDR}" '#')#g" \ + -e "s#__CUBE_ADMIN_TOKEN__#$(escape_sed "${CUBE_ADMIN_TOKEN}" '#')#g" NGINX_TEMPLATE="${PROXY_DIR}/nginx.conf.template" NGINX_CONF="${PROXY_DIR}/nginx.conf" @@ -115,7 +136,15 @@ render_template_atomic \ -e "s#__CUBE_PROXY_BUILD_CONTEXT__#$(escape_sed "${BUILD_CONTEXT_DIR}" '#')#g" \ -e "s#__CUBE_PROXY_CERT_DIR__#$(escape_sed "${CERT_DIR}" '#')#g" \ -e "s#__CUBE_PROXY_GLOBAL_CONF__#$(escape_sed "${GLOBAL_CONF}" '#')#g" \ - -e "s#__CUBE_PROXY_NGINX_CONF__#$(escape_sed "${NGINX_CONF}" '#')#g" + -e "s#__CUBE_PROXY_NGINX_CONF__#$(escape_sed "${NGINX_CONF}" '#')#g" \ + -e "s#__CUBE_SIDECAR_REDIS_ADDR__#$(escape_sed "${CUBE_SIDECAR_REDIS_ADDR}" '#')#g" \ + -e "s#__CUBE_SIDECAR_REDIS_PASSWORD__#$(escape_sed "${CUBE_SIDECAR_REDIS_PASSWORD}" '#')#g" \ + -e "s#__CUBE_SIDECAR_REDIS_DB__#$(escape_sed "${CUBE_SIDECAR_REDIS_DB}" '#')#g" \ + -e "s#__CUBE_SIDECAR_PROXY_ADMIN_URLS__#$(escape_sed "${CUBE_SIDECAR_PROXY_ADMIN_URLS}" '#')#g" \ + -e "s#__CUBE_SIDECAR_CUBEMASTER_URL__#$(escape_sed "${CUBE_SIDECAR_CUBEMASTER_URL}" '#')#g" \ + -e "s#__CUBE_SIDECAR_LISTEN_ADDR__#$(escape_sed "${CUBE_SIDECAR_LISTEN_ADDR}" '#')#g" \ + -e "s#__CUBE_SIDECAR_DEFAULT_IDLE_TIMEOUT__#$(escape_sed "${CUBE_SIDECAR_DEFAULT_IDLE_TIMEOUT}" '#')#g" \ + -e "s#__CUBE_ADMIN_TOKEN__#$(escape_sed "${CUBE_ADMIN_TOKEN}" '#')#g" if [[ "${PREPARE_ONLY}" == "1" ]]; then log "cube proxy runtime files prepared under ${PROXY_DIR}" diff --git a/docs/.vitepress/config.mjs b/docs/.vitepress/config.mjs index 0d5d2b114..f4b19dbf9 100644 --- a/docs/.vitepress/config.mjs +++ b/docs/.vitepress/config.mjs @@ -119,6 +119,7 @@ export default withMermaid(defineConfig({ { text: 'Core Concepts', items: [ + { text: 'Sandbox Lifecycle', link: '/guide/lifecycle' }, { text: 'Templates Overview', link: '/guide/templates' }, { text: 'Digital Assistant', link: '/guide/digital-assistant' }, { text: 'Performance Benchmark', link: '/guide/performance-benchmark' } @@ -225,6 +226,7 @@ export default withMermaid(defineConfig({ { text: '核心概念', items: [ + { text: '沙箱生命周期', link: '/zh/guide/lifecycle' }, { text: '模板概览', link: '/zh/guide/templates' }, { text: '数字助手', link: '/zh/guide/digital-assistant' }, { text: '性能测试', link: '/zh/guide/performance-benchmark' } diff --git a/docs/guide/lifecycle.md b/docs/guide/lifecycle.md new file mode 100644 index 000000000..1052c49af --- /dev/null +++ b/docs/guide/lifecycle.md @@ -0,0 +1,167 @@ +# Sandbox Lifecycle + +A sandbox is the core runtime unit of Cube-Sandbox. This page covers a sandbox's full lifecycle — from creation to teardown — and how to let the platform manage it automatically to save resources. + +> The SDK shape mirrors [e2b](https://e2b.dev/docs/sandbox) so existing e2b code can port with minimal changes. + +## State Model + +A sandbox is always in exactly one of these states: + +| State | Meaning | +|--------------|------------------------------------------------------------------------------------------------| +| `running` | Active. Real CPU/memory in use. Accepts requests and executes code. | +| `pausing` | Platform is taking the VM snapshot (transient). | +| `paused` | Snapshot persisted to disk. **Zero** CPU/memory cost. Full state preserved. | +| `resuming` | Platform is restoring the snapshot (transient). | +| `terminated` | Killed (`kill()`) or reaped after `on_timeout="kill"`. Cannot be brought back. | + +Two settings drive transitions: + +- **`timeout`**: how many seconds of inactivity trigger "timeout" (defaults to a fixed value in SDK Config, e.g. 300s). +- **`on_timeout`**: what happens at timeout — `"kill"` (default; destroy) or `"pause"` (snapshot for later). + +``` + ┌──────────────────────────────────────┐ + │ │ + create() ┌────▼────┐ timeout & on_timeout=pause ┌─────────┐ + ───────────────►│ running │ ──────────────────────────────►│ paused │ + │ │◄──────── connect() or │ │ + └─┬─────┬─┘ auto_resume-triggered req └────┬────┘ + │ │ │ + kill() │ │ timeout & on_timeout=kill │ kill() + ────────────┘ └─────────────────┐ │ + ▼ ▼ + ┌────────────┐ + │ terminated │ + └────────────┘ +``` + +## Create + +```python +from cubesandbox import Sandbox + +# Create a sandbox that auto-destroys after 60 seconds of idle. +# (Default on_timeout is "kill".) +sandbox = Sandbox.create( + template="", + timeout=60, # seconds +) + +print(sandbox.sandbox_id) +``` + +Key parameters of `Sandbox.create()`: + +| Parameter | Description | +|-------------------------|----------------------------------------------------------------------------------------------| +| `template` | Template ID used to boot the sandbox; defaults to env var `CUBE_TEMPLATE_ID`. | +| `timeout` | Idle timeout in **seconds**. (Note: e2b's `timeoutMs` is milliseconds; Cube uses seconds.) | +| `lifecycle` | Lifecycle policy — see [Platform-managed auto-pause / auto-resume](#platform-managed-auto-pause--auto-resume) below. | +| `metadata` | Arbitrary key/value pairs stored on the sandbox; readable from the list / detail endpoints. | +| `env_vars` | Environment variables injected into the sandbox process. | +| `allow_internet_access` | Whether outbound internet is allowed; `network` provides finer-grained egress control. | + +> Cube doesn't impose hard wall-clock ceilings (24h Pro / 1h Base) the way hosted e2b does. The idle `timeout` is still required — it prevents stranded sandboxes from holding resources indefinitely. + +## Inspect a Running Sandbox + +```python +info = sandbox.get_info() +print(info) +# { +# "sandboxID": "iiny0783cype8gmoawzmx-ce30bc46", +# "templateID": "rki5dems9wqfm4r03t7g", +# "state": "running", +# "startedAt": "2026-06-17T12:34:56Z", +# "endAt": "2026-06-17T12:39:56Z", +# "metadata": {...} +# } +``` + +`endAt` is the projected next-timeout instant given the current `timeout`. It is refreshed every time the sandbox receives a request (or when you call `set_timeout`, when available). + +## List Running Sandboxes + +```python +for sb in Sandbox.list(): + print(sb["sandboxID"], sb["state"]) +``` + +## Explicit Shutdown + +```python +sandbox.kill() +``` + +`kill()` is **irreversible**: unlike pause, a killed sandbox cannot be brought back, even when `lifecycle.on_timeout="pause"` was set — `kill()` always wins and discards the snapshot. + +## Explicit Pause / Resume + +```python +sandbox.pause() # snapshot manually, free CPU/memory +# ... time passes ... +sandbox.connect() # restore from snapshot +sandbox.run_code("print('back!')") # carry on as if never paused +``` + +See [`examples/code-sandbox-quickstart/pause.py`](https://github.com/tencentcloud/CubeSandbox/blob/master/examples/code-sandbox-quickstart/pause.py) for a full demo. + +## Platform-managed Auto-pause / Auto-resume + +Most agent workloads aren't continuously busy: the user types code → the model thinks → the sandbox executes → it sits idle until the next turn. Auto-pausing during the idle stretch and **transparently resuming** on the next request can dramatically cut resource cost. + +Cube exposes the exact same [`lifecycle`](https://e2b.dev/docs/sandbox/auto-resume) shape e2b uses: + +```python +sandbox = Sandbox.create( + template="", + timeout=300, # 5 min of idle triggers on_timeout + lifecycle={ + "on_timeout": "pause", # at timeout → pause (instead of kill) + "auto_resume": True, # next request after pause → resume + }, +) +``` + +### Behaviour + +- **`on_timeout="pause"`**: after `timeout` seconds idle, the platform schedules a pause. State flips to `paused`, the VM memory is frozen to the snapshot store. +- **`auto_resume=True`**: when any request next arrives for a `paused` sandbox (HTTP, `run_code`, file I/O, …), the platform wakes it up before the request lands. Callers never see the pause; typical resume latency is sub-second to a few seconds. +- If `auto_resume=False` (or unset), the sandbox stays paused until you explicitly `Sandbox.connect(sandbox_id=...)`. Useful for "wait for the user" workflows. + +### Timeout reset on auto-resume + +Each successful auto-resume gives the sandbox a **fresh** `timeout` countdown (matching e2b semantics). The "resume → short use → idle out → pause again" loop can repeat indefinitely. + +### What counts as activity + +Any of these resets the idle clock: + +- SDK calls: `sandbox.run_code(...)`, `sandbox.commands.run(...)`, `sandbox.files.read(...)` / `write(...)`. +- Direct HTTP traffic to a service inside the sandbox (e.g. via the URL returned by `getHost()`). + +Sandboxes that don't opt in (no `lifecycle` argument) keep the original behaviour: idle timeout → destroy. + +### End-to-end example + +[`examples/code-sandbox-quickstart/auto-resume.py`](https://github.com/tencentcloud/CubeSandbox/blob/master/examples/code-sandbox-quickstart/auto-resume.py) is a TUI demo that creates a `lifecycle.on_timeout=pause` sandbox, idles past the timeout to trigger auto-pause, then issues a fresh request to trigger auto-resume — and verifies that both kernel memory and the filesystem are byte-identical across the cycle. + +```bash +export CUBE_TEMPLATE_ID= +python examples/code-sandbox-quickstart/auto-resume.py +``` + +## Operational Notes + +- **Pause fidelity**: CPU registers, process memory, TCP state (with no external peer), and filesystem mutations all survive the snapshot. Outbound sockets the sandbox itself opened are dropped on pause and must be reopened by the application after resume. +- **Cluster coordination**: auto-pause is driven by `cube-proxy-sidecar`, co-resident with each CubeProxy container. It consumes lifecycle events CubeMaster publishes via Redis stream and broadcasts state to every CubeProxy instance. Cross-replica races are resolved by Redis `SETNX` state locks so the same sandbox is never paused or resumed twice concurrently. +- **Failure mode**: when an auto-resume RPC fails, CubeProxy returns `503 + Retry-After` to the client immediately rather than hanging on a long timeout. +- **Diagnostics**: `/data/log/cube-proxy/sidecar.log` is the sidecar's runtime log. Look for `create event applied`, `auto-paused sandbox`, `auto-resumed sandbox`. + +## Next Steps + +- [Templates Overview](./templates.md) — sandboxes boot from templates; the template's build also shapes cold-start cost. +- [Quick Start](./quickstart.md) — the shortest path through "create sandbox → run code → tear down". +- Upstream references: [e2b · Sandbox lifecycle](https://e2b.dev/docs/sandbox), [e2b · Auto-resume](https://e2b.dev/docs/sandbox/auto-resume). diff --git a/docs/zh/guide/lifecycle.md b/docs/zh/guide/lifecycle.md new file mode 100644 index 000000000..50c3dbddd --- /dev/null +++ b/docs/zh/guide/lifecycle.md @@ -0,0 +1,166 @@ +# 沙箱生命周期 + +沙箱（Sandbox）是 Cube-Sandbox 的核心运行单元。本页介绍沙箱从创建到销毁的**完整生命周期**，以及如何让平台自动管理生命周期、降低成本。 + +> 本页 SDK 形态与 [e2b](https://e2b.dev/docs/sandbox) 保持一致，便于已有 e2b 用户直接迁移。 + +## 状态模型 + +一个沙箱在它的生命周期里会处于以下几种状态之一： + +| 状态 | 含义 | +|-------------|----------------------------------------------------------------------| +| `running` | 正在运行，CPU/内存被实际占用，可以接收请求与执行代码 | +| `pausing` | 平台正在暂停沙箱（保存 VM 快照中），瞬时态 | +| `paused` | 沙箱已暂停，VM 内存已落盘为快照，**不消耗** CPU 与内存，状态完整保留 | +| `resuming` | 平台正在从快照恢复沙箱，瞬时态 | +| `terminated`| 沙箱被显式销毁（`kill`）或因 `on_timeout="kill"` 超时被回收，无法恢复 | + +状态转换主要由两个变量驱动： + +- **`timeout`**：空闲多久后触发"超时"（默认在 SDK Config 里给一个固定值，比如 300 秒）。 +- **`on_timeout`**：超时之后做什么 —— `"kill"`（默认，直接销毁）或 `"pause"`（暂停以备恢复）。 + +``` + ┌──────────────────────────────────────┐ + │ │ + create() ┌────▼────┐ timeout & on_timeout=pause ┌─────────┐ + ───────────────►│ running │ ──────────────────────────────►│ paused │ + │ │◄──────── connect() 或 │ │ + └─┬─────┬─┘ auto_resume 触发的请求 └────┬────┘ + │ │ │ + kill() │ │ timeout & on_timeout=kill │ kill() + ────────────┘ └─────────────────┐ │ + ▼ ▼ + ┌────────────┐ + │ terminated │ + └────────────┘ +``` + +## 创建沙箱 + +```python +from cubesandbox import Sandbox + +# 创建沙箱，空闲 60 秒后自动销毁（默认 on_timeout="kill"） +sandbox = Sandbox.create( + template="", + timeout=60, # 单位：秒 +) + +print(sandbox.sandbox_id) +``` + +`Sandbox.create()` 关键参数： + +| 参数 | 说明 | +|-------------------------|----------------------------------------------------------------------------| +| `template` | 模板 ID，沙箱基于它启动；缺省读环境变量 `CUBE_TEMPLATE_ID` | +| `timeout` | 空闲超时，**秒**（注意：e2b 的 `timeoutMs` 是毫秒，Cube 是秒） | +| `lifecycle` | 生命周期策略，详见下文 "[平台自动暂停 / 自动恢复](#平台自动暂停-自动恢复)" | +| `metadata` | 任意键值对，写入沙箱元数据，可在列表 / 详情接口中读出 | +| `env_vars` | 注入沙箱进程的环境变量 | +| `allow_internet_access` | 是否允许出公网；`network` 提供更细粒度的出站策略 | + +> Cube 的最大单次运行时长不像托管 e2b 那样有严格的 24h/1h 平台上限——但 idle `timeout` 仍然是必需的，它防止意外遗漏的沙箱长期占用资源。 + +## 查询沙箱信息 + +```python +info = sandbox.get_info() +print(info) +# { +# "sandboxID": "iiny0783cype8gmoawzmx-ce30bc46", +# "templateID": "rki5dems9wqfm4r03t7g", +# "state": "running", +# "startedAt": "2026-06-17T12:34:56Z", +# "endAt": "2026-06-17T12:39:56Z", +# "metadata": {...} +# } +``` + +`endAt` 表示按当前 `timeout` 估算的下一次超时时间。每次接收到新请求或调用 `set_timeout`（若有），`endAt` 会被刷新。 + +## 列出运行中的沙箱 + +```python +for sb in Sandbox.list(): + print(sb["sandboxID"], sb["state"]) +``` + +## 显式销毁 + +```python +sandbox.kill() +``` + +`kill()` 是不可逆的：与暂停不同，被 kill 的沙箱**不能**恢复。即便 `lifecycle.on_timeout="pause"`，调用 `kill()` 仍然立即终止并丢弃快照。 + +## 显式暂停 / 恢复 + +```python +sandbox.pause() # 主动保存快照，释放 CPU/内存 +# ... 一段时间过去 ... +sandbox.connect() # 从快照恢复 +sandbox.run_code("print('back!')") # 像没暂停过一样继续用 +``` + +可参考示例：[`examples/code-sandbox-quickstart/pause.py`](https://github.com/tencentcloud/CubeSandbox/blob/master/examples/code-sandbox-quickstart/pause.py)。 + +## 平台自动暂停 / 自动恢复 + +很多 Agent 工作负载并不持续繁忙：用户敲一段代码 → 模型推理 → 沙箱执行 → 等待下一轮交互。在等待期间让沙箱**自动暂停**，下次请求来时再**自动恢复**，可以显著降低资源占用。 + +Cube 提供与 e2b [`lifecycle`](https://e2b.dev/docs/sandbox/auto-resume) 完全一致的配置形态： + +```python +sandbox = Sandbox.create( + template="", + timeout=300, # 5 分钟空闲后触发 on_timeout + lifecycle={ + "on_timeout": "pause", # 空闲超时后 → 暂停（而不是销毁） + "auto_resume": True, # 暂停后下一次请求 → 透明恢复 + }, +) +``` + +### 行为说明 + +- **`on_timeout="pause"`**：沙箱空闲 `timeout` 秒后，平台调度暂停流程，`state` 变为 `paused`，VM 内存被冷藏到快照存储。 +- **`auto_resume=True`**：当再有任何请求路由到这个 `paused` 沙箱（HTTP 请求、`run_code`、文件读写等），平台自动唤醒它，调用方**无需**显式 `connect()`；典型恢复时间在亚秒级到秒级。 +- 如果 `auto_resume=False`（或省略），沙箱暂停后必须显式 `Sandbox.connect(sandbox_id=...)` 才能再用 —— 适合"等用户决定"的场景。 + +### 自动恢复后的 timeout 重置 + +每次自动恢复成功后，沙箱获得一个**全新的 `timeout` 计时窗口**（与 e2b 同样语义），所以"恢复 → 短暂使用 → 再次空闲超时 → 再次暂停"的循环可以无缝持续。 + +### 何时算"活跃" + +下列动作都会重置 idle 计时： + +- 通过 SDK 调用：`sandbox.run_code(...)`、`sandbox.commands.run(...)`、`sandbox.files.read(...)` / `write(...)`。 +- 通过 HTTP 直连沙箱内的服务（例如 `getHost()` 返回的 URL）。 + +未配置 `auto_pause` / 不传 `lifecycle` 的沙箱完全保留旧行为：空闲超时直接销毁。 + +### 端到端示例 + +[`examples/code-sandbox-quickstart/auto-resume.py`](https://github.com/tencentcloud/CubeSandbox/blob/master/examples/code-sandbox-quickstart/auto-resume.py) 是一个 TUI 演示：创建带 `lifecycle.on_timeout=pause` 的沙箱、空闲触发自动暂停、再发请求触发自动恢复，最终对比"内核内存 + 文件系统"两层状态，验证全状态保留。 + +```bash +export CUBE_TEMPLATE_ID= +python examples/code-sandbox-quickstart/auto-resume.py +``` + +## 设计与运维要点 + +- **暂停的状态保真度**：CPU 寄存器、进程内存、TCP 连接（无外部对端）、文件系统改动都会随快照保留；面向外部的连接（如 sandbox 主动建立的 outbound socket）会在暂停时断开，恢复后由应用层自行重连。 +- **集群一致性**：自动暂停由部署在 CubeProxy 容器内的 `cube-proxy-sidecar` 协调；它消费 CubeMaster 通过 Redis stream 发布的生命周期事件，对所有 CubeProxy 实例广播状态。多副本环境下用 Redis SETNX 互斥锁确保同一沙箱不会被并发暂停或恢复。 +- **失败回退**：自动恢复 RPC 失败时，CubeProxy 直接对客户端返回 503 + `Retry-After`，不会让用户卡在长超时上。 +- **故障排查**：`/data/log/cube-proxy/sidecar.log` 是 sidecar 的运行日志，关键事件包括 `create event applied`、`auto-paused sandbox`、`auto-resumed sandbox`。 + +## 下一步 + +- [模板概览](./templates.md) —— 沙箱基于模板启动，模板的构建过程也会影响首次冷启动开销。 +- [快速开始](./quickstart.md) —— 完整跑通"创建沙箱 → 执行代码 → 销毁"的最短路径。 +- 上游参考：[e2b · Sandbox lifecycle](https://e2b.dev/docs/sandbox)、[e2b · Auto-resume](https://e2b.dev/docs/sandbox/auto-resume)。 diff --git a/examples/code-sandbox-quickstart/README.md b/examples/code-sandbox-quickstart/README.md index f39b16ab8..d2b312d38 100644 --- a/examples/code-sandbox-quickstart/README.md +++ b/examples/code-sandbox-quickstart/README.md @@ -124,6 +124,7 @@ hello cube | `create.py` | `sandbox.get_info()` — retrieve sandbox metadata | | `read.py` | `sandbox.files.read()` — read a file from the sandbox filesystem | | `pause.py` | `sandbox.pause()` / `sandbox.connect()` — snapshot and restore | +| `auto_resume.py` | `lifecycle={"on_timeout": "pause", "auto_resume": True}` — let the platform pause idle sandboxes and resume them on the next request | | `network_no_internet.py` | `allow_internet_access=False` — fully air-gapped sandbox | | `network_allowlist.py` | `allow_out` — whitelist specific CIDRs, block everything else | | `network_denylist.py` | `deny_out` — block specific CIDRs, allow the rest | @@ -155,6 +156,26 @@ with Sandbox.create(template=template_id) as sandbox: print(sandbox.get_info()) ``` +### auto_resume.py — Auto Pause & Auto Resume + +Like `pause.py`, but the platform handles the pause/resume cycle on its own. +The `lifecycle` argument mirrors the e2b SDK +([reference](https://e2b.dev/docs/sandbox/auto-resume)) — set +`on_timeout="pause"` to opt into idle-timeout pausing and `auto_resume=True` +so the next request automatically wakes the sandbox up: + +```python +sandbox = Sandbox.create( + template=template_id, + timeout=30, # idle threshold the auto-pause sidecar uses + lifecycle={"on_timeout": "pause", "auto_resume": True}, +) +sandbox.run_code("print('first call')") +time.sleep(45) # exceeds the timeout — sidecar pauses the sandbox +sandbox.run_code("print('back from a transparent resume')") +sandbox.kill() +``` + ### Network Policies ```bash @@ -188,6 +209,7 @@ code-sandbox-quickstart/ ├── create.py # Create sandbox and inspect metadata ├── read.py # Read files from the sandbox filesystem ├── pause.py # Pause and resume a sandbox +├── auto_resume.py # Auto-pause / auto-resume on idle timeout ├── network_no_internet.py # Fully air-gapped sandbox ├── network_allowlist.py # Outbound CIDR allowlist ├── network_denylist.py # Outbound CIDR denylist diff --git a/examples/code-sandbox-quickstart/README_zh.md b/examples/code-sandbox-quickstart/README_zh.md index 448b6b3de..62030a2ca 100644 --- a/examples/code-sandbox-quickstart/README_zh.md +++ b/examples/code-sandbox-quickstart/README_zh.md @@ -119,6 +119,7 @@ hello cube | `create.py` | `sandbox.get_info()` — 获取沙箱元数据 | | `read.py` | `sandbox.files.read()` — 读取沙箱文件系统中的文件 | | `pause.py` | `sandbox.pause()` / `sandbox.connect()` — 快照与恢复 | +| `auto_resume.py` | `lifecycle={"on_timeout": "pause", "auto_resume": True}` — 平台在空闲超时后自动暂停沙箱，下一次请求自动恢复 | | `network_no_internet.py` | `allow_internet_access=False` — 完全断网沙箱 | | `network_allowlist.py` | `allow_out` — 白名单 CIDR，拦截其余所有出口 | | `network_denylist.py` | `deny_out` — 黑名单 CIDR，其余放行 | @@ -150,6 +151,25 @@ with Sandbox.create(template=template_id) as sandbox: print(sandbox.get_info()) ``` +### auto_resume.py — 自动暂停与自动恢复 + +与 `pause.py` 类似，但暂停/恢复完全交给平台自动管理。`lifecycle` 参数与 e2b SDK 对齐 +（参考 [e2b 文档](https://e2b.dev/docs/sandbox/auto-resume)）：`on_timeout="pause"` +表示空闲超时后由 sidecar 触发暂停；`auto_resume=True` 让下一次请求命中 +暂停沙箱时自动恢复： + +```python +sandbox = Sandbox.create( + template=template_id, + timeout=30, # auto-pause sidecar 用作空闲阈值 + lifecycle={"on_timeout": "pause", "auto_resume": True}, +) +sandbox.run_code("print('first call')") +time.sleep(45) # 超过空闲阈值，sidecar 暂停沙箱 +sandbox.run_code("print('back from a transparent resume')") +sandbox.kill() +``` + ### 网络策略 ```bash @@ -183,6 +203,7 @@ code-sandbox-quickstart/ ├── create.py # 创建沙箱并查看元数据 ├── read.py # 读取沙箱文件系统中的文件 ├── pause.py # 暂停与恢复沙箱 +├── auto_resume.py # 自动暂停 / 自动恢复（基于空闲超时） ├── network_no_internet.py # 完全断网沙箱 ├── network_allowlist.py # 出口 CIDR 白名单 ├── network_denylist.py # 出口 CIDR 黑名单 diff --git a/examples/code-sandbox-quickstart/auto-resume.py b/examples/code-sandbox-quickstart/auto-resume.py new file mode 100755 index 000000000..cdaa67afc --- /dev/null +++ b/examples/code-sandbox-quickstart/auto-resume.py @@ -0,0 +1,361 @@ +#!/bin/env python3 +# Copyright (c) 2026 Tencent Inc. +# SPDX-License-Identifier: Apache-2.0 +# +# Auto-pause / auto-resume TUI demo. +# +# Sister demo to pause.py — same five-step storyline, but the pause/resume +# transitions are driven by the *platform* (cube-proxy + lifecycle manager) +# rather than explicit pause()/connect() calls from the SDK. +# +# Lifecycle config mirrors the e2b SDK +# (https://e2b.dev/docs/sandbox/auto-resume) so existing e2b code ports +# with minimal changes. +# +# Run: +# export CUBE_API_URL=http://:3000 +# export CUBE_TEMPLATE_ID= +# python auto-resume.py [--dark] + +import argparse +import os +import time + +from cubesandbox import Sandbox +from rich import box +from rich.console import Console +from rich.panel import Panel +from rich.progress import BarColumn, Progress, TextColumn +from rich.syntax import Syntax +from rich.table import Table + +parser = argparse.ArgumentParser( + description="Cube Sandbox Auto-Pause / Auto-Resume TUI Demo" +) +parser.add_argument( + "--dark", + action="store_true", + help="Use dark-terminal color theme (default: light)", +) +parser.add_argument( + "--idle-timeout", + type=int, + default=30, + help="Sandbox idle timeout in seconds before the platform auto-pauses it", +) +args = parser.parse_args() + +# ── Color palette ──────────────────────────────────────────────────────────── + +if args.dark: + PAL = dict( + accent="bold cyan", + ok="bold green", + warn="bold yellow", + layer="cyan", + key="magenta", + val="green", + muted="dim", + border_run="green", + border_pause="yellow", + border_code="blue", + border_out="green", + border_ok="green", + syntax="monokai", + match_yes="bold green", + match_no="bold red", + ) +else: + PAL = dict( + accent="bold #1e40af", # deep blue + ok="bold #166534", # forest green + warn="bold #9a3412", # burnt sienna + layer="bold #155e75", # dark teal + key="bold #6b21a8", # dark purple + val="#15803d", # medium green + muted="#6b7280", # slate gray + border_run="#166534", + border_pause="#9a3412", + border_code="#1e40af", + border_out="#166534", + border_ok="#166534", + syntax="friendly", + match_yes="bold #166534", + match_no="bold #b91c1c", + ) + +console = Console() +template_id = os.environ["CUBE_TEMPLATE_ID"] + +DEMO_CODE = """\ +import hashlib + +data = "Cube Sandbox auto-pause/auto-resume demo" +hash_val = hashlib.sha256(data.encode()).hexdigest()[:16] +pi_approx = 4 * sum((-1)**k / (2*k + 1) for k in range(500000)) + +print(f"sha256 = {hash_val}") +print(f"pi_approx = {pi_approx:.10f}") +""" + +CHECKPOINT = "/tmp/checkpoint.txt" +IDLE_TIMEOUT_SECONDS = args.idle_timeout + + +def collect_lines(raw_lines): + """on_stdout may deliver multi-line blobs; normalize to individual lines.""" + return [l for l in "\n".join(raw_lines).splitlines() if l.strip()] + + +def parse_kv(line): + """Extract value from 'key = value' formatted output line.""" + return line.split("=", 1)[1].strip() + + +def status_panel(status, sandbox_id, detail=None): + if status == "running": + indicator = f"[{PAL['ok']}]● Running[/]" + border = PAL["border_run"] + elif status == "paused": + indicator = f"[{PAL['warn']}]⏸ Paused[/]" + border = PAL["border_pause"] + else: + indicator = f"[{PAL['muted']}]? {status!s}[/]" + border = PAL["border_pause"] + body = f" Status: {indicator}\n ID: [bold]{sandbox_id}[/]" + if detail: + body += f"\n Detail: [{PAL['muted']}]{detail}[/]" + return Panel(body, title="Sandbox Status", border_style=border, padding=(0, 2)) + + +def lifecycle_panel(): + """Render the lifecycle config the demo will pass to Sandbox.create.""" + body = ( + f" [{PAL['key']}]on_timeout[/] : " + f"[{PAL['val']}]\"pause\"[/] " + f"[{PAL['muted']}]# park the VM instead of killing it[/]\n" + f" [{PAL['key']}]auto_resume[/] : " + f"[{PAL['val']}]True[/] " + f"[{PAL['muted']}]# next request transparently wakes it up[/]\n" + f" [{PAL['key']}]timeout[/] : " + f"[{PAL['val']}]{IDLE_TIMEOUT_SECONDS}s[/] " + f"[{PAL['muted']}]# idle threshold the sidecar uses[/]" + ) + return Panel( + body, + title="Lifecycle Config (e2b-compatible)", + border_style=PAL["accent"], + padding=(0, 2), + ) + + +def fetch_state(sandbox): + """Best-effort state fetch — backend may briefly 5xx during transitions.""" + try: + info = sandbox.get_info() + return info.get("state") or "unknown" + except Exception as exc: # noqa: BLE001 + return f"unreachable ({type(exc).__name__})" + + +# ── Title ──────────────────────────────────────────────────────────────────── + +console.print() +console.print( + Panel( + "[bold]Cube Sandbox · Auto-Pause / Auto-Resume Demo[/]", + box=box.DOUBLE, + style=PAL["accent"], + expand=True, + ) +) + +sandbox = Sandbox.create( + template=template_id, + timeout=IDLE_TIMEOUT_SECONDS, + lifecycle={"on_timeout": "pause", "auto_resume": True}, +) + +try: + sid = sandbox.sandbox_id + + # ── Step 1: Create Sandbox & Run Computation ───────────────────────────── + + console.rule(f"[{PAL['accent']}]Step 1 · Create Sandbox & Run Computation[/]") + console.print(lifecycle_panel()) + console.print(status_panel("running", sid)) + + console.print( + Panel( + Syntax( + DEMO_CODE.strip(), + "python", + theme=PAL["syntax"], + line_numbers=True, + ), + title="Executing in Sandbox", + border_style=PAL["border_code"], + ) + ) + + stdout_raw = [] + sandbox.run_code(DEMO_CODE, on_stdout=lambda m: stdout_raw.append(m.line)) + output_lines = collect_lines(stdout_raw) + + console.print( + Panel( + "\n".join(f" {line}" for line in output_lines), + title="Sandbox Output", + border_style=PAL["border_out"], + ) + ) + + hash_before = parse_kv(output_lines[0]) + pi_before = parse_kv(output_lines[1]) + + ckpt_content = f"hash={hash_before}\npi={pi_before}\n" + sandbox.files.write(CHECKPOINT, ckpt_content) + console.print(f" Writing checkpoint file... [{PAL['ok']}]✓[/]\n") + + before = {"hash_val": hash_before, "pi_approx": pi_before, "file_lines": "2 lines"} + + tbl = Table(title="State Snapshot (Before Idle)", box=box.ROUNDED) + tbl.add_column("Layer", style=PAL["layer"]) + tbl.add_column("Key", style=PAL["key"]) + tbl.add_column("Value", style=PAL["val"]) + tbl.add_row("Kernel Memory", "hash_val", before["hash_val"]) + tbl.add_row("Kernel Memory", "pi_approx", before["pi_approx"]) + tbl.add_row("Filesystem", CHECKPOINT, before["file_lines"]) + console.print(tbl) + console.print() + + # ── Step 2: Idle — Watch the Platform Pause It For Us ──────────────────── + + console.rule( + f"[{PAL['warn']}]Step 2 · Idle — Watch the Platform Auto-Pause[/]" + ) + console.print( + f" Sandbox timeout is [{PAL['warn']}]{IDLE_TIMEOUT_SECONDS}s[/]. " + f"We will idle past it and let the lifecycle manager step in.\n" + ) + + wait_for = IDLE_TIMEOUT_SECONDS + 15 + with Progress( + TextColumn("{task.description}"), + BarColumn(), + TextColumn("{task.completed:.0f}/{task.total:.0f}s"), + transient=False, + ) as progress: + task = progress.add_task( + f"[{PAL['warn']}]Idle (no SDK calls — platform is in charge)[/]", + total=wait_for, + ) + for _ in range(wait_for): + time.sleep(1) + progress.advance(task) + + state_after_idle = fetch_state(sandbox) + console.print() + console.print( + status_panel( + state_after_idle, + sid, + ( + f"Platform paused the VM after {IDLE_TIMEOUT_SECONDS}s of inactivity" + if state_after_idle == "paused" + else "Platform did not pause within window — feature may be misconfigured" + ), + ) + ) + + # ── Step 3: Issue a Request — Watch the Platform Resume It ─────────────── + + console.rule( + f"[{PAL['ok']}]Step 3 · Issue Request — Watch the Platform Auto-Resume[/]" + ) + console.print( + f" Next [bold]run_code[/] call will hit a paused sandbox. " + f"[{PAL['ok']}]auto_resume=True[/] tells the platform to wake it up " + f"transparently before our request lands.\n" + ) + + after_raw = [] + started = time.monotonic() + with console.status( + f"[{PAL['ok']}]Sending exec — platform is resuming sandbox in the background...[/]" + ): + sandbox.run_code( + 'print(f"sha256 = {hash_val}")\n' + 'print(f"pi_approx = {pi_approx:.10f}")', + on_stdout=lambda m: after_raw.append(m.line), + ) + elapsed = time.monotonic() - started + + console.print(status_panel("running", sid, f"Resume + exec took {elapsed:.2f}s")) + + # ── Step 4: Verify State Preserved Across the Auto-Pause/Resume Cycle ──── + + console.rule(f"[{PAL['ok']}]Step 4 · Verify State Preserved[/]") + + after_lines = collect_lines(after_raw) + after_file = sandbox.files.read(CHECKPOINT) + + after = { + "hash_val": parse_kv(after_lines[0]), + "pi_approx": parse_kv(after_lines[1]), + "file_lines": f"{len(after_file.strip().splitlines())} lines", + } + + cmp_tbl = Table(title="State Comparison", box=box.ROUNDED) + cmp_tbl.add_column("Layer", style=PAL["layer"]) + cmp_tbl.add_column("Key", style=PAL["key"]) + cmp_tbl.add_column("Before Idle", style=PAL["warn"]) + cmp_tbl.add_column("After Auto-Resume", style=PAL["ok"]) + cmp_tbl.add_column("Match", justify="center") + + all_pass = True + for key, label, layer in [ + ("hash_val", "hash_val", "Kernel Memory"), + ("pi_approx", "pi_approx", "Kernel Memory"), + ("file_lines", CHECKPOINT, "Filesystem"), + ]: + ok = before[key] == after[key] + all_pass = all_pass and ok + mark = f"[{PAL['match_yes']}]PASS[/]" if ok else f"[{PAL['match_no']}]FAIL[/]" + cmp_tbl.add_row(layer, label, before[key], after[key], mark) + + console.print(cmp_tbl) + console.print() + + if all_pass: + verdict = ( + f"[{PAL['ok']}]All state perfectly preserved across " + f"auto-pause / auto-resume![/]" + ) + verdict_border = PAL["border_ok"] + else: + verdict = ( + f"[{PAL['warn']}]One or more layers diverged after auto-resume — " + f"check sidecar / lifecycle config.[/]" + ) + verdict_border = PAL["border_pause"] + + console.print( + Panel( + verdict, + box=box.DOUBLE, + border_style=verdict_border, + expand=True, + ) + ) + console.print() + +finally: + # Clean up so the demo doesn't leave a sandbox behind on every run. + # We do this in a try/finally rather than a `with` block because the + # auto-pause path means the sandbox spends real wall time off-CPU, and + # we want exactly one explicit kill at the end regardless of failure. + try: + sandbox.kill() + except Exception as exc: # noqa: BLE001 + console.print(f"[{PAL['muted']}]cleanup: sandbox.kill() raised {exc!r}[/]") diff --git a/sdk/python/cubesandbox/sandbox.py b/sdk/python/cubesandbox/sandbox.py index 5ccc02d72..437c9c74e 100644 --- a/sdk/python/cubesandbox/sandbox.py +++ b/sdk/python/cubesandbox/sandbox.py @@ -40,6 +40,31 @@ def _check_response(resp: requests.Response) -> None: raise ApiError(msg, code) +_VALID_ON_TIMEOUT = ("kill", "pause") + + +def _serialize_lifecycle(lifecycle: Dict[str, Any]) -> Dict[str, Any]: + """Translate a snake_case ``lifecycle`` dict to the camelCase wire shape + used by CubeAPI (and by the e2b SDK). + + Validates ``on_timeout`` early so misspellings ("paused", "Pause") raise + a clean ``ValueError`` at the call site instead of producing an opaque + HTTP 4xx from the server. + """ + out: Dict[str, Any] = {} + on_timeout = lifecycle.get("on_timeout") + if on_timeout is not None: + if on_timeout not in _VALID_ON_TIMEOUT: + raise ValueError( + f"lifecycle.on_timeout must be one of {_VALID_ON_TIMEOUT!r}, " + f"got {on_timeout!r}" + ) + out["onTimeout"] = on_timeout + if "auto_resume" in lifecycle: + out["autoResume"] = bool(lifecycle["auto_resume"]) + return out + + class Sandbox: """A CubeSandbox code execution environment. @@ -98,6 +123,7 @@ def create( metadata: Dict[str, str] | None = None, allow_internet_access: bool = True, network: Dict[str, Any] | None = None, + lifecycle: Dict[str, Any] | None = None, config: Config | None = None, **kwargs: Any, ) -> "Sandbox": @@ -108,8 +134,9 @@ def create( timeout: Sandbox TTL in seconds. Defaults to ``Config.timeout`` (300). env_vars: Environment variables injected into the sandbox. metadata: Arbitrary key-value metadata (e.g. network-policy, host-mount). + allow_internet_access: When ``False``, the sandbox is blocked from + making outbound traffic to the public internet. network: Egress network policy. Accepts keys: - - ``allow_out`` / ``deny_out``: lists of CIDRs or hostnames (L3/L4). - ``rules``: list of :class:`~cubesandbox.Rule` dataclasses (or equivalent dicts with snake_case keys) for L7 host/path/SNI @@ -118,13 +145,29 @@ def create( ``{"api.example.com": [{"transform": {"headers": {...}}}]}``) is also accepted and converted into equivalent CubeEgress inject rules. + lifecycle: Optional dict mirroring the e2b SDK's ``lifecycle`` + object (https://e2b.dev/docs/sandbox/auto-resume). Accepts + two keys: + + - ``on_timeout``: ``"kill"`` (default) or ``"pause"``. When + ``"pause"``, an idle sandbox is suspended instead of + deleted; its memory snapshot survives across the pause. + - ``auto_resume``: ``bool``, default ``False``. Only + meaningful when ``on_timeout="pause"``. When ``True``, the + next request hitting the paused sandbox transparently + wakes it back up. When ``False`` you must call + :meth:`connect` to resume manually. + + Absent ``lifecycle`` keeps today's behaviour (idle sandboxes + are killed). config: SDK config. Uses default (env-based) config if omitted. Returns: A running :class:`Sandbox` instance. Raises: - ValueError: If no template ID is provided. + ValueError: If no template ID is provided, or if ``lifecycle`` + contains an unsupported ``on_timeout`` value. ApiError: On unexpected backend error (HTTP 500). """ cfg = config or Config() @@ -162,6 +205,11 @@ def create( net["rules"] = [_serialize_rule(r) for r in normalized_rules] if net: payload["network"] = net + # Lifecycle: opt-in. Wire shape mirrors e2b + # (https://e2b.dev/docs/sandbox/auto-resume) — a nested object with + # camelCase keys. Absent => server-side default ("kill" on timeout). + if lifecycle: + payload["lifecycle"] = _serialize_lifecycle(lifecycle) payload.update(kwargs) s = requests.Session() diff --git a/sdk/python/tests/test_sandbox.py b/sdk/python/tests/test_sandbox.py index 941cf3586..0608b3785 100644 --- a/sdk/python/tests/test_sandbox.py +++ b/sdk/python/tests/test_sandbox.py @@ -159,6 +159,73 @@ def test_create_network_empty_not_in_payload(self): body = m.call_args.kwargs["json"] assert "network" not in body + def test_create_default_lifecycle_omitted(self): + # Default behavior must remain wire-compatible with pre-feature + # callers: when ``lifecycle`` isn't set, the SDK must emit a + # payload that's byte-identical to the historical one — no + # ``lifecycle``, ``autoPause``, or ``autoResume`` keys. + with patch("requests.Session.post", return_value=mock_response(SANDBOX_DATA, status=201)) as m: + Sandbox.create(config=make_config()) + body = m.call_args.kwargs["json"] + assert "lifecycle" not in body + assert "autoPause" not in body + assert "autoResume" not in body + + def test_create_lifecycle_pause_on_timeout(self): + # `on_timeout="pause"` alone (no auto_resume) should ship a + # camelCase nested object and nothing else lifecycle-related. + with patch("requests.Session.post", return_value=mock_response(SANDBOX_DATA, status=201)) as m: + Sandbox.create( + lifecycle={"on_timeout": "pause"}, + config=make_config(), + ) + body = m.call_args.kwargs["json"] + assert body["lifecycle"] == {"onTimeout": "pause"} + + def test_create_lifecycle_pause_with_auto_resume(self): + # The full e2b-shaped lifecycle: pause on timeout AND auto-resume on + # next request. Wire shape mirrors + # https://e2b.dev/docs/sandbox/auto-resume verbatim. + with patch("requests.Session.post", return_value=mock_response(SANDBOX_DATA, status=201)) as m: + Sandbox.create( + lifecycle={"on_timeout": "pause", "auto_resume": True}, + config=make_config(), + ) + body = m.call_args.kwargs["json"] + assert body["lifecycle"] == {"onTimeout": "pause", "autoResume": True} + + def test_create_lifecycle_kill_explicit(self): + # Explicit "kill" is identical to the default but the user might + # set it for clarity. Make sure it round-trips. + with patch("requests.Session.post", return_value=mock_response(SANDBOX_DATA, status=201)) as m: + Sandbox.create( + lifecycle={"on_timeout": "kill"}, + config=make_config(), + ) + body = m.call_args.kwargs["json"] + assert body["lifecycle"] == {"onTimeout": "kill"} + + def test_create_lifecycle_invalid_on_timeout_raises(self): + # Mistyped values must fail client-side, before we hit the network, + # so the user gets a stack trace pointing at their source. + with pytest.raises(ValueError, match="on_timeout"): + Sandbox.create( + lifecycle={"on_timeout": "Pause"}, # capital P + config=make_config(), + ) + + def test_create_lifecycle_auto_resume_only(self): + # Asymmetric input — auto_resume without on_timeout — is allowed + # and just translates literally; it's the server's job to enforce + # that auto_resume only matters when on_timeout="pause". + with patch("requests.Session.post", return_value=mock_response(SANDBOX_DATA, status=201)) as m: + Sandbox.create( + lifecycle={"auto_resume": True}, + config=make_config(), + ) + body = m.call_args.kwargs["json"] + assert body["lifecycle"] == {"autoResume": True} + # ── domain filtering (DNS allow-list) ────────────────────────────────────────