fix(idalib): don't reap busy headless workers on health-probe timeout - #483
fix(idalib): don't reap busy headless workers on health-probe timeout#483lich0821 wants to merge 1 commit into
Conversation
resolve_session() probed each worker (TCP + JSON-RPC ping) before forwarding
a tool call and reaped it on the first failure. A worker mid auto-analysis or
mid decompile is single-threaded and cannot answer the ping until it yields,
so a busy worker was misclassified as dead and torn down — losing the live
session. With aggressive warmup (run_auto_analysis/build_caches/init_hexrays)
on a large database this is reliably reproducible: the worker is reaped during
its own analysis.
Distinguish three worker states instead of two:
- process exited -> dead, reap ("not reachable")
- process alive, no ping -> busy, keep and retry with backoff
- alive + unresponsive long -> wedged, reap after a grace window
Also:
- health-probe timeouts widened and made env-overridable
(IDA_MCP_HEALTH_TCP_TIMEOUT=2.0, IDA_MCP_HEALTH_RPC_TIMEOUT=10.0)
- retry with backoff before reaping
(IDA_MCP_HEALTH_RETRIES=3, IDA_MCP_HEALTH_RETRY_BACKOFF=1.0)
- wedged grace window IDA_MCP_WEDGED_GRACE_SEC=300 (0 = never reap a
live worker)
- _prune_dead_worker_sessions_locked prunes only exited processes
(is_alive), not unreachable-but-live ones
- idle_ttl_sec <= 0 now means "never self-exit" for long-lived workers
All defaults preserve existing behavior for healthy workers; only the
handling of an unreachable worker changes (now conservative instead of
trigger-happy).
|
Nice fix, but the busy path is gated on Two smaller ones: on a busy worker |
Problem
resolve_session()probed each worker (TCP connect + JSON-RPCping) beforeforwarding a tool call, and reaped the worker on the first failed probe. A
worker mid
auto_wait()/decompile is single-threaded and cannot answer theping until it yields, so a busy worker was misclassified as dead and torn
down — losing the live session. With aggressive warmup
(
run_auto_analysis/build_caches/init_hexrays) on a large database thisis reliably reproducible: the worker gets reaped during its own analysis.
Fix
Distinguish three worker states instead of two:
Also:
(
IDA_MCP_HEALTH_TCP_TIMEOUT=2.0,IDA_MCP_HEALTH_RPC_TIMEOUT=10.0)(
IDA_MCP_HEALTH_RETRIES=3,IDA_MCP_HEALTH_RETRY_BACKOFF=1.0)IDA_MCP_WEDGED_GRACE_SEC=300(0 = never reap a liveworker)
_prune_dead_worker_sessions_lockednow prunes only exited processes(
is_alive()), not unreachable-but-live onesidle_ttl_sec <= 0now means "never self-exit" for long-lived workersAll defaults preserve existing behavior for healthy workers; only the
handling of an unreachable worker changes (now conservative instead of
trigger-happy).
Test plan
tests/test_idalib_supervisor.py(extended with busy/wedged statecoverage)
tests/test_worker_lifecycle.py