process cache: replace single refcnt with structured ProcessLifecycle for safe cache eviction

[sglushko]

The process cache uses a single atomic.Uint32 reference counter (refcnt) to track process lifecycle and determine when a process can be safely evicted. This design has several correctness issues:

1. Double decrement on exit + cleanup When a process is replaced by execve, both an exit event and a cleanup event fire for the old process. Each handler independently calls proc.RefDec("process") and parent.RefDec("parent"), leading to refcnt underflow:

process++: 1, process--: 2  →  refcnt wraps to 4294967295
parent++: 1, parent--: 2    →  parent refcnt wraps to 4294967295

2. Premature cache eviction The garbage collector uses refcnt.Load() != 0 to decide whether a process can be deleted. However, refcnt == 0 does not mean the process lifecycle is complete — a process can reach refcnt == 0 before its exit event arrives, causing premature eviction and loss of process context for subsequent events.

3. No separation of concerns A single counter mixes different lifecycle aspects (process start/exit, parent/child relationships, ancestor tracking). This makes it impossible to determine why a process is still alive or which component is holding a reference.

4. No idempotency guarantees Multiple code paths can decrement the same logical reference (e.g., both exit and cleanup decrement "parent"), with no mechanism to ensure each logical operation happens exactly once.

[sglushko]

Proposed Solution
Introduce a ProcessLifecycler interface and a default ProcessLifecycle implementation, embedded in ProcessInternal, that provides structured lifecycle tracking separate from refcnt:

// ProcessLifecycler defines the lifecycle contract for process cache eviction.
type ProcessLifecycler interface {
    MarkExited() bool
    ChildRefInc()
    ChildRefDec()
    TryClaimParentRelease() bool
    CanRemove() bool
}
 
// ProcessLifecycle is the default implementation.
type ProcessLifecycle struct {
    exitSeen              atomic.Bool   // CAS: true after first exit/cleanup (idempotent)
    childRefs             atomic.Int32  // +1 on child exec/clone, -1 when child is ready for removal
    parentReleaseClaimed  atomic.Bool   // CAS: claim right to decrement parent's childRefs (exactly once)
}

Key design decisions:

• Interface + factory function allows downstream consumers to provide extended implementations
• exitSeen CAS (Compare-And-Set) solves double process-- from exit + cleanup — second call is a no-op
• childRefs explicitly tracks parent↔child relationship (replaces parent++/parent--)
• TryClaimParentRelease() CAS guarantees parent notification happens exactly once
• CanRemove() is exitSeen && childRefs <= 0 — clear, deterministic eviction criteria
• ancestor refs are left on the existing refcnt — they represent event-level reference counting (viewer count), not process lifecycle state

The GC check changes from refcnt.Load() == 0 to lifecycle.CanRemove(), ensuring processes are only evicted when their lifecycle is truly complete.

[dwindsor]

Yes this looks reasonable to me. Interested to see the implementation.

Please look over the existing exec tests (and helpers) and see how many more you can add that address the edge cases that you want to fix here. Some cases will be impossible to test in unit tests, as they'll require sleeping. We may need to develop integration tests for these, not sure.

[dwindsor]

Also would be interested to see performance metrics before / after this fix. See if you can collect cache metrics for a high-churn k8s environment (a job with parallelism: 50 or similar). In a high-churn environment, these extra levels of indirection for ProcessLifecycle during refcount ops actually may have some impact but it's likely going to be negated by better cache performance (less evictions, etc).

[sglushko]

I also wanted to mention another potential issue. It might be out of scope for this task, but I'd like to keep it in mind. If an exec or clone event is duplicated for some reason, the following happens:

Silent replacement on duplicate exec/clone events: cache.add() uses lru.Add() which silently replaces the existing entry when the same exec_id is added twice. The old ProcessInternal becomes unreachable while the parent's refcnt has already been incremented — this increment is never balanced, causing the parent to be retained in the cache indefinitely. The LRU eviction callback is not triggered for replacements, only for capacity overflows.

We could protect cache.add() from silent replacement by switching from lru.Add() to lru.ContainsOrAdd() and adding a log:

func (pc *Cache) add(process *ProcessInternal) bool {
    found, evicted := pc.cache.ContainsOrAdd(process.process.ExecId, process)
    if found {
        logger.GetLogger().Warn("process cache: duplicate add for exec_id, ignoring",
            "exec_id", process.process.ExecId)
        return false
    }
    ...
}

[sglushko]

I found another issue that might be a potential leak. There are processes in the cache with refcnt == 0 and color == deletePending. Our usual dump command, tetra dump processcache --exclude-execve-map-processes --skip-zero-refcnt --max-recv-size 1000000000, does not show them because of the zero refcount. Even when running it without that flag, it still looks like everything is fine and as if the process is about to be deleted.

Problem
When a process in the GC deleteQueue is no longer removable (e.g. a new child appeared between ticks), the GC drops it from the queue but does not reset its color back to inUse. https://github.com/cilium/tetragon/blob/main/pkg/process/cache.go#L79
If the process later becomes removable again and deletePending() sends it to deleteChan, the handler sees color != inUse and skips re-adding it to the queue. https://github.com/cilium/tetragon/blob/main/pkg/process/cache.go#L98
The process is permanently stuck — never collected, only reclaimable via LRU eviction.

I plan to include the fix as part of the current task.

[sglushko]

A quick update: work is in progress. The initial approach has changed slightly, but I am moving forward.