[BUG]: Runtime metrics undercount gunicorn RSS after pre-fork enable; entity_id tags dropped

[asaf400]

Tracer Version(s)

3.19.0, 4.7.1, 4.12.2

Python Version(s)

Python 3.11.x, Python 3.13

Pip Version(s)

pip from python:3.13-slim-bookworm

Bug Report

Summary

When using Python runtime metrics (DD_RUNTIME_METRICS_ENABLED=true) with gunicorn (and similarly with Kubernetes single-step instrumentation / SSI, which enables products in the master before fork), runtime.python.mem.rss is often wrong by an order of magnitude: Datadog shows roughly master-sized RSS and usually one runtime-id, while worker processes each hold hundreds of MiB and the container cgroup is ~N× larger.

Separately, even when metrics emit, RuntimeWorker.periodic() can overwrite DogStatsD constant_tags and drop dd.internal.entity_id, so Kubernetes origin tags (pod_name, kube_namespace) never attach — despite DD_ENTITY_ID being set correctly.

Neither behavior is documented in the public Runtime Metrics (Python) docs (tracing/metrics/runtime_metrics/?tab=python) or the Kubernetes origin-detection docs. Operators comparing runtime.python.mem.rss to container.memory.rss will see a large “leak” that is actually a client fork / tagging bug.

Environment

• Gunicorn multi-worker APIs on AWS EKS (Bottlerocket, cgroup v2)
• Datadog Operator admission / SSI targeting Python tracer major 4 (injected ~4.12.2)
• DD_RUNTIME_METRICS_ENABLED=true, DD_RUNTIME_METRICS_RUNTIME_ID_ENABLED=true
• DD_ENTITY_ID from the downward API (pod UID); DD_AGENT_HOST = node Agent

---

Issue A — Gunicorn / pre-fork: RSS bound to master PID (primary)

Symptom

• Series exist and may have pod tags.
• Often one runtime-id per pod.
• runtime.python.mem.rss ≈ master VmRSS (tens of MiB).
• Each gunicorn worker’s /proc/<pid>/status VmRSS is hundreds of MiB; cgroup RSS ≈ N× worker.
• avg:runtime.python.mem.rss vs container.memory.rss looks like a multi‑GB discrepancy — they are not comparable until per-worker series exist.

Root cause (library behavior)

Two behaviors compound after fork:

1. RuntimeWorker.enable() is idempotent. If _instance is already set (master enabled via SSI / preload.py / config import), a later enable() in the worker is a no-op. import ddtrace.bootstrap.sitecustomize after preload is already in sys.modules also ends up calling enable() and does not rebind.

2. RSS collector freezes PID at enable time (psutil.Process(os.getpid()) in the metric collector). If enable runs in the master before fork(), workers inherit the singleton, the same runtime-id, and a collector still pointed at the master PID.

Typical SSI + gunicorn + ddtrace 4.x path:

1. Master: SSI / preload → RuntimeWorker.enable() before fork.
2. Gunicorn forks workers; children inherit _instance + master-bound collector.
3. Worker post_worker_init that only calls enable() again or import sitecustomize does not fix it.
4. DogStatsD reports one runtime-id with master-sized RSS.

Workaround that works

In every gunicorn post_worker_init:

from ddtrace.internal.runtime.runtime_metrics import RuntimeWorker

RuntimeWorker.disable()
RuntimeWorker.enable()

That issues a new runtime-id and rebinds psutil to the worker PID. After rebind you typically get 1 + N series (master + N workers) if the master also enabled pre-fork — that is expected and correct for summing process RSS.

Controlled Docker matrix (ddtrace 4.12.2, workers=4, ballast=64 MiB)

Full repro: https://github.com/asaf400/repro-ddtrace-gunicorn-rss

Mode	Master enable?	Worker post_worker_init	n runtime-ids	dd_sum MiB	Σ VmRSS MiB	dd_sum / Σ VmRSS
master_only	yes	none	1	47.8	470.9	0.10
sitecustomize_post	yes	import sitecustomize only	1	47.3	510.6	0.09
rebind_post	yes	disable(); enable()	5	471.2	471.2	1.00

Correctness check is dd_sum ≈ Σ /proc VmRSS. Cgroup anon can be lower than Σ VmRSS after fork because of CoW shared pages — that gap is expected and is not a ddtrace bug.

Docs gap (Issue A)

The Runtime Metrics Python page documents enabling via env / ddtrace-run / auto instrumentation, but does not mention:

• gunicorn / prefork / uWSGI-style process models
• that enabling RuntimeWorker in the master before fork makes worker RSS report the master
• that post-fork import ddtrace.bootstrap.sitecustomize / enable() alone is insufficient after a successful master enable
• that RuntimeWorker.disable(); enable() (or an official post-fork hook) is required for correct per-worker RSS
• how runtime.python.mem.rss should be aggregated vs container.memory.rss (sum by runtime-id, not a single avg)

---

Issue B — RuntimeWorker.periodic() drops Kubernetes entity ID tags

Symptom

• runtime.python.* appears in Metric Summary.
• Dashboards / explorers filtered by pod_name or kube_namespace show nothing useful.
• Agent origin detection is already on (DD_DOGSTATSD_ORIGIN_DETECTION=true, client mode).
• Pods already have DD_AGENT_HOST and DD_ENTITY_ID=<pod-uid>.

Root cause

RuntimeWorker.periodic() rebuilds the DogStatsD client’s constant_tags and drops the entity-ID tag derived from DD_ENTITY_ID. Packets leave the pod without dd.internal.entity_id, so the Agent cannot attach Kubernetes origin tags.

Observed on both 3.19.x and 4.7.x / 4.12.x.

Controlled check from a pod: the same custom DogStatsD metric with dd.internal.entity_id:<pod-uid> gets pod_name + kube_namespace; without it, no origin tags. Agent-side config alone cannot fix client packets that never carry an entity ID. Cgroup v2 container-ID fallback also does not rescue this path when entity-ID handling is broken / overwritten.

Workaround

Re-assert the entity ID via DD_TAGS, which ddtrace re-applies after rewriting constant_tags:

DD_TAGS=dd.internal.entity_id:$(DD_ENTITY_ID)

(Kubernetes expands $(DD_ENTITY_ID) from the downward-API sibling env var.)

Docs gap (Issue B)

Docs describe DD_ENTITY_ID / origin detection, but do not warn that the Python runtime-metrics flusher can strip the entity tag, or that DD_TAGS=dd.internal.entity_id:… is a necessary workaround for runtime.python.* on Kubernetes.

---

Related documentation / product gaps (not separate library defects, but missing from docs)

These made the above much harder to diagnose:

1. patch_all() does not start RuntimeWorker. Runtime metrics are started from ddtrace/bootstrap/preload.py only (ddtrace-run, import ddtrace.auto, import ddtrace.bootstrap.sitecustomize, or successful SSI). Apps that only call ddtrace.patch_all(...) never enable runtime metrics even with DD_RUNTIME_METRICS_ENABLED=true. Docs discuss the env flag in the context of auto/ddtrace-run, but do not state clearly that patch_all alone never enables the product (and patch_all is deprecated in favour of import ddtrace.auto).

2. SSI major mismatch (3.x app + 4.x inject). Admission injects tracer major 4 on PYTHONPATH. If the app pins ddtrace 3.x, auto-bootstrap fails (e.g. ModuleNotFoundError: ddtrace.internal.settings) and the process continues on 3.x without preload — so no runtime metrics unless the app imports sitecustomize itself. This interaction is not called out next to Runtime Metrics + SSI docs.

---

Expected behavior

1. After gunicorn fork, each worker should emit its own runtime-id with RSS for that PID (or docs / SSI should provide an official post-fork rebind).
2. Runtime-metrics flushes should preserve dd.internal.entity_id derived from DD_ENTITY_ID so Agent origin detection can attach pod_name / kube_namespace.
3. Public docs should document gunicorn/prefork caveats, the patch_all vs preload distinction, and the entity-ID / DD_TAGS requirement for Kubernetes.

Actual behavior

1. Pre-fork enable → workers inherit master collector → ~10% of real Σ process RSS in our matrix until disable(); enable().
2. Periodic flush can strip entity ID → metrics exist but are unusable for pod-scoped queries without DD_TAGS workaround.
3. Docs are silent on both.

Reproduction Code

Public repro (Docker Compose matrix): https://github.com/asaf400/repro-ddtrace-gunicorn-rss

git clone https://github.com/asaf400/repro-ddtrace-gunicorn-rss.git
cd repro-ddtrace-gunicorn-rss

DD_REPRO_MODE=master_only        COMPARE_WAIT_S=45 docker compose run --rm --build repro
DD_REPRO_MODE=sitecustomize_post COMPARE_WAIT_S=45 docker compose run --rm repro
DD_REPRO_MODE=rebind_post        COMPARE_WAIT_S=45 docker compose run --rm repro

Modes:

DD_REPRO_MODE	Behavior
master_only	RuntimeWorker.enable() in master (config import) only — simulates SSI pre-fork
sitecustomize_post	Same + import ddtrace.bootstrap.sitecustomize in post_worker_init
rebind_post	Same + RuntimeWorker.disable(); enable() in post_worker_init

Look at printed RATIO dd_sum/sum_workers. Expect ~0.1 for the first two modes and ~1.0 for rebind_post.

Minimal post-fork fix demonstrated in the repro:

def post_worker_init(_worker):
    from ddtrace.internal.runtime.runtime_metrics import RuntimeWorker
    RuntimeWorker.disable()
    RuntimeWorker.enable()

Error Logs

No hard crash. Behavioral:

• Datadog: single / few runtime-ids; runtime.python.mem.rss ≈ master VmRSS while workers are large.
• With SSI + app on ddtrace 3.x: Error in sitecustomize; … ModuleNotFoundError: ddtrace.internal.settings (bootstrap aborts; process continues without RuntimeWorker).
• With entity-ID strip: metrics present in Metric Summary but empty when filtered by pod_name / kube_namespace.

DD_TRACE_DEBUG=true was not required to observe the DogStatsD RSS undercount; the in-process StatsD sink in the repro captures raw runtime.python.mem.rss lines.

Libraries in Use

Repro image (Dockerfile):

ddtrace==4.12.2
gunicorn==23.0.0
uvicorn==0.34.0
starlette==0.46.2

Also observed in production with ddtrace 3.19.0 / 3.18.1 / 4.7.1 under gunicorn + FastAPI / Uvicorn workers on EKS.

Operating System

Bottlerocket v1.62.1

[forevercrab321-svg]

The docs gap is real, but it isn't why this survived 3.19 → 4.12. The wrong number was self-consistent. One runtime-id reporting master-sized RSS isn't a hole in a dashboard — it renders, it's stable, it has no error attached, and nothing else the library emits contradicts it. Your dd_sum ≈ Σ /proc VmRSS check works precisely because you brought a number in from outside ddtrace. Nobody who only reads ddtrace's own output has any reason to suspect it.

Ours, same shape: we asked our logs for every caller in a window and got back a clean list of exactly 1000 rows. The third-largest caller — 103 calls — was not on it. The row cap is silent: no error, no truncation flag, no gap in the shape of the answer. What exposed it was an aggregate: group by summing to 1,899, which matched the total we already knew. Had we shipped off the list, we'd have said that machine didn't run that window. It ran 103 times.

Free half, and it's inside the library rather than in a docs paragraph, because a paragraph can't fire: the collector already knows the PID it bound to at enable(). Comparing os.getpid() to it once per flush is one integer compare, and bound-PID ≠ current-PID is the fork bug, exactly and with no false positives — a worker that inherited the singleton is in that state and a correctly-rebound one never is. Emit it as a tag or a one-shot warning and the undercount stops being silent, whether or not the operator knows to compare against container.memory.rss.

The thing only your matrix can answer, and it decides whether rebind_post is a fix or an instantaneous fix: does dd_sum / Σ VmRSS hold at 1.00 across worker recycling (--max-requests)? A recycled worker takes a new PID and a new runtime-id, while the dead worker's series just stops rather than going to zero — so a sum by runtime-id over any window longer than a worker lifetime may carry both. If it does, the ratio is 1.00 at every instant and wrong on every dashboard with a time range.

[asaf400]

Indeed, sum by runtime-id via query: sum:runtime.python.mem.rss{env:new-mexico, kube_namespace:blackmesa, service:Lambda-core} by {pod_name,runtime-id}.rollup(avg)
this results in ratio around of 0.8~ - meaning that DD metric from query above was actually higher then either DD(Cgroup) or Promteheus memory queries:
max:container.memory.rss{env:new-mexico, kube_namespace:blackmesa, service:Lambda-core} by {pod_name,service}
sum by(pod) (container_memory_rss{container!="", namespace="blackmesa", pod="lambda-core-54c6d8ccdf-hgzkr"})

I'm investigating how is that possible as well, how can sum of DD metrics is now higher then what cgroup \ cadvisor reports the RSS of the pod..

DD runtime: 1001MB
DD k8s: 769MB
Prom: 789MB

Note, without rollup, the query reaches for the skies, and goes to 40GB memory, which doesn't make sense, because I guess sum aggregates all raw data points across the interval, detached from intended purpose of just summing all 5 runtime-ids (master+4 workers) (66+(160~*4))=700MB

[forevercrab321-svg]

I named that mechanism and got the sign wrong, so let me correct myself first: I said the ratio would sit at 1.00 at every instant and only break on a time range. Yours is 0.77 the other way. Over, not under.

Over is what recycling looks like from inside a sum by. A recycled worker takes a new PID and a new runtime-id; the dead worker's series doesn't go to zero, it just stops. Sum a window and you carry the corpse's last value next to its replacement. Your own arithmetic sizes it: you expect 66 + 160×4 ≈ 706MB, you're reading 1001MB, and the excess is ~295MB — call it one and a bit worker-sized series that shouldn't be in the addition.

The other tell is which number is the outlier. cgroup says 769 and Prometheus says 789 — two independent collectors, 2.6% apart, agreeing. The one that disagrees is the only one built by adding series together. That points at the set being summed, not at memory.

Two checks, neither needs the theory to be right:

• Run the same query over 5 minutes and over 4 hours. Ghost-series contamination scales with window length; a real memory difference does not.
• Count the distinct runtime-ids the by-clause returns for one pod. Five processes should give five. If you get six or seven, look at the last datapoint of each — a series that ends mid-window is a recycled worker, and that's the whole answer with no inference in it.

Your 40GB reading is consistent with your own guess, and it's worth naming as a separate bug: 40GB / 1001MB ≈ 40, so unrolled sum is folding every raw point in the interval rather than one point per series. That number is measuring points-per-interval. It isn't memory at all.

Which brings this back to where the issue started. If the excess is dead ids, rebind_post is correct and the metric is still wrong on any dashboard carrying a time range — you'd have replaced a silent 10× undercount with a quieter ~30% overcount, and both render fine. Same trap, one turn later.

The thing only your deployment can answer: is --max-requests (or --max-requests-jitter) set on that gunicorn config, and over a 4h window does the pod's runtime-id cardinality exceed worker count + 1? If yes, the 0.77 is arithmetic rather than memory, and the fix belongs in how the query bounds the series set, not in the collector.