[BUG]: Setting DD_API_KEY adds ~2.2s to interpreter shutdown in 4.14.0

[ruidc]

Tracer Version(s)

4.14.0, and 4.14.0rc1–rc4 (4.13.1 and earlier unaffected)

Python Version(s)

Python 3.14.7

Pip Version(s)

pip 26.2.1

Bug Report

Since 4.14.0rc1, a process with DD_API_KEY set spends ~2.2s in the atexit telemetry flush. import ddtrace alone reproduces it — no application, no Agent, no valid key. Import is unaffected (~0.09s); the cost is entirely at shutdown, so it lands on every short-lived process, container SIGTERM, and test that spawns a subprocess.

Cause. Bisected by building either side of #19368 ("port telemetry to the rust client"), same image, same CPython 3.14.7, same deps, only the commit differing:

commit	DD_API_KEY set
68004ea (merge parent)	0.129s
965a7ce (#19368)	2.322s

DD_API_KEY alone selects agentless (writer.py: agentless = config.AGENTLESS_MODE or config.API_KEY not in (None, "")) — unchanged since 4.13.1, so not itself the regression. What changed is the flush:

+        if force_flush:
+            try:
+                self._worker.flush()
+            except Exception:
+                log.debug("Failed to flush the native telemetry worker", exc_info=True)
 
     def app_shutdown(self) -> None:
-        if self.started:
-            self.periodic(force_flush=True, shutting_down=True)
+        if self._worker is not None:
+            self.periodic(force_flush=True)
         self.disable()

periodic() also now calls app_started() and _report_dependencies() first. Combined, a process that previously sent nothing at exit now emits a full lifecycle and blocks on it. Against a logging Agent listener, 4.13.1 sends 0 telemetry requests at shutdown; 4.14.0 sends 3 sequential POSTs (app-started, app-dependencies-loaded+app-heartbeat, app-closing). Each is a full HTTPS round-trip to the intake — 0.64–0.85s measured with curl — hence ~2.2s.

The absolute figure is region-dependent. These numbers are from Australia, and instrumentation-telemetry-intake.datadoghq.com resolves to AWS us-east-1 (3.233.158.54). The invariant is three sequential blocking round-trips at process exit; the cost is 3× the RTT to your intake, so reproducing from close to your Datadog site will show proportionally less, and users far from theirs will see more. (I am deliberately not quoting a DNS/TCP/TLS phase split: the measurement ran inside Docker Desktop, whose userspace networking reports a sub-millisecond TCP connect to a US endpoint, so per-phase attribution is unreliable. The totals are unaffected.)

Notes

• A local Agent does not avoid it: DD_AGENT_HOST set alongside DD_API_KEY still goes agentless (2.316s).
• The flush is unbounded in Agent mode too — against an Agent that responds after 3s, shutdown takes 9.158s (three round-trips). A dead Agent is fine (~0.09s); only a slow-but-alive one hurts.
• There appears to be no opt-out: AGENTLESS_MODE resolves to civisibility.agentless.enabled, so with DD_API_KEY set agentless cannot be disabled short of DD_INSTRUMENTATION_TELEMETRY_ENABLED=false.

Workaround

DD_INSTRUMENTATION_TELEMETRY_ENABLED=false restores shutdown to 0.090s. It does not cost APM tracing: against an Agent listener the app still sends POST /v0.5/traces with a byte-identical payload, and only the telemetry POSTs disappear. What is lost is the SDK dependency/library inventory.

Where a local Agent is available, removing DD_API_KEY from the application environment (leaving it on the Agent) also avoids it: telemetry then resolves to agentless=False, stays enabled, and the payloads go to the Agent's /telemetry/proxy/api/v2/apmtelemetry endpoint instead.

Suggested fix

1. Bound the shutdown flush with a timeout — this is the regression. Neither a slow intake nor a slow Agent should dominate process exit, and as above the Agent path is unbounded too.
2. Provide an opt-out for telemetry agentless mode, independent of CI Visibility. The second workaround above only helps if DD_API_KEY can be removed from the application environment. Where it has to stay set, agentless cannot be turned off at all — DD_AGENT_HOST does not escape it (2.316s) and neither does DD_CIVISIBILITY_AGENTLESS_ENABLED=false (2.271s) — leaving disabling instrumentation telemetry wholesale as the only lever.
3. Document the change. refactor(telemetry): port telemetry to the rust client, shared with data-pipeline #19368 shipped no releasenotes/notes/*.yaml, and neither the 4.14.0rc1 nor the 4.14.0 notes mention the telemetry transport rewrite or the new shutdown behaviour — the only telemetry-adjacent entries are OpenTelemetry ones and an unrelated uWSGI atexit fix. Note the agentless selection is unchanged since 4.13.1; what changed is its consequence — selecting agentless used to cost nothing at exit because nothing was sent, and now costs three blocking round-trips. A transport rewrite that changes the effect of an existing setting that much is worth a note even without the timeout bug.
4. Gate new transport behaviour on a benchmark. benchmarks/startup already times the whole subprocess (subprocess.check_call(args=args, env=env)), so process shutdown is inside what it measures, and the scenario already takes an env dict — but config.yaml sets env: {} and no scenario sets DD_API_KEY, so the one benchmark that would have caught this never enters the affected path. A variant with DD_API_KEY set (any value; the intake rejects a dummy key just as slowly) turns this into a ~20x regression the suite can fail on. Worth doing generally: a change that alters when or how the library talks to the network should be exercised under the env that selects each transport.

Reproduction Code

for V in 4.13.1 4.14.0; do
  python3 -m venv /tmp/dd-$V && /tmp/dd-$V/bin/pip -q install ddtrace==$V
  for MODE in unset set; do
    [ $MODE = set ] && E=(env DD_API_KEY=dummy0000000000000000000000000000) || E=(env -u DD_API_KEY)
    T=$( { TIMEFORMAT=%R; time "${E[@]}" /tmp/dd-$V/bin/python -c "import ddtrace" >/dev/null 2>&1; } 2>&1 )
    printf "ddtrace %-8s DD_API_KEY %-5s -> %ss\n" $V $MODE $T
  done
done

ddtrace 4.13.1   DD_API_KEY unset -> 0.106s
ddtrace 4.13.1   DD_API_KEY set   -> 0.092s
ddtrace 4.14.0   DD_API_KEY unset -> 0.106s
ddtrace 4.14.0   DD_API_KEY set   -> 2.285s   <-- regression

Error Logs

DD_TRACE_DEBUG=true emits nothing relevant — no tracer error is raised; the flush succeeds, it is
simply slow. The useful artefact is a faulthandler.dump_traceback_later(1.0) armed just before
shutdown:

Timeout (0:00:01)!
Thread 0x0000ffff86924160 [_TelemetryDepen] (most recent call first):
  <no Python frame>

Thread 0x0000ffff89b664c0 [python] (most recent call first):
  File ".../ddtrace/internal/telemetry/writer.py", line 847 in periodic
  File ".../ddtrace/internal/telemetry/writer.py", line 857 in app_shutdown
  File ".../ddtrace/internal/atexit.py", line 31 in _wrapped

Libraries in Use

Reproduces in a venv containing only ddtrace and its own dependencies:

bytecode==0.19.0
ddtrace==4.14.0
envier==0.6.1
opentelemetry-api==1.44.0
typing_extensions==4.16.0
wrapt==2.3.0

Also reproduced in a 158-package application environment where ddtrace was the only package swapped
(verified by diffing the full distribution set and the two uv.lock files: exactly one package
drifts).

Operating System

Linux 7.0.12-linuxkit #1 SMP PREEMPT aarch64 GNU/Linux — Debian 13 (trixie) container