[BUG] system-probe resolvePidCgroupFallback Failed to add pid

[ajax-bychenok-y]

Agent version

7.73.0-jmx

Bug Report

We use datadog-operator on EKS and in latest stable release of operator (helm chart version 2.15.2) it produces a lot of warnings like:

system-probe 2025-12-17 15:40:12 UTC | SYS-PROBE | INFO | (pkg/security/resolvers/cgroup/resolver.go:270 in resolvePidCgroupFallback) | Failed to add pid 1910503, error on fallback to resol
ve its cgroup: no cgroup mount point found
system-probe 2025-12-17 15:40:12 UTC | SYS-PROBE | INFO | (pkg/security/resolvers/cgroup/resolver.go:270 in resolvePidCgroupFallback) | Failed to add pid 1910504, error on fallback to resol
ve its cgroup: no cgroup mount point found
system-probe 2025-12-17 15:40:14 UTC | SYS-PROBE | INFO | (pkg/security/resolvers/cgroup/resolver.go:270 in resolvePidCgroupFallback) | Failed to add pid 1910505, error on fallback to resol
ve its cgroup: no cgroup mount point found
system-probe 2025-12-17 15:40:14 UTC | SYS-PROBE | INFO | (pkg/security/resolvers/cgroup/resolver.go:270 in resolvePidCgroupFallback) | Failed to add pid 1910515, error on fallback to resol
ve its cgroup: no cgroup mount point found
system-probe 2025-12-17 15:40:14 UTC | SYS-PROBE | INFO | (pkg/security/resolvers/cgroup/resolver.go:270 in resolvePidCgroupFallback) | Failed to add pid 1910516, error on fallback to resol
ve its cgroup: no cgroup mount point found
system-probe 2025-12-17 15:40:14 UTC | SYS-PROBE | INFO | (pkg/security/resolvers/cgroup/resolver.go:270 in resolvePidCgroupFallback) | Failed to add pid 1910517, error on fallback to resol
ve its cgroup: no cgroup mount point found

Tried also latest RC 2.16.0-dev.6 but problem is the same. Am I doing something wrong?

Reproduction Steps

Just setup latest stable datadog-operator 2.15.2 in EKS.

Agent configuration

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
spec:
  features:
    admissionController:
      enabled: true
    apm:
      enabled: false
    clusterChecks:
      enabled: true
    cspm:
      enabled: false
    cws:
      enabled: false
    dogstatsd:
      hostPortConfig:
        enabled: false
      originDetectionEnabled: true
      tagCardinality: low
      unixDomainSocketConfig:
        enabled: true
    ebpfCheck:
      enabled: false
    eventCollection:
      collectKubernetesEvents: true
    externalMetricsServer:
      enabled: false
    kubeStateMetricsCore:
      enabled: true
    liveContainerCollection:
      enabled: true
    liveProcessCollection:
      enabled: true
    logCollection:
      containerCollectAll: false
      enabled: false
    npm:
      enabled: false
    oomKill:
      enabled: true
    orchestratorExplorer:
      enabled: true
    otlp:
      receiver:
        protocols:
          grpc:
            enabled: false
          http:
            enabled: false
    processDiscovery:
      enabled: true
    prometheusScrape:
      enabled: false
    remoteConfiguration:
      enabled: false
    sbom:
      enabled: false
    tcpQueueLength:
      enabled: true
    usm:
      enabled: false
  global:
    clusterName: sandbox-apps
    credentials:
      apiSecret:
        keyName: api-key
        secretName: datadog-secret
      appSecret:
        keyName: app-key
        secretName: datadog-secret
    criSocketPath: /run/dockershim.sock
    logLevel: info
    podLabelsAsTags:
      app.kubernetes.io/env-name: env_name
      app.kubernetes.io/env-tag: env_tag
      app.kubernetes.io/team: team
      env: env
      env_name: env_name
      env_tag: env_tag
      statefulset.kubernetes.io/pod-name: pod_name
      team: team
    registry: our.own.registry.mirror/ecr-public/datadog
  override:
    clusterAgent:
      replicas: 2
    nodeAgent:
      env:
      - name: DD_EC2_PREFER_IMDSV2
        value: "true"
      - name: DD_COLLECT_EC2_TAGS
        value: "true"
      - name: DD_SECRET_BACKEND_COMMAND
        value: /readsecret_multiple_providers.sh
      extraConfd:
        configDataMap:
          kube_metrics_server.yaml: |-
            ad_identifiers:
              - metrics-server
            init_config:
            instances:
              - {"prometheus_url": "https://%%host%%:10250/metrics", "tls_verify": false}
      image:
        jmxEnabled: true
      tolerations:
      - operator: Exists

Operating System

2.1.5+bottlerocket

Other environment details

No response

[virtualroot]

We are seeing the same on EKS after upgrading our cluster and switching from AL2 to AL2023.

We are running v7.73.0, chart version v2.15.2.

Container runtime
containerd://2.1.5

Kubelet version
v1.33.5-eks-ecaa3a6

OS image
Amazon Linux 2023.9.20251208

[ajax-bychenok-y]

How critical is it? Does it influence on functionality?

[virtualroot]

@ajax-bychenok-y I say it influences billing and quotas more than anything else.
We created a filter in DataDog to work around that, but it's still unnecessarily consuming bandwidth.

[spikat]

Hi, could you please try using operator v1.21.0 or helm version >= 3.144.1? It should solve the issue, sorry for the inconvenience.
The addition of those logs revealed issues in both helm and operator setup, leading to a systematic process cgroup context resolution failure and spamming this log.
Operator PRs fixing the issue:
DataDog/datadog-operator#2323
DataDog/datadog-operator#2365

[ajax-bychenok-y]

try using operator v1.21.0

v2.16.0 of helm chart (there is 1.21.0 version of app inside) doen't fix the problem, same log.

[lebauce]

@ajax-bychenok-y Thank you for giving it a try, I'm sorry that this doesn't fix the issue. What features are you enabling in the DatagentAgent CRD ?

[ajax-bychenok-y]

@ajax-bychenok-y Thank you for giving it a try, I'm sorry that this doesn't fix the issue. What features are you enabling in the DatagentAgent CRD ?

Hi, @lebauce. I've provided full DatadogAgent CRD in a body of this PR. But I can add more details if needed.

[ajax-bychenok-y]

Guys, any update on this?

[cdonadeo]

Just found this issue while attempting to diagnose the same problem. Some information:

• AWS EKS, Kubernetes version 1.34
• Node OS Amazon Linux 2023.9.20251117
• Kernel version 6.12.55-74.119.amzn2023.x86_64
• Kubelet version v1.34.2-eks-ecaa3a6
• Helm chart datadog-operator-2.17.0 with all defaults (no values)
• Operator image gcr.io/datadoghq/operator:1.22.0
• Agent image gcr.io/datadoghq/agent:7.72.1

This is the CRD we are using:

---
apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    # You must set a cluster name to get the cluster to show up in Datadog's
    # Kubernetes infrastructure pages.
    clusterName: "<REDACTED>"
    kubelet:
      tlsVerify: false
    credentials:
      apiSecret:
        secretName: datadog-secret
        keyName: DATADOG_API_KEY
      appSecret:
        secretName: datadog-secret
        keyName: DATADOG_APP_KEY
  features:
    logCollection:
      enabled: true
      containerCollectAll: true
    oomKill:
      enabled: true
    kubeStateMetricsCore:
      enabled: true
    clusterChecks:
      enabled: true
      useClusterChecksRunners: true
    admissionController:
      enabled: true
      agentCommunicationMode: "socket"
    eventCollection:
      collectKubernetesEvents: true
    dogstatsd:
      unixDomainSocketConfig:
        enabled: true
  override:
    clusterAgent:
      replicas: 2
    nodeAgent:
      tolerations:
        - operator: "Exists" # Tolerate all taints

Symptom: the system-probe container in an agent pod spams messages similar to this:

2026-02-03 17:56:18 UTC | SYS-PROBE | WARN | (pkg/security/resolvers/cgroup/resolver.go:263 in resolvePidCgroupFallback) | Failed to add pid 2668223, error on fallback to resolve its cgroup: open /proc/2668205/task/2668205/cgroup: no such file or directory

For one pod, lines similar to this were emitted 2.93 million times in one hour.

[lebauce]

@ajax-bychenok-y @cdonadeo Sorry for the late reply and thank you for providing the CRD.
We were able to reproduce the problem, which seems to be that npm is automatically started even if is explicitely disabled in the CRD. npm requires the host cgroup filesystem to be mounted inside the system-probe, which is not the case.

We are actively working on a fix and will keep you posted.
As a workaround, you can disable the oomKill and tcpQueueLength feature (which are causing system-probe to be started)

[cdonadeo]

Confirmed:

    oomKill:
      # enabled: true
      enabled: false

does work around the issue for us. We've never enabled tcpQueueLength so did not need to disable it. Thank you very much for looking into this!