Rootless podman trying to join a stale process PID from pause.pid and display `cannot set user namespace` error

[fangpenlin]

Issue Description

When you're running rootless podman, the first time it will create pause.pid in the runtime folder. For example, /tmp/storage-run-1000/libpod/tmp/pause.pid. But if the pid is actually stale, the original pause process created by podman rootless was no longer there, however, there might be another process happen to have the same pid, you will see an error like:

cannot set user namespace

The problem can be resolved by removing the stale pid as described here

#10135 (comment)

But it still seems like a bug to me, as I would expect podman to know that the process pid is stale and fallback to creating a new one and clear the stale one.

Steps to reproduce the issue

Steps to reproduce the issue

1. Setup an environment to run podman in rootless mode (update /etc/subuid and /etc/subgid, create new user and etc)
2. Add a pause.pid file at the expected location, like, for example /tmp/storage-run-1000/libpod/tmp/pause.pid, write a pid pointing to a running process that's not the a valid pause created by podman
3. Run almost any podman cmd, even as simple as such as podman system info with the rootless user you've just created
4. You should see cannot set user namespace error as the podman in root

I created a repo with docker compose in it to make reproducing the bug much easier:

https://github.com/fangpenlin/podman-stale-pause-pid-bug

Usage, build the image:

docker build -t podman-stale-pause-pid-bug .

Run with Compose:

docker compose up

Describe the results you received

cannot set user namespace

Describe the results you expected

podman should have a way to tell if the processing provided by the pause.pid file an actual pause process created by podman to help speeding up the exec. and if the pid is already stale, it should not print cannot set user namespace and exit, it should remove the pid file and create a new one

podman info output

podman info
host:
  arch: amd64
  buildahVersion: 1.42.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.13+ds1-2_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: unknown'
  cpuUtilization:
    idlePercent: 85.72
    systemPercent: 3.79
    userPercent: 10.48
  cpus: 4
  databaseBackend: sqlite
  distribution:
    codename: forky
    distribution: debian
    version: unknown
  eventLogger: file
  freeLocks: 2048
  hostname: makerrepo-web-worker-85f59c568c-kdrzt
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.1.158-180.294.amzn2023.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 614301696
  memTotal: 16558186496
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
    package: netavark_1.16.1-3.1_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.16.1
  ociRuntime:
    name: runc
    package: runc_1.3.3+ds1-2_amd64
    path: /usr/bin/runc
    version: |-
      runc version 1.3.3+ds1
      commit: 1.3.3+ds1-2
      spec: 1.2.1
      go: go1.24.9
      libseccomp: 2.5.4
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: true
    path: /tmp/storage-run-1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 1420h 47m 43.00s (Approximately 59.17 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  localhost:7000:
    Blocked: false
    Insecure: true
    Location: localhost:7000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:7000
    PullFromMirror: ""
store:
  configFile: /home/makerrepo/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.18.1
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev
  graphRoot: /home/makerrepo/.local/share/containers/storage
  graphRootAllocated: 137359241216
  graphRootUsed: 49960837120
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /tmp/storage-run-1000/containers
  transientStore: true
  volumePath: /home/makerrepo/.local/share/containers/storage/volumes
version:
  APIVersion: 5.7.0
  BuildOrigin: Debian
  Built: 1764725792
  BuiltTime: Wed Dec  3 01:36:32 2025
  GitCommit: ""
  GoVersion: go1.24.9
  Os: linux
  OsArch: linux/amd64
  Version: 5.7.0

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[kuzmich321]

+1

I integrate rootless podman in the build system called ya make. And I also found that when you call podman image load it fails the first time because pause.pid seems to be missing. Command fails, but you can call it one more time and then it succeeds

[giuseppe]

that is expected. How was the pause process terminated?

Is /tmp on a tmpfs? Did it survive a reboot?

[kuzmich321]

that is expected. How was the pause process terminated?

Is /tmp on a tmpfs? Did it survive a reboot?

For me it's on completely clean dir. We have a so-called BUILD_ROOT for our building system (it's a temporary dir to store outputs while something is being built). So, then we create a dirstructure and configs for podman. Then we initiate env vars based on this dir structure.

Env values we provide for podman:

• XDG_RUNTIME_DIR
• XDG_CONFIG_HOME
• XDG_DATA_HOME

And when we do podman image load -i my_image.tar -q it fails (strace shows that it tries to read pause.pid which is not even present at that time). And the workaround right now: is to run this command twice so the second one does not fail due to the fact that pause.pid file is present

JYI: I found that it became an issue only on ubuntu 24.04+, on earlier versions it seems to be working correctly

[giuseppe]

it tries to read it, but if it is not found then the pause process is created.

What happens after? After the first podman command, is the pause process killed somehow and there is a pause.pid that refers to the old process?

[fangpenlin]

Is /tmp on a tmpfs? Did it survive a reboot?

My use case is a bit special. I have a Kubernetes deployment, where I need to run pull image first in an initialContainer to ensure it's there, and the volume will be inherit by a normal container which anticipates the image to be there. It's not on a tmpfs as I need the second container to see the files left by the fist initial container. That's why you would see pause.pid survive

I wonder if there are other situations, where the pause.pid could survive reboot, and the pid is reused by another process 🤔 it seems like there's no strong guarantee that files in /tmp will be gone after reboot for Linux OS. If not, there's a chance you could run into this situation.

[giuseppe]

My use case is a bit special. I have a Kubernetes deployment, where I need to run pull image first in an initialContainer to ensure it's there, and the volume will be inherit by a normal container which anticipates the image to be there. It's not on a tmpfs as I need the second container to see the files left by the fist initial container. That's why you would see pause.pid survive

you only need the storage root directory to be persistent, the rundir must be on a tmpfs that is not recycled across different environments (like two different containers).

I wonder if there are other situations, where the pause.pid could survive reboot, and the pid is reused by another process 🤔 it seems like there's no strong guarantee that files in /tmp will be gone after reboot for Linux OS. If not, there's a chance you could run into this situation.

using /tmp is a fallback mechanism, by default podman uses XDG_RUNTIME_DIR (that usually points to /run).

If you use the same store across different environments, you need to make sure the rundir is not reused

[fangpenlin]

I see, you're right. I probably shouldn't let the runtime dir persists between containers. If there are more people running into similar situation, we may need to consider about how to improve it. Given this is expected behavior and there are more people running into the same situations, an improved error message might be helpful. Otherwise, please feel free to close this issue, and thanks for explaining.

[giuseppe]

opened a PR:

• rootless: detect and remove stale pause.pid with recycled PIDs #28323