-
Notifications
You must be signed in to change notification settings - Fork 3k
Description
Issue Description
If you have your own registry, it's more efficient to make a single request for a scoped token for multiple images to your authorization endpoint and then let the OCI client perform a direct pull to the registry using the Authorization: Bearer <token>.
In docker/moby, this is enabled by the registrytoken option for requests that require authentication, however this value is ignored by podman, despite being accepted as input.
This is somewhat related to #25609 although the error reported in that case is different
Steps to reproduce the issue
- Get a scoped token for an image from your auth server
- Try to pull the image
REGISTRY="registry.my-domain.com"
IMAGE_PATH="my-image"
TOKEN="my-scoped-bearer-token"
AUTH_HEADER=$(jq -n x--arg tok "$TOKEN" \
'{registrytoken: $tok}' | base64 -w 0)
# Combine into a full reference
FULL_REF="$REGISTRY/v2/$IMAGE_PATH@sha256:my-image-digest"
# Encode the URI using jq's @uri filter
ENCODED_REF=$(jq -rn --arg ref "$FULL_REF" '$ref | @uri')
# Execute the pull via the socket
curl -v --unix-socket /var/run/docker.sock \
-X POST \
-H "X-Registry-Auth: $AUTH_HEADER" \
"localhost/images/create?fromImage=$ENCODED_REF"
Describe the results you received
Podman returns a 403 error
* Connected to localhost (/var/run/docker.sock) port 0
* using HTTP/1.x
> POST /images/create?fromImage=registry2.balena-cloud.com%2Fv2%2Fmy-image%40sha256%3Amy-image-digest HTTP/1.1
> Host: localhost
> User-Agent: curl/8.14.1
> Accept: */*
> X-Registry-Auth: my-scoped-bearer-token
>
* Request completely sent off
< HTTP/1.1 403 Forbidden
< Api-Version: 1.41
< Content-Type: application/json
< Libpod-Api-Version: 5.7.1
< Server: Libpod/5.7.1 (linux)
< X-Reference-Id: 0x400011cbb8
< Date: Fri, 20 Feb 2026 18:41:24 GMT
< Transfer-Encoding: chunked
<
{"message":"denied: requested access to the resource is denied"}
* Connection #0 to host localhost left intact
Service logs show the service trying to perform an authorization workflow instead of using the token directly
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Looking up image \"registry.my-domain.com/v2/my-image@sha256:my-image-digest\" in local containers storage"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Normalized platform linux/arm64 to {arm64 linux [] }"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Trying \"registry.my-domain.com/v2/my-image@sha256:my-image-digest\" ..."
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="reference \"[overlay@/var/home/core/.local/share/containers/storage+/run/user/501/containers]registry.my-domain.com/v2/my-image@sha256:my-image-digest\" does not resolve to an image ID"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Trying \"registry.my-domain.com/v2/my-image@sha256:my-image-digest\" ..."
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="reference \"[overlay@/var/home/core/.local/share/containers/storage+/run/user/501/containers]registry.my-domain.com/v2/my-image@sha256:my-image-digest\" does not resolve to an image ID"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Trying \"registry.my-domain.com/v2/my-image@sha256:my-image-digest\" ..."
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Pulling image registry.my-domain.com/v2/my-image@sha256:my-image-digest (policy: always)"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Looking up image \"registry.my-domain.com/v2/my-image@sha256:my-image-digest\" in local containers storage"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Normalized platform linux/arm64 to {arm64 linux [] }"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Trying \"registry.my-domain.com/v2/my-image@sha256:my-image-digest\" ..."
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="reference \"[overlay@/var/home/core/.local/share/containers/storage+/run/user/501/containers]registry.my-domain.com/v2/my-image@sha256:my-image-digest\" does not resolve to an image ID"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Trying \"registry.my-domain.com/v2/my-image@sha256:my-image-digest\" ..."
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="reference \"[overlay@/var/home/core/.local/share/containers/storage+/run/user/501/containers]registry.my-domain.com/v2/my-image@sha256:my-image-digest\" does not resolve to an image ID"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Trying \"registry.my-domain.com/v2/my-image@sha256:my-image-digest\" ..."
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\""
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/000-shortnames.conf\""
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/999-podman-machine.conf\""
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Normalized platform linux/arm64 to {arm64 linux [] }"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Attempting to pull candidate registry.my-domain.com/v2/my-image@sha256:my-image-digest for registry.my-domain.com/v2/my-image@sha256:my-image-digest"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="parsed reference into \"[overlay@/var/home/core/.local/share/containers/storage+/run/user/501/containers]registry.my-domain.com/v2/my-image@sha256:my-image-digest\""
Feb 20 15:34:11 localhost.localdomain podman[1693]: Trying to pull registry.my-domain.com/v2/my-image@sha256:my-image-digest...
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Copying source image //registry.my-domain.com/v2/my-image@sha256:my-image-digest to destination image [overlay@/var/home/core/.local/share/containers/storage+/run/user/501/containers]registry.my-domain.com/v2/my-image@sha256:my-image-digest"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Using registries.d directory /etc/containers/registries.d"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Trying to access \"registry.my-domain.com/v2/my-image@sha256:my-image-digest\""
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="No credentials matching registry.my-domain.com/v2/my-image found in /run/user/501/containers/auth.json"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="No credentials matching registry.my-domain.com/v2/my-image found in /var/home/core/.config/containers/auth.json"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="No credentials matching registry.my-domain.com/v2/my-image found in /var/home/core/.docker/config.json"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="No credentials matching registry.my-domain.com/v2/my-image found in /var/home/core/.dockercfg"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="No credentials for registry.my-domain.com/v2/my-image found"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg=" No signature storage configuration found for registry.my-domain.com/v2/my-image@sha256:my-image-digest, using built-in default file:///var/home/core/.local/share/containers/sigstore"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.my-domain.com"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="GET https://registry.my-domain.com/v2/"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="Ping https://registry.my-domain.com/v2/ status 401"
Feb 20 15:34:11 localhost.localdomain podman[1693]: time="2026-02-20T15:34:11-03:00" level=debug msg="GET https://api.my-domain.com/auth/v1/token?scope=repository%3Av2%2Fmy-image%3Apull&service=registry.my-domain.com"
Feb 20 15:34:12 localhost.localdomain podman[1693]: time="2026-02-20T15:34:12-03:00" level=debug msg="Increasing token expiration to: 60 seconds"
Feb 20 15:34:12 localhost.localdomain podman[1693]: time="2026-02-20T15:34:12-03:00" level=debug msg="GET https://registry.my-domain.com/v2/v2/my-image/manifests/sha256:my-image-digest"
Feb 20 15:34:12 localhost.localdomain podman[1693]: time="2026-02-20T15:34:12-03:00" level=debug msg="Detected insufficient_scope error, will retry request with updated scope"
Feb 20 15:34:12 localhost.localdomain podman[1693]: time="2026-02-20T15:34:12-03:00" level=debug msg="GET https://api.my-domain.com/auth/v1/token?scope=repository%3Av2%2Fmy-image%3Apull&scope=repository%3Av2%2Fmy-image%3Apull&service=registry.my-domain.com"
Feb 20 15:34:12 localhost.localdomain podman[1693]: time="2026-02-20T15:34:12-03:00" level=debug msg="Increasing token expiration to: 60 seconds"
Feb 20 15:34:12 localhost.localdomain podman[1693]: time="2026-02-20T15:34:12-03:00" level=debug msg="GET https://registry.my-domain.com/v2/v2/my-image/manifests/sha256:my-image-digest"
Feb 20 15:34:13 localhost.localdomain podman[1693]: time="2026-02-20T15:34:13-03:00" level=debug msg="Content-Type from manifest GET is \"application/json\""
Feb 20 15:34:13 localhost.localdomain podman[1693]: time="2026-02-20T15:34:13-03:00" level=debug msg="Discarding non-primary errors:"
Feb 20 15:34:13 localhost.localdomain podman[1693]: time="2026-02-20T15:34:13-03:00" level=debug msg=" unauthorized: authentication required"
Feb 20 15:34:13 localhost.localdomain podman[1693]: time="2026-02-20T15:34:13-03:00" level=debug msg="Accessing \"registry.my-domain.com/v2/my-image@sha256:my-image-digest\" failed: reading manifest sha256:my-image-digest in registry.my-domain.com/v2/my-image: requested access to the resource is denied"
Feb 20 15:34:13 localhost.localdomain podman[1693]: time="2026-02-20T15:34:13-03:00" level=debug msg="Error pulling candidate registry.my-domain.com/v2/my-image@sha256:my-image-digest: unable to copy from source docker://registry.my-domain.com/v2/my-image@sha256:my-image-digest: initializing source docker://registry.my-domain.com/v2/my-image@sha256:my-image-digest: reading manifest sha256:my-image-digest in registry.my-domain.com/v2/my-image: requested access to the resource is denied"
Feb 20 15:34:13 localhost.localdomain podman[1693]: @ - - [20/Feb/2026:15:34:11 -0300] "POST /images/create?fromImage=registry.my-domain.com%2Fv2%2Fmy-image%40sha256%3Amy-image-digest HTTP/1.1" 403 65 "" "curl/8.14.1"
Feb 20 15:34:13 localhost.localdomain podman[1693]: time="2026-02-20T15:34:13-03:00" level=debug msg="IdleTracker:idle 1m+0h/1t connection(s)" X-Reference-Id=0x400011c2c8
Feb 20 15:34:13 localhost.localdomain podman[1693]: time="2026-02-20T15:34:13-03:00" level=debug msg="IdleTracker:closed 1m+0h/1t connection(s)" X-Reference-Id=0x400011c2c8
Feb 20 15:34:13 localhost.localdomain podman[1693]: 2026-02-20 15:34:13.346601332 -0300 -03 m=+1.878803469 image pull-error registry.my-domain.com/v2/my-image@sha256:my-image-digest unable to copy from source docker://registry.my-domain.com/v2/my-image@sha256:my-image-digest: initializing source docker://registry.my-domain.com/v2/my-image@sha256:my-image-digest: reading manifest sha256:my-image-digest in registry.my-domain.com/v2/my-image: requested access to the resource is denied
Describe the results you expected
The image pull should start, here is what happens with docker
* Connected to localhost (/Users/felipe/.docker/run/docker.sock) port 0
* using HTTP/1.x
> POST /images/create?fromImage=registry.my-domain.com%2Fv2%2Fmy-image%40sha256%3Amy-image-digest HTTP/1.1
> Host: localhost
> User-Agent: curl/8.14.1
> Accept: */*
> X-Registry-Auth: ewogICJyZWdpc3RyeXRva2VuIjogImV5SmhiR2NpT2lKRlV6STFOaUlzSW5SNWNDSTZJa3BYVkNJc0ltdHBaQ0k2SWxKWlZGazZUbEUzV0RwS1NEVkNPbEZGV0ZrNlJrVTJUanBMVGxWVk9rbFdOVEk2VEZGUlFUbzNValJXT2pKVlVGSTZRazlJU2pwRE5rbFBJbjAuZXlKcWRHa2lPaUl3T1RrNU1XUTJPUzB5WkRGaExUUXpZbU10T0RjNE5DMDFOekU1TlRjd05ERTJZemtpTENKdVltWWlPakUzTnpFMk1UTXhNREFzSW1GalkyVnpjeUk2VzNzaWJtRnRaU0k2SW5ZeUwyRmhPV0UwWlRSa05tRTFaakl3Wm1RMlpHTTJZMkU1Wm1ZNVpHUTBaVEpoSWl3aWRIbHdaU0k2SW5KbGNHOXphWFJ2Y25raUxDSmhZM1JwYjI1eklqcGJJbkIxYkd3aVhYMWRMQ0pwWVhRaU9qRTNOekUyTVRNeE1UQXNJbVY0Y0NJNk1UYzNNVFkwTVRreE1Dd2lZWFZrSWpvaWNtVm5hWE4wY25reUxtSmhiR1Z1WVMxamJHOTFaQzVqYjIwaUxDSnBjM01pT2lKaGNHa3VZbUZzWlc1aExXTnNiM1ZrTG1OdmJTSXNJbk4xWWlJNklpSjkuWmNod1pxcVdZcVc1bzRSbHV3NHZmLThoQVVzMjFmaThxM1I2TFFnZDJaejkwOGlaeW1vS3lUMG5xR3Rac013WlZ0U3p6ZkthRzcyTzVVRjlOeTUyd2ciCn0K
>
* Request completely sent off
< HTTP/1.1 200 OK
< Api-Version: 1.52
< Content-Type: application/json
< Date: Fri, 20 Feb 2026 18:45:13 GMT
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/29.1.3 (linux)
< Transfer-Encoding: chunked
<
{"status":"Pulling from v2/my-image","id":"registry.my-domain.com/v2/my-image@sha256:my-image-digest"}
podman info output
Client:
APIVersion: 5.7.1
BuildOrigin: brew
Built: 1765311063
BuiltTime: Tue Dec 9 17:11:03 2025
GitCommit: ""
GoVersion: go1.25.5
Os: darwin
OsArch: darwin/arm64
Version: 5.7.1
host:
arch: arm64
buildahVersion: 1.42.2
cgroupControllers:
- cpu
- io
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.1.13-2.fc43.aarch64
path: /usr/bin/conmon
version: 'conmon version 2.1.13, commit: '
cpuUtilization:
idlePercent: 99.92
systemPercent: 0.05
userPercent: 0.03
cpus: 5
databaseBackend: sqlite
distribution:
distribution: fedora
variant: coreos
version: "43"
emulatedArchitectures:
- linux/386
- linux/amd64
- linux/arm64be
eventLogger: journald
freeLocks: 2048
hostname: localhost.localdomain
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 1000000
uidmap:
- container_id: 0
host_id: 501
size: 1
- container_id: 1
host_id: 100000
size: 1000000
kernel: 6.17.7-300.fc43.aarch64
linkmode: dynamic
logDriver: journald
memFree: 1557405696
memTotal: 2038820864
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: aardvark-dns-1.17.0-1.fc43.aarch64
path: /usr/libexec/podman/aardvark-dns
version: aardvark-dns 1.17.0
package: netavark-1.17.1-1.fc43.aarch64
path: /usr/libexec/podman/netavark
version: netavark 1.17.1
ociRuntime:
name: crun
package: crun-1.24-1.fc43.aarch64
path: /usr/bin/crun
version: |-
crun version 1.24
commit: 54693209039e5e04cbe3c8b1cd5fe2301219f0a1
rundir: /run/user/501/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
os: linux
pasta:
executable: /usr/sbin/pasta
package: passt-0^20250919.g623dbf6-1.fc43.aarch64
version: |
pasta 0^20250919.g623dbf6-1.fc43.aarch64-pasta
Copyright Red Hat
GNU General Public License, version 2 or later
<https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
remoteSocket:
exists: true
path: unix:///run/user/501/podman/podman.sock
rootlessNetworkCmd: pasta
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: true
slirp4netns:
executable: /usr/sbin/slirp4netns
package: slirp4netns-1.3.1-3.fc43.aarch64
version: |-
slirp4netns version 1.3.1
commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
libslirp: 4.9.1
SLIRP_CONFIG_VERSION_MAX: 6
libseccomp: 2.6.0
swapFree: 0
swapTotal: 0
uptime: 1h 5m 55.00s (Approximately 0.04 days)
variant: v8
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- docker.io
store:
configFile: /var/home/core/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /var/home/core/.local/share/containers/storage
graphRootAllocated: 106769133568
graphRootUsed: 4254433280
graphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "true"
Supports d_type: "true"
Supports shifting: "false"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 0
runRoot: /run/user/501/containers
transientStore: false
volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
APIVersion: 5.7.1
BuildOrigin: 'Copr: packit/containers-podman-27732'
Built: 1765238400
BuiltTime: Mon Dec 8 21:00:00 2025
GitCommit: f845d14e941889ba4c071f35233d09b29d363c75
GoVersion: go1.25.4 X:nodwarf5
Os: linux
OsArch: linux/arm64
Version: 5.7.1Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
Using podman-machine on Mac
Additional information
Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting