Get "dns request got empty response" and "port unreachable" after dns resolving

[rei-ber]

Hi,

I don't know, if this is the right place to ask for it, but we don't have any further ideas how to handle this problem.

We have a virtual machine for development with some podman pods. A couple of days ago, we migrated from Debian 11 to Debian 12 and from now on, we are using the newest podman, netavark and aardvark-dns versions. And since the migration we face the problem, that the dns resolution still works as you can see in the tshark/tcpdump. But the container reacts with an unreachable port. We also tested it, after a podman system reset. We dont' have a specific containers.conf. We use the default one under /usr/share/containers/.

The setup is as follows: We have a bridge network and a container which has to reach example.domain. This domain is only resolvable by our internal dns internal-dns.server and resolves to <ipv6-of-example.domain>. As I said, the resolution works, our internal dns resolves to the right address. But however the container reacts with an unreachable port icmp packet.

On Debian 11 with older versions (podman 5.3.0, netavark 1.13.0, aardvark-dns 1.16.0.) it worked and currently works on test machines which we not yet migrated.

The following shows what we are doing.

podman network create --subnet '10.90.1.0/24' --gateway 10.90.1.1 --ipv6  test

podman run --rm -it --network test docker.io/library/alpine:3.23.2
getent ahosts example.domain

podman network inspect test
[
     {
          "name": "test",
          "id": "4bccd6576fdc5afe24ffbbdcdb86e96507664925695dc8f06058cc83a589902d",
          "driver": "bridge",
          "network_interface": "podman1",
          "created": "2026-01-20T12:55:42.228154777Z",
          "subnets": [
               {
                    "subnet": "10.90.1.0/24",
                    "gateway": "10.90.1.1"
               },
               {
                    "subnet": "fd43:3e42:f505:6063::/64",
                    "gateway": "fd43:3e42:f505:6063::1"
               }
          ],
          "ipv6_enabled": true,
          "internal": false,
          "dns_enabled": true,
          "ipam_options": {
               "driver": "host-local"
          },
          "containers": {
               "4bb200a4cef50c79c952828d37b1dc86ab84026dfefd9dd3c08677fab0b24c67": {
                    "name": "ecstatic_brattain",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.90.1.5/24",
                                        "gateway": "10.90.1.1"
                                   },
                                   {
                                        "ipnet": "fd43:3e42:f505:6063::5/64",
                                        "gateway": "fd43:3e42:f505:6063::1"
                                   }
                              ],
                              "mac_address": "f6:dd:41:27:fe:18"
                         }
                    }
               }
          }
     }
]

$ podman unshare --rootless-netns tcpdump -l -i any host 10.90.1.1
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
13:31:06.188466 veth0 P   IP 10.90.1.5.44835 > 10.90.1.1.domain: 1432+ A? example.domain. (50)
13:31:06.188466 podman1 In  IP 10.90.1.5.44835 > 10.90.1.1.domain: 1432+ A? example.domain. (50)
13:31:06.188601 veth0 P   IP 10.90.1.5.44835 > 10.90.1.1.domain: 2612+ AAAA? example.domain. (50)
13:31:06.188601 podman1 In  IP 10.90.1.5.44835 > 10.90.1.1.domain: 2612+ AAAA? example.domain. (50)
13:31:06.689384 veth0 P   IP 10.90.1.5.44835 > 10.90.1.1.domain: 1432+ A? example.domain. (50)
13:31:06.689384 podman1 In  IP 10.90.1.5.44835 > 10.90.1.1.domain: 1432+ A? example.domain. (50)
13:31:06.689536 veth0 P   IP 10.90.1.5.44835 > 10.90.1.1.domain: 2612+ AAAA? example.domain. (50)
13:31:06.689536 podman1 In  IP 10.90.1.5.44835 > 10.90.1.1.domain: 2612+ AAAA? example.domain. (50)
13:31:07.857995 podman1 Out IP 10.90.1.1.domain > 10.90.1.5.44835: 2612* 1/0/0 AAAA <ipv6-of-example.domain> (78)
13:31:07.858006 veth0 Out IP 10.90.1.1.domain > 10.90.1.5.44835: 2612* 1/0/0 AAAA <ipv6-of-example.domain> (78)
13:31:07.858044 veth0 P   IP 10.90.1.5 > 10.90.1.1: ICMP 10.90.1.5 udp port 44835 unreachable, length 114
13:31:07.858044 podman1 In  IP 10.90.1.5 > 10.90.1.1: ICMP 10.90.1.5 udp port 44835 unreachable, length 114
13:31:07.858072 podman1 Out IP 10.90.1.1.domain > 10.90.1.5.44835: 1432* 0/1/0 (103)
13:31:07.858075 veth0 Out IP 10.90.1.1.domain > 10.90.1.5.44835: 1432* 0/1/0 (103)
13:31:07.858088 veth0 P   IP 10.90.1.5 > 10.90.1.1: ICMP 10.90.1.5 udp port 44835 unreachable, length 139
13:31:07.858088 podman1 In  IP 10.90.1.5 > 10.90.1.1: ICMP 10.90.1.5 udp port 44835 unreachable, length 139
13:31:08.359416 podman1 Out IP 10.90.1.1.domain > 10.90.1.5.44835: 2612* 1/0/0 AAAA <ipv6-of-example.domain> (78)
13:31:08.359425 veth0 Out IP 10.90.1.1.domain > 10.90.1.5.44835: 2612* 1/0/0 AAAA <ipv6-of-example.domain> (78)
13:31:08.359450 veth0 P   IP 10.90.1.5 > 10.90.1.1: ICMP 10.90.1.5 udp port 44835 unreachable, length 114
13:31:08.359450 podman1 In  IP 10.90.1.5 > 10.90.1.1: ICMP 10.90.1.5 udp port 44835 unreachable, length 114
13:31:08.359519 podman1 Out IP 10.90.1.1.domain > 10.90.1.5.44835: 1432* 0/1/0 (103)
13:31:08.359522 veth0 Out IP 10.90.1.1.domain > 10.90.1.5.44835: 1432* 0/1/0 (103)
13:31:08.359531 veth0 P   IP 10.90.1.5 > 10.90.1.1: ICMP 10.90.1.5 udp port 44835 unreachable, length 139
13:31:08.359531 podman1 In  IP 10.90.1.5 > 10.90.1.1: ICMP 10.90.1.5 udp port 44835 unreachable, length 139

$ podman unshare --rootless-netns tshark -i podman1 -Y "dns"
Running as user "root" and group "root". This could be dangerous.
Capturing on 'podman1'
 ** (tshark:55449) 13:30:49.993874 [Main MESSAGE] -- Capture started.
 ** (tshark:55449) 13:30:49.994004 [Main MESSAGE] -- File: "/var/tmp/wireshark_podman17RFVI3.pcapng"
    1 0.000000000    10.90.1.5 → 10.90.1.1    DNS 92 Standard query 0x0598 A example.domain
    2 0.000093199 fd43:3e42:f505:6063::5 → fd43:3e42:f505:6063::1 DNS 112 Standard query 0x0598 A example.domain
    3 0.000134999    10.90.1.5 → 10.90.1.1    DNS 92 Standard query 0x0a34 AAAA example.domain
    4 0.000164291 fd43:3e42:f505:6063::5 → fd43:3e42:f505:6063::1 DNS 112 Standard query 0x0a34 AAAA example.domain
    5 0.500917487    10.90.1.5 → 10.90.1.1    DNS 92 Standard query 0x0598 A example.domain
    6 0.501032060 fd43:3e42:f505:6063::5 → fd43:3e42:f505:6063::1 DNS 112 Standard query 0x0598 A example.domain
    7 0.501069847    10.90.1.5 → 10.90.1.1    DNS 92 Standard query 0x0a34 AAAA example.domain
    8 0.501096200 fd43:3e42:f505:6063::5 → fd43:3e42:f505:6063::1 DNS 112 Standard query 0x0a34 AAAA example.domain
    9 1.669410713 fd43:3e42:f505:6063::1 → fd43:3e42:f505:6063::5 DNS 165 Standard query response 0x0598 A example.domain SOA internal-dns.server
   10 1.669439507 fd43:3e42:f505:6063::5 → fd43:3e42:f505:6063::1 ICMPv6 213 Destination Unreachable (Port unreachable)
   11 1.669514828 fd43:3e42:f505:6063::1 → fd43:3e42:f505:6063::5 DNS 140 Standard query response 0x0a34 AAAA example.domain AAAA <ipv6-of-example.domain>
   12 1.669526793 fd43:3e42:f505:6063::5 → fd43:3e42:f505:6063::1 ICMPv6 188 Destination Unreachable (Port unreachable)
   13 1.669528476    10.90.1.1 → 10.90.1.5    DNS 120 Standard query response 0x0a34 AAAA example.domain AAAA <ipv6-of-example.domain>
   14 1.669578297    10.90.1.5 → 10.90.1.1    ICMP 148 Destination unreachable (Port unreachable)
   15 1.669605614    10.90.1.1 → 10.90.1.5    DNS 145 Standard query response 0x0598 A example.domain SOA internal-dns.server
   16 1.669621574    10.90.1.5 → 10.90.1.1    ICMP 173 Destination unreachable (Port unreachable)
   17 2.170949785    10.90.1.1 → 10.90.1.5    DNS 120 Standard query response 0x0a34 AAAA example.domain AAAA <ipv6-of-example.domain>
   18 2.170983957    10.90.1.5 → 10.90.1.1    ICMP 148 Destination unreachable (Port unreachable)
   19 2.171036657 fd43:3e42:f505:6063::1 → fd43:3e42:f505:6063::5 DNS 165 Standard query response 0x0598 A example.domain SOA internal-dns.server
   20 2.171052543    10.90.1.1 → 10.90.1.5    DNS 145 Standard query response 0x0598 A example.domain SOA internal-dns.server
   21 2.171064760    10.90.1.5 → 10.90.1.1    ICMP 173 Destination unreachable (Port unreachable)
   22 2.171068215 fd43:3e42:f505:6063::5 → fd43:3e42:f505:6063::1 ICMPv6 213 Destination Unreachable (Port unreachable)
   23 2.171092421 fd43:3e42:f505:6063::1 → fd43:3e42:f505:6063::5 DNS 140 Standard query response 0x0a34 AAAA example.domain AAAA <ipv6-of-example.domain>
   24 2.171105938 fd43:3e42:f505:6063::5 → fd43:3e42:f505:6063::1 ICMPv6 188 Destination Unreachable (Port unreachable)

$ journalctl --user SYSLOG_IDENTIFIER=aardvark-dns -f
Jan 20 13:31:07 hostname aardvark-dns[55178]: 2612 dns request got empty response
Jan 20 13:31:07 hostname aardvark-dns[55178]: 1432 dns request got empty response
Jan 20 13:31:07 hostname aardvark-dns[55178]: 2612 dns request got empty response
Jan 20 13:31:07 hostname aardvark-dns[55178]: 1432 dns request got empty response
Jan 20 13:31:08 hostname aardvark-dns[55178]: 1432 dns request got empty response
Jan 20 13:31:08 hostname aardvark-dns[55178]: 1432 dns request got empty response
Jan 20 13:31:08 hostname aardvark-dns[55178]: 2612 dns request got empty response
Jan 20 13:31:08 hostname aardvark-dns[55178]: 2612 dns request got empty response

Here is the filter table while the container is running:

$ podman unshare --rootless-netns /sbin/iptables -L -v -n -t filter
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  120 10432 NETAVARK_INPUT  0    --  *      *       0.0.0.0/0            0.0.0.0/0            /* netavark firewall rules */

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 NETAVARK_FORWARD  0    --  *      *       0.0.0.0/0            0.0.0.0/0            /* netavark firewall rules */

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain NETAVARK_FORWARD (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       0    --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
    0     0 ACCEPT     0    --  *      *       0.0.0.0/0            10.90.1.0/24         ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     0    --  *      *       10.90.1.0/24         0.0.0.0/0

Chain NETAVARK_INPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     17   --  *      *       10.90.1.0/24         0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     6    --  *      *       10.90.1.0/24         0.0.0.0/0            tcp dpt:53

Chain NETAVARK_ISOLATION_2 (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain NETAVARK_ISOLATION_3 (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       0    --  *      podman1  0.0.0.0/0            0.0.0.0/0
    0     0 NETAVARK_ISOLATION_2  0    --  *      *       0.0.0.0/0            0.0.0.0/0

This is the resolv.conf from within the container:

$ cat /etc/resolv.conf
search dns.podman devel.company.org company.org
nameserver 10.90.1.1
nameserver fd43:3e42:f505:6063::1
options timeout:1

$ podman info
host:
  arch: amd64
  buildahVersion: 1.42.2
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon_100:2.2.0-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.2.0, commit: ff908cce92cf89167b6b97ed240e91a6b147acc1'
  cpuUtilization:
    idlePercent: 90.3
    systemPercent: 2.53
    userPercent: 7.17
  cpus: 8
  databaseBackend: sqlite
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: journald
  freeLocks: 2047
  hostname: <hostname>
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 2499
      size: 1
    - container_id: 1
      host_id: 1214112
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 2712
      size: 1
    - container_id: 1
      host_id: 1214112
      size: 65536
  kernel: 6.1.0-42-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 11136004096
  memTotal: 16768479232
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: podman-aardvark-dns_100:1.17.0-1_amd64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.17.0
    package: podman-netavark_100:1.17.1-1_amd64
    path: /usr/libexec/podman/netavark
    version: netavark 1.17.1
  ociRuntime:
    name: crun
    package: crun_100:1.26-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.26
      commit: 3241e671f92c33b0c003cd7de319e4f32add6231
      rundir: /run/user/2712/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_100:0.0+20251223.2ba9fd58-1_amd64
    version: |
      pasta 0.0+20251223.2ba9fd58
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/2712/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_100:1.3.3-1_amd64
    version: |-
      slirp4netns version 1.3.3
      commit: unknown
      libslirp: 4.9.1
      SLIRP_CONFIG_VERSION_MAX: 6
      libseccomp: 2.5.4
  swapFree: 3998978048
  swapTotal: 4000313344
  uptime: 25h 28m 37.00s (Approximately 1.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  our-registry.domain:
    Blocked: false
    Insecure: false
    Location: our-registry.domain
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: our-registry.domain
    PullFromMirror: ""
  search:
  - our-registry.domain
store:
  configFile: /home/username/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/username/containers/storage
  graphRootAllocated: 58627768320
  graphRootUsed: 24927387648
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/2712/containers
  transientStore: false
  volumePath: /home/username/containers/storage/volumes
version:
  APIVersion: 5.7.1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.25.5
  Os: linux
  OsArch: linux/amd64
  Version: 5.7.1

We appreciate it, if you have any idea what happens here. If you need more information, please let me know :)