dns request failed: request timed out

[MartinX3]

I'm using arch linux, so the packages should have the newest version.

I'm using firewallD and rootless podman with netavark and aardvark-dns.

I understand, that rootless podman with netavark won't manage my firewallD, but I would like to know which rules I need to activate to avoid the spam in my journal.
And if the rule need to be in my loopback or network interface. (Also if it is enough to allow communication with the host instead of having an open port in the internet.

My dns resolver is systemd-resolved

$ ls -lha /etc/resolv.conf
lrwxrwxrwx 1 root root 39 31. Okt 10:22 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

My journal spam:
aardvark-dns[6156]: 21433 dns request failed: request timed out

The rootless container itself can ping to google.com.
I didn't test if they can ping to a container dns name.

[MartinX3]

Ah, after turning /etc/resolv.conf again into a symlink of systemd-resolved
sudo ln -rsf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
the messages disappear.
I assume it wants now to call the localhost ip of the systemd-resolved instead of the network ip of the outside resolver.

I don't know how I could let this plugin verbose debug logging into the journal, but I'm glad that it is fixed now.

[MartinX3]

And it's back
dns request failed: request timed out

It seems to happen if I execute nslookup in a container.
But I get a result. Weird.

[Szwendacz99]

I detected the same problem on my raspberry pi with Fedora and rootless podman containers.

[jmaris]

Having the exact same issue. DNS lookup is extremely unstable, only two thirds of lookups work.

This is on Fedora Silverblue 37 using rootless containers and the new networking stack.

[MartinX3]

Now using a nextcloud container which communicates with a PostgreSQL and a LDAP container.
I access the nextcloud container with a nginx reverse proxy container from the internet.

1/3 of the nextcloud web interface results into a 502 bad gateway, because of the unstable DNS.
And my logs are getting spammed with

aardvark-dns[3617]: 50310 dns request failed: request timed out

I hope the next podman release will include the new network stack which hopefully fixes this issue.

[baude]

It is impossible to help with these issues as reporters did not provide versions for podman, netavark, and aarvark-dns. please provide as much relevant information as possible.

[MartinX3]

podman: 4.3.1
netavark: 1.4.0
aardvark-dns: 1.4.0

[baude]

would you say your machine/vm is high performent or maybe has slow IO/processor/RAM limitations? I'm trying to understand if you might have a race.

[MartinX3]

The first bare metal server server with this issue has the HDD connected to SATA with the server.
CPU: Intel Xeon E3-1231 v3 - 3.4 GHz - 4 core(s)
RAM: 32GB - DDR3

The second bare metal server has a 870 Evo SSD connected via SATA to the server.
CPU: AMD Phenom 2 955 X4 3.2 GHz 4 cores
RAM: 6GB - DDR2

The second server spams this issue many times.
The first server spams it less often but also regularly.

[baude]

@flouthoc wdyt?

[MartinX3]

Now my faster server spammed it at night while the server pods weren't used by clients.

[flouthoc]

We used to see similar issues in older versions of netavark and aardvark in Podman CI as well but it was fixed in newer versions with #220 but I guess there might be some issue which is not being reproduced in our CI, I'll try to reproduce this locally and see if i can reproduce this.