Network issues on boot after upgrading to 1.12+

[nint8835]

I'm on attempt number two to upgrade to 1.12 and am running into the same issue on 1.12.4 as I did on 1.12.0 - on boot, pods seem like they may be starting midway through the Flannel startup process, or something like that? It manifests as pods such as my ingress controller never successfully receiving any liveness or readiness probes. To resolve I've got to delete the Flannel pod, wait for it to come up, then delete the impacted pod.

The pods started during this time have the error
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cf21ae921e096b3bea3f0c405fa08bd0db1048c96674e33527a5f01c89c3a3f1": plugin type="flannel" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory

I threw Claude Code at the problem and let it poke around in my cluster for a bit before posting here and its leading theory is 1.12 has sped up CRI startup times making an existing race condition more prevalent, and it seems reasonable but I'm not sure how true that is (it initially thought it was something related to 1.12 trusting KVM's network device but I could find no evidence of that in the docs or changelog and it came back with this after). Regardless, its claim is that:

• Flannel takes time to populate /run/flannel/subnet.env
• The CRI starts trying to create pod sandboxes immediately
• This results in pods launched in this time getting broken networking

The suggestion was to use machine.files to write this file on boot automatically but I figure there's zero chance that's the right answer 😆

If it's relevant:

• I'm running the cluster in VMs in Proxmox
• Standard Talos + Flannel networking config, have done nothing besides set the machines hostnames
• Running 3 control planes + 5 workers
• This was encountered reliably when attempting to upgrade any worker from v1.11.6 to v1.12.0 and v1.12.4
• Cluster was initially spun up on 1.6.7 and has been upgraded between versions since
• Using the standard ghcr.io/siderolabs/installer:v1.12.4 installer image, no addons
• My machine configs are generated and managed via Terraform and that code can be found here in cluster.tf and modules/node. An example complete machineconfig is:

version: v1alpha1
debug: false
persist: true
machine:
    type: worker
    token: REDACTED
    ca:
        crt: REDACTED
        key: ""
    certSANs: []
    kubelet:
        image: ghcr.io/siderolabs/kubelet:v1.34.0
        defaultRuntimeSeccompProfileEnabled: true
        disableManifestsDirectory: true
    network:
        hostname: k8s-worker-1
    install:
        disk: /dev/sda
        image: ghcr.io/siderolabs/installer:v1.12.0
        wipe: false
    features:
        rbac: true
        stableHostname: true
        apidCheckExtKeyUsage: true
        diskQuotaSupport: true
        kubePrism:
            enabled: true
            port: 7445
        hostDNS:
            enabled: true
            forwardKubeDNSToHost: true
    nodeLabels:
        topology.kubernetes.io/region: hera
        topology.kubernetes.io/zone: aphrodite
cluster:
    id: REDACTED
    secret: REDACTED
    controlPlane:
        endpoint: https://cluster.ops.bootleg.technology:6443
    clusterName: hera
    network:
        dnsDomain: cluster.local
        podSubnets:
            - 10.244.0.0/16
        serviceSubnets:
            - 10.96.0.0/12
    token: REDACTED
    ca:
        crt: REDACTED
        key: ""
    discovery:
        enabled: true
        registries:
            kubernetes:
                disabled: true
            service: {}

• I don't have any sort of really fancy networking setup, the VMs are bridged to my UniFi network and the only tweak I've done is have Proxmox tag the VLAN for me to put them in a dedicated Kubernetes VLAN.
• A friend with a very similar setup to me has tested and is not running into this

[smira]

The error:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cf21ae921e096b3bea3f0c405fa08bd0db1048c96674e33527a5f01c89c3a3f1": plugin type="flannel" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory

Is completely normal as Flannel hasn't started yet, and kubelet will keep restarting the pod until it succeeds to run (once Flannel initializes itself).

So what is the issue with Flannel? You need to grab more logs to understand things in a "broken" state.

[smira]

I guess you need to dig further into the logs, we do test upgrades, and Flannel of course. We haven't seen this issue.

Flannel seems to be happy in both cases, so what does block your pod from being restarted - it's hard to say with the information you provided.

[nint8835]

Further details I've been able to gather:

• I did a reset and reinstalled with the metal-amd64.iso from the 1.12.5 release on GitHub and after the node was freshly installed it was working fine
• However the dashboard was reporting 1.12.0 as the installed version (ISO reported 1.12.5 during maintenance mode so my guess is this is something Terraform did) so I did an upgrade to 1.12.5 via talosctl upgrade --nodes 10.8.2.5 --image ghcr.io/siderolabs/installer:v1.12.5 and after that upgrade it broke again

I'll see about the feasibility of getting a minimal repro for this this weekend, but it seems almost like it's something with me upgrading to 1.12.0+ via talosctl and not anything about 1.12.0+ itself?

[nint8835]

Never mind, it works right after a fresh install but a reboot breaks it as well. I'll keep digging

[smira]

If the reboot breaks it, the only idea I can come up with is bad hardware/firmware which is incompatible with kexec. Not sure how it manifests as Flannel issue though.

You can try testing with reboot --mode=powercycle.

[nint8835]

No luck, unfortunately.

• On 1.12.0, doing an upgrade to 1.12.5 with talosctl upgrade --nodes 10.8.2.5 --image ghcr.io/siderolabs/installer:v1.12.5 --reboot-mode powercycle -> breaks
• On 1.12.0, doing a reboot with talosctl reboot -n 10.8.2.5 --mode powercycle -> breaks
• On 1.12.5, downgrading to 1.11.6 with talosctl upgrade --nodes 10.8.2.5 --image ghcr.io/siderolabs/installer:v1.11.6 --reboot-mode powercycle -> fixes it

I'll continue taking a look at the bad hardware / firmware route though, I'm running in Proxmox so presumably the bad hardware / firmware would be the virtual devices exposed to the VM rather than my physical hardware. I'll compare my VM config to the VMs of my friends with similar setups who are not encountering this issue.