Install of Talos on OVHCloud dedicated server with private(vrack) and public interface

[morind]

Hi,
I've been struggling with a problem for several days.
I want to install Talos onto one OVH dedicated server with the default public interface and one interface linked to the "vrack" offer proposed by OVH (Level 2 routing).
I want to use a dedicated VLAN (VLAN ID: 3 in my case)
After lot of investigations and watching plenty of issues on github... I'm a bit lost ;(
I use metal platform.
The first boot is ok. I enforce the bootLoaderPath with OVHCloud API to boot my server with talos and I generate the image with a custom "schematic"
I've tried a basic one, with just the platform "metal" and a more complicated schematic conf based on Metal Network Configuration

I've tried with Talos 1.9.6 and 1.11.1. And with 1.11.1 with the default CNI (Flannel) and now (my target) with Cilium.
At the end, Talos is well bootstrapped. Cilium is OK.
Connectivity is ok with the "public" world from the default interface but I can't reach anything on my private network.
Here after traces in a "netshoot" Pod (arping and tcpdump performed from within a container in the Pod)

arping -c 1 10.3.1.241
ARPING 10.3.1.241 from 10.3.1.240 enp1303449.3
Sent 1 probes (1 broadcast(s))
Received 0 response(s)

18:08:51.262905 xx:xx:xx:c5:cd > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP (0x0806), Request who-has 10.3.1.241 (ff:ff:ff:ff:ff:ff) tell 10.3.1.240, length 28

The VLAN ID 3 seems well defined in the request and sent to the right interface but not response nonetheless.
The IP 10.3.1.241 is the other dedicated server on Ubuntu and 10.3.1.240 is the IP of my Talos server I've enforced.
Subnet is 10.3.0.0/16
I don't want/have to use the gateway 10.3.0.1 for now.
List of (relevant) interfaces:

8: ens3f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether xx:xx:xx:xx:2d:9a brd ff:ff:ff:ff:ff:ff
    altname enp193s0f0np0
    altname enxb8e9241c2d9a
    inet XX:XX:XX.235/32 metric 1024 brd XX:XX:XX.235 scope global ens3f0np0
       valid_lft forever preferred_lft forever
    inet6 fe80::XXXX:2d9a/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
9: ens3f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether xx:xx:xx:xx:2d:9b brd ff:ff:ff:ff:ff:ff
    altname enp193s0f1np1
    altname enxb8e9241c2d9b
    inet6 fe80::XXXX:2d9b/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
10: enp129s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether xx:xx:xx:xx:c5:cc brd ff:ff:ff:ff:ff:ff
    altname enxe09d7311c5cc
    inet6 fe80::XXX:XXX:fe11:c5cc/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
11: enp129s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether xx:xx:xx:xx:c5:cd brd ff:ff:ff:ff:ff:ff
    altname enxe09d7311c5cd
    inet 10.3.1.240/32 brd 10.3.1.240 scope global enp129s0f1np1
       valid_lft forever preferred_lft forever
    inet6 fe80::XXX:XXX:fe11:c5cd/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
12: enp1303449.3@enp129s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether xx:xx:xx:xx:c5:cd brd ff:ff:ff:ff:ff:ff
    inet 10.3.1.240/16 brd 10.3.255.255 scope global enp1303449.3
       valid_lft forever preferred_lft forever
    inet6 fe80::XXX:XXX:fe11:c5cd/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

ens3f0np0: the public interface with DHCP
ens3f1np1: not used
enp129s0f0np0: not used
enp129s0f1np1: the interface linked to my OVH private network (Vrack)
enp1303449.3: The Vlan interface I've configured based on enp129s0f1np1

Any ideas ? Any suggestions ? I take everything ;)

Note: with Ubuntu on same type of dedicated server it works. I just had to follow this doc

[morind]

HI @dedene
You've already successfully dealt with OVH and Talos according to your post here
Any tips for me to configure my dedicated server with 2 interfaces (Vrack + Public) / Talos / Cilium ?

[BeArchiTek]

Hello @morind

Why did you let 10.3.1.240/32 inenp129s0f1np1 ?
As you are using VLAN (3), I would expect to have the IP only on the sub-interface.

[morind]

Thanks @BeArchiTek !
Yes, good catch. I've made so many tries and Im not sure to have tested with IP only defined on VLAN interface and not on the physical one.
I'm going to test that with some couple of new tests
If I found the solution, I will post it here..

[morind]

I let that problem for a while and I came back on it after a while. FYI with Talos 1.12 it works perfectly ! I was able to make it work with my OVH baremetal (dedicated server) with 2 interfaces (private + public)

[Biscottex]

Glad it work for you. On my side, I am getting stuck with Talos on a OVHcloud bare metal server.
@morind Maybe you could help.

Are you able to have a Service of type LoadBalancer with Cilium and the Gateway API? On my side it stay forever on ClusterIP.
I have a single node Kubernetes Cluster on a dedicated server. It was working 9 month ago but then I reinstalled from scratch with a new version of cilium/talos. I might have lost some config and it does not work anymore. I have been struggling the last 3 days on it.

I do not know where should I put the server IP info or if I should have a vrack in order to have a K8S Service LoadBalancer.

talosctl gen config $CLUSTER_NAME https://$CONTROL_PLANE_IP:6443 --install-disk /dev/$DISK_NAME --config-patch @patch.yaml --with-secrets secrets.yaml

patch.yaml

cluster:
  allowSchedulingOnControlPlanes: true
  network:
    cni:
      name: none
  proxy:
    disabled: true

machine:
  network:
    nameservers:
      - 213.186.33.99
      - 213.251.188.146
      - 213.251.188.141
      - 8.8.8.8
      - 1.1.1.1

Cilium CLI

cilium install --version 1.19.1 \
    --namespace kube-system \
    --set operator.replicas=1 \
    --set ipam.mode=kubernetes \
    --set kubeProxyReplacement=true \
    --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
    --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
    --set cgroup.autoMount.enabled=false \
    --set cgroup.hostRoot=/sys/fs/cgroup \
    --set autoDirectNodeRoutes=false \
    --set global.bpf.masquerade="true" \
    --set k8sClientRateLimit.qps=32 \
    --set k8sClientRateLimit.burst=64 \
    --set k8sServiceHost=127.0.0.1 \
    --set k8sServicePort=7445 \
    --set gatewayAPI.hostNetwork.enabled=true \
    --set gatewayAPI.enabled=true \
    --set gatewayAPI.enableAlpn=true \
    --set gatewayAPI.enableAppProtocol=true \
    --set bgpControlPlane.enabled=true \
    --set l2announcements.enabled=true

BGP or L2?

If I remember well I used BGP but it does not make sense. It should be L2.

BGP

#https://labs.ovhcloud.com/en/bgp-service/

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: cilium-bgp
spec:
  nodeSelector:
    matchLabels:
      rack: rack0
  bgpInstances:
    - name: "instance-65000"
      localASN: 65000
      peers:
        - name: "peer-65000-rack0"
          peerASN: 65000
          peerAddress: fd00:10:0:0::1
          peerConfigRef:
            name: "cilium-peer"

apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: bgp-advertisement
  labels:
    advertise: bgp
spec:
  advertisements:
    - advertisementType: "PodCIDR"
    - advertisementType: "Service"
      service:
        addresses:
          - LoadBalancerIP
      selector:
        matchExpressions:
          - { key: somekey, operator: NotIn, values: ["never-used-value"] } #Select all the LoadBalancer services to be available externally

apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: cilium-peer
spec:
  timers:
    holdTimeSeconds: 9
    keepAliveTimeSeconds: 3
  ebgpMultihop: 2
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 15
  families:
    - afi: ipv4
      safi: unicast
      advertisements:
        matchLabels:
          advertise: "bgp"
    - afi: ipv6
      safi: unicast
      advertisements:
        matchLabels:
          advertise: "bgp"

apiVersion: "cilium.io/v2"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "cilium-lb-ippool"
spec:
  blocks:
    - cidr: "xx.xx.xxx.xx/32"

L2

About the L2, what ip should I put in the CIDR, the IP or the dedicated server or a vrack?

 apiVersion: "cilium.io/v2"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "cilium-lb-ippool"
spec:
  blocks:
    - cidr: "xx.xx.xx.xx/32"

apiVersion: "cilium.io/v2alpha1"
kind: CiliumL2AnnouncementPolicy
metadata:
  name: l2-announcement-policy
spec:
  nodeSelector:
    matchLabels: {}
  # Apply to all services/gateways
  serviceSelector:
    matchLabels: {}
  interfaces:
    - ^eth[0-9]+
    - ^eno[0-9]+
    - ^ens[0-9]+
    - ^enp[0-9]+
  externalIPs: true
  loadBalancerIPs: true

K8S Gateway API

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: xx-xx-gateway
  namespace: xxxx-ns
  annotations:
    cert-manager.io/cluster-issuer: xx-xx-letsencrypt-issuer
spec:
  gatewayClassName: cilium
  #It looks like it is cilium by default. 
  addresses:
    - type: IPAddress
      value: xx.xx.xxx.xx
  listeners:

kubectl exec netshoot -n kube-system -- ip route
default via xx.68.232.254 dev eno1 src xx.68.232.96 metric 1024
10.244.0.0/24 via 10.244.0.170 dev cilium_host proto kernel src 10.244.0.170
10.244.0.170 dev cilium_host proto kernel scope link
xx.68.232.0/24 dev eno1 proto kernel scope link src xx.68.232.96 metric 1024