Skip to content

Add a way to control node selection for PVC backup jobs when an RWX PVC is mounted by multiple pods #1182

Description

@alexloz93

Summary

As a cluster operator using K8up with node-local backup storage
I want to control how K8up selects the target node for PVC backup jobs, especially when a single RWX PVC is mounted by multiple pods on different nodes
So that I can run the backup job on the node that has access to my local backup repository and avoid conflicting node selectors that leave the job Pending

Context

We are using K8up to back up PVCs to a repository stored on local node storage (hostPath / node-local path), so backup jobs must run on a specific node.

In our setup:

  • the backup job is constrained through podConfigRef / nodeSelector to run on a dedicated backup node
  • a single PVC is ReadWriteMany and volumeMode: Filesystem
  • that same RWX PVC is mounted by more than one pod
  • one of those pods runs on the node where we want the backup job to run
  • another pod mounting the same PVC may run on a different node

What we observe is that K8up automatically adds:

kubernetes.io/hostname=<some node>

to the backup job, apparently based on one of the pods mounting that PVC.

This causes a conflict with our own node selection, for example:

  • our config adds backup=true
  • K8up adds kubernetes.io/hostname=workerX

If both selectors do not point to the same node, the backup job remains Pending.

From the documentation, RWX PVCs are described as being grouped together, while node-based grouping and same-node scheduling are documented for RWO PVCs. Also, labelSelectors only allow filtering PVCs and PreBackupPods, not the pods consuming a PVC.

Problem

At the moment there seems to be no supported way to tell K8up:

  • which pod should be considered when deriving the node for a PVC backup job
  • to ignore some consumer pods based on labels
  • to disable automatic hostname pinning for RWX PVC backup jobs
  • to prefer the node defined by podConfigRef / custom nodeSelector

This makes node-local backup repositories hard or impossible to use in scenarios where an RWX PVC is mounted by multiple pods on different nodes.

Expected behavior

It should be possible to configure one of the following behaviors:

  1. Disable automatic hostname pinning for PVC backup jobs
  2. Prefer the nodeSelector from podConfigRef over automatically derived hostname selection
  3. Filter consumer pods by label/selector before K8up derives the node for the backup job
  4. Explicitly select which pod or node should be used for node derivation in multi-pod RWX scenarios

Any of these would make this use case manageable.

Actual behavior

K8up injects kubernetes.io/hostname=<node> into the backup job, which can conflict with a custom node selector and leave the job in Pending.

Why this would help

This would make K8up usable in environments where:

  • the backup repository is node-local
  • an RWX PVC is mounted by multiple pods
  • operators need deterministic placement of backup jobs
  • not all pods mounting the same PVC should influence job placement
  • Additional notes

Additional notes

PodConfig is documented as allowing overriding most of the podSpec, and labelSelectors are documented as filtering PVCs and PreBackupPods, but neither seems to solve this node selection case for PVC backups.

Minimal example

Observed backup job node selectors:

nodeSelector:
  backup: "true"
  kubernetes.io/hostname: workerlv1...

Result:

  • Job stays Pending when both do not match the same node.`

Expected result:

  • It should be possible to prevent automatic hostname pinning, or
  • prefer the node selection from podConfigRef, or
  • filter consumer pods before deriving the backup Job node

Out of Scope

This request is not about:

  • changing backup repository types
  • application-aware backups as a replacement for PVC backups
  • mutating Jobs externally with Kyverno or admission webhooks
  • custom forks/patches of K8up

The request is specifically about adding native control in K8up for node selection of PVC backup Jobs in multi-pod RWX scenarios.

Further links

Acceptance Criteria

This request would be satisfied if at least one of the following is implemented:

  • A way to disable automatic kubernetes.io/hostname node pinning for PVC backup Jobs
  • A way to prefer podConfigRef / user-defined nodeSelector over automatically derived hostname pinning
  • A way to filter consumer pods by label before deriving the node for a PVC backup Job
  • A way to explicitly select which pod or node should be used for node derivation in multi-pod RWX scenarios

Additionally:

  • The behavior should be documented clearly for RWX PVCs mounted by multiple pods
  • The resulting behavior should be deterministic and not leave Jobs in Pending because of conflicting node selectors

Implementation Ideas

Possible implementation options:

  • add a flag/field to disable automatic hostname pinning
  • add a flag/field to prefer user-defined node selection from podConfigRef
  • add a selector to filter consumer pods used for node derivation
  • only apply hostname pinning for RWO PVCs, unless explicitly enabled for RWX

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions