Kubernetes Apps: handle pod security requirements

[gainsley]

For managed k8s platforms, or the k8ssite platform, where our platform does not create the cluster ourselves, we may need to deploy to clusters that have security policies in place. These security policies may then require that pod/container deployments have specific security contexts. A pretty typical set of security contexts are:

pod:

  runAsNonRoot: true
  runAsGroup: 1000
  runAsUser: 1000
  fsGroup: 1000

container:

  allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL

We need some way to handle this situation. A couple of options:

1. Detect the requirements from the target cluster and automatically insert security contexts. This is difficult, as some security restrictions are applies by admission controllers (binary containers) where we do not have visibility into their behavior. There are also several different software packages at play: Kubewarden, Kyverno, etc. This also doesn't apply to helm charts, as the security contexts are typically an option in the values.yaml somewhere.
2. Allow the user to specify the pod/container security policies as yaml excerpts on the App or AppInst, which we then inject into the generate manifests. This is the typical approach for helm charts. This is likely the the best approach, although now the user needs to understand how to write Kubernetes security contexts. This also won't apply the helm charts, as it belongs in the values.yaml somewhere.
3. Have a default set of security contexts per Cluster. Not really recommended, as it doesn't apply to helm charts or custom manifests, and is a different approach from setting the contexts on a per-AppInst basis.

Likely (2) is the best approach, though it needs to be discussed.