Accounting of sidecar/OS resources in kubernetes clusters

[gainsley]

For auto-clusters, we currently create a Kubernetes cluster based on the resource requirements and min node size requirements specified by the App definition, i.e. App.KubernetesResources. We are currently ignoring resource needs for:
a) sidecar apps, like prometheus, nfs-provisioner, nvidia-operator
b) OS needs

We are also calculating currently used resources by ignoring the above. We need to decide on how many resources to allocate for each of the above and then add those to our resource calculations.