k8s operator taints, tolerations and affinities

[PaulRudin]

I've deployed the vineyard operator to a test k8s cluster. The cluster has various node pools with a variety of taints.

How can I create a cluster so that vineyard instances will tolerate certain taints. Or how can I specify affinities to encourage vineyard instances onto certain nodes?

TIA

[dashanji]

@PaulRudin Hi, Paul. Currently, configuring affinity and anti-affinity is not supportable refer to the vineyard operator doc, but we will implement it ASAP.

Also, you can dump the yaml files of vineyard cluster and add some affinity or anti-affinity manually to work around it. All templates about vineyard instances are under the two directory vineyardd and etcd. FYI.

If you encounter any problems, please feel free to report them here.

[sighingnow]

Hi @PaulRudin,

Thanks for your insightful question.

From k8s point of view Client 1 has used 2 Gig, vineyard has used 0 bytes for vineyard objects?

That's correct. We use memfd for shared memory so the memory usage will be accounted for pod where client 1 locates.

Now client 2 gets the numpy array from vineyard. Again, this is accessing the same shared memory so no new memory is allocated?

Actually, yes.

This somehow seems wrong since if we were to delete Client 1 we'd still expect client 2 to be able to get the array from Vineyard

In our previous experiments, the memory usage won't be accounted for both vineyardd pod and client 2 pod if client 1 pod exits. From k8s perspective, the memory usage won't be double counted but that is true that the k8s scheduler may over schedule pod to that machine (I'm not sure if k8s scheduling considers the actual memory usage of the node or just compute the sum of pod's requests).

The root cause is that in K8s pod spec, the memory limits/requests don't apply to shared memory. A recent version of k8s introduced a feature gate SizeMemoryBackedVolumes which can be used to limit the size of shared memory but still failed to specify the "requests" of shared memory.

We currently choose to account the shared memory usage to the "writer" process, i.e., the process that created the objects. Does such a design work for your scenario?

[PaulRudin]

We currently choose to account the shared memory usage to the "writer" process, i.e., the process that created the objects. Does such a design work for your scenario?

Thanks for your explanation. I think this can work fine for my use case. I have one writer that creates all the arrays (client 1 in the above scenario) and potentially many replicas of the Client 2 pod (but these only consume - never write to shared memory). As long as I can schedule several replicas of the client 2 pod on the same node then it should work fine. I've got a proof of concept deployed now and it seems to work OK. Of course there's a way to go from a test to something that could be used in production, but I'm cautiously optimistic that it could work.

[sighingnow]

There's another approach that we can count all the memory usage for the vineyardd pod. With such a deployment, there would be no physical memory page allocation even during the put process and the memory usage won't be accounted for both the reader and writer. With such setting, Kubernetes scheduler won't over-schedule anymore.

Such a deployment option is not exposed to Kubernetes helm charts yet, but exposing the command line flags of vineyardd (-reserve_memory) should be easy.

It would be a little startup-time penalty for vineyardd container (about 1 minutes for reserving 200GB memory). Without the flag, the penalty for reserving physical memory pages is amortized during putting objects to vineyard, thus such a choose should be OK in production environment.

[PaulRudin]

Yeah, I'm wondering about just making a daemonset for the vineyard instances rather than using the operator, at least for now - because in any case I want to be able to get the vineyard instances on particular nodes; and whilst global objects are interesting and I can see some potential uses for that, I can achieve my immediate objective keeping objects local to each node.

[sighingnow]

Yeah, I'm wondering about just making a daemonset for the vineyard instances rather than using the operator, at least for now

That could work as well. See https://github.com/v6d-io/v6d/blob/main/k8s/pkg/templates/vineyardd/deployment.yaml for the pod spec template.

and whilst global objects are interesting and I can see some potential uses for that, I can achieve my immediate objective keeping objects local to each node.

Migrate client.get(fetch=True) works in these releases and further optimization on the migration implementation will be applied in the next several weeks.

[sighingnow]

Kubernetes operator is in a heavy development process currently. @PaulRudin do you have the plan to introduce Vineyard to your workflows on Kubernetes?

We are willing to know feedback and feature requests from end-users.

[PaulRudin]

I'm experimenting at the moment. I have a requirement to share memory between containers on the same k8s node and vineyard looks like it might be a good way to proceed.

I've got a proof of concept working on a single machine using docker compose so now I want to see if I can get the same thing deployed to a k8s cluster.