What is ControllerReferencesOnly=true? Argocd does not mention it.

[groundnuty]

The argocd docs for sync does not mention the argocd.argoproj.io/sync-options: ControllerReferencesOnly=true used in those charts.

What is that option and how is it supposed to work?

[sd109]

Thanks for raising this. I believe it's related to some WIP (or stale) Argo CD changes based on argoproj/argo-cd#12210 and argoproj/gitops-engine#503 which were attempts to make Argo CD play nicely with Cluster API but never gained much traction with the Argo maintainers. In the mean time we've found that Flux CD is much more suitable for use with these charts than Argo CD anyway so we should probably remove references to Argo to avoid confusion.

[groundnuty]

@sd109, the argocd annotations proved to be very helpful when using those charts with argocd. I guess removing things like PruneLast=true will be a breaking change, and we who use argocd will need to apply a patch on our side.

Are you moving away from ArgoCD towards Flux when using those charts? If so it would be of great benefit yo users to maybe showcase the it's usage with Flux somehow?

[sd109]

Ah sorry, I guess I misunderstood... My understanding was that there was some fundamental issue using ArgoCD with some of the Cluster API objects in these charts and that it wouldn't work at all with out the previously linked WIP patch.

So is it the case that the ControllerReferencesOnly=true is not a core ArgoCD feature since it only works with the patch I linked previously but the PruneLast=true is required for your ArgoCD use cases? If the latter are required for your use case then I'm sure we can reinstate them.

FWIW we haven't used ArgoCD at all with these charts in the past couple of years (hence why I was keen to avoid making it look like we explicitly support / test against Argo). We have an example of how we use these charts with Flux in our capi-helm-fluxcd-config repo and personally I find that the abstractions provided by Flux HelmChart and HelmRelease CRDs are much simpler to work and reason about with than Argo's Application abstractions.

N.B. It's worth noting that the capi-helm-fluxcd-config repo goes further in that it provides a Python script to bootstrap a workload cluster using an ephemeral management cluster and then transfer the management cluster responsibilities onto the workload cluster so that it's self-managed, but it should nonetheless give an idea of how to define CAPI clusters via Flux CRDs for simpler separate management cluster and workload cluster deployments.

[groundnuty]

Thank you for such an extensive answer!

1. We are managing fleets of clusters using these charts with argocd. Back when we first started, the argocd annotations were extremely helpful to get our heads around CAPI/CAPO resource dependencies!

2. thanks for the capi-helm-fluxcd-config resource. I could provide a similar example for argocd if that would help a community that uses chosen charts with argocd.

3. I've also patched a subset of cluster-addons to use CAAPH cluster-api-addon-provider-helm. I could provide a branch with such changes if there were plans to replace cluster-api-addon-provider. It all works perfectly under CAAPH.

4. I will make sure that PruneLast=true is still required (everything works with it, not sure without!) and get back to you.

[sd109]

That's useful to know, thanks! I guess we probably should add a short docs section mentioning what is and isn't supported / tested in terms of GitOps tooling and we can add links to any relevant Flux CD and Argo CD examples there.

I'll re-open the issue for now so that we don't forget about it and will await the result of your testing without the PruneLast sync option before deciding whether we need to reinstate that or not.

[dalees]

Thanks for opening this issue and your work @groundnuty - we currently deploy our CAPI Management Cluster using these charts and ArgoCD.

We don't re-deploy or migrate management of the management cluster to itself though, but we do rebuild the management cluster occasionally. We haven't seen many issues with ArgoCD using these charts (I think Matt said it was relating to resource cleanup?) but I'd be interested to know more about them so we can identify them when they arise!

Replacing cluster-api-addon-provider with CAAPH is on our roadmap, if you've already done it @groundnuty I'd be grateful for sharing the required changes somewhere, especially how the Manifest resource can be handled.

[sd109]

Thanks for the input @dalees - that's very useful to know that you're also using Argo with these charts. FWIW we have also talked about replacing the cluster-api-addon-provider with CAAPH (or possibly Flux CD) and have tentative plans to explore this at some point too, so if either you or @groundnuty are interested in contributing to or collaborating on that we'd be keen to discuss it in more detail!

[JohnGarbutt]

Sounds like we need to find a way to get test with Argo, so we don't break that.

We are using them with fluxCD, but that is "just helm", so its already covered in some way.

@sd109 @dalees we have been talking about getting a regular office-hours for azimuth-cloud related things, including these charts. Would something like 0900UTC on Tuesday work for both of you?

Should we revert the patch to avoid the breakage? We can revisit the tidy ups: https://github.com/azimuth-cloud/capi-helm-charts/pull/544/files

[groundnuty]

@sd109 @dalees @JohnGarbutt I'm happy to discuss it and collaborate/share work. I'm happy that I took the same direction that the project is heading :-) Did you have a meeting on Tuesday? I'm available all next week - happy to join a call and talk.

[groundnuty]

There is some activity around the said argocd option that might unblock smooth management of CAPI Clusters with ArgoCD
argoproj/gitops-engine#771

@JohnGarbutt, any news about those regular office hours now that the vacation period is over? ;-)

[JohnGarbutt]

@groundnuty good point, we should totally get that started ASAP.
I will try get something uploaded in the org README here:
https://github.com/azimuth-cloud
(Great new about the ArgoCD progress!)