may not add resource with an already registered id: ~G_v1_Namespace|~X|flux-system

[dev-samples]

I have the below base:

apps/mycluster/base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - release.yaml

And the below two overlays :

##########################################################################
#
# Overlay a
#
##########################################################################

apps/mycluster/team-a-jenkins/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: flux-system
resources:
  - namespace.yaml
  - patch.yaml
  - ../base

apps/mycluster/team-a-jenkins/namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: team-a-jenkins

apps/mycluster/team-a-jenkins/patch.yaml

---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: team-a-jenkins
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  prune: true
  validation: client
  path: "./apps/mycluster/team-a-jenkins/"
  postBuild:
    substitute:
      SEED_REPO: "git@team-a.git"

##########################################################################
#
# Overlay b
#
##########################################################################

apps/mycluster/team-b-jenkins/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: flux-system
resources:
  - namespace.yaml
  - patch
  - ../base

apps/mycluster/team-b-jenkins/namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: team-b-jenkins

apps/mycluster/team-b-jenkins/patch.yaml

---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: team-b-jenkins
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  prune: true
  validation: client
  path: "./apps/mycluster/team-b-jenkins/"
  postBuild:
    substitute:
      SEED_REPO: "git@team-b.git"

Both overlays needs to use the flux-system namespace because that namespace contains a secret they both use - see: #2500 (reply in thread)

But when I build the above with kustomize (and eventually flux) I get the below error:

  kustomize build /home/user/repos/samples/apps/mycluster/
Error: accumulating resources: accumulation err='accumulating resources from 'sandbox-02-devops': '/home/user/repos/samples/apps/mycluster/team-b-jenkins' must resolve to a file': recursed merging from path '/home/user/repos/samples/apps/mycluster/team-b-jenkins': may not add resource with an already registered id: ~G_v1_Namespace|~X|flux-system

How do I register multiple overlays in the flux-system namespace using the same base?

I know this might be a kustomize specific question but maybe flux offers some tricks to get it to work?

[kingdonb]

If the overlay includes the flux-system namespace, you can delete it like this:

https://github.com/kingdonb/github-actions-demo/blob/2c6a88da6e11ffe1e236218a866b53ad165aa476/apps/podinfo/ex1-naive-dev-automation/kustomization.yaml#L12-L18

If you have multiple kustomize overlays in the same Flux kustomization, and they are composed together using Kustomize, they will have to be sure that individually the Kustomize overlays don't create identical resources. In this case they can't all create the flux-system namespace.

[dev-samples]

Actually that exactly what I saw earlier, the flux-system namespace got deleted and I had to re-boostrap flux. I decided I would ignore this for now and look into it at a later point until I have the current issue solved.

But I need to have the flux-system in:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: flux-system    # <<<<<<
resources:
  - namespace.yaml
  - patch
  - ../base

because of the sops/private key issue see: #2500 (reply in thread)

[kingdonb]

Those are not kustomization Objects, but kustomization Overlays

https://fluxcd.io/docs/faq/#are-there-two-kustomization-types

Please take note of this, understand the difference between keeping the Flux Kustomization in the flux-system namespace, where it can access the secrets needed for decryption, and setting the namespace flag in a SIG-CLI Kustomize Overlay, where it has the effect of doing Namespace Transformation on the underlaying manifests. These are two totally different Kustomizations and the line that I highlighted does not have the effect that you are ascribing to it. This is equivalent to targetNamespace in a Flux Kustomization.

[dev-samples]

Yes I know they are two different things but still I don't see how I can:

1. Have a kustomize.toolkit.fluxcd.io/v1beta2 per app that only runs a spec.postBuild.substitute to substitute variables from the base.
2. Have a secret in each namespace that is decrypted using the same private key from the flux-system namespace.
3. Deploy each app into its separate namespace and patch values.

It seems that mixing variable substitution, values patching and secret decryption is not possible at the same time?

E.g. here:
https://github.com/dev-samples/sample-flux/pull/1/files

is an example where:

1. Value patching
2. Secret decoding

are working which is great.

But I still have not managed to add variable substitution without breaking one of the above.

One example of the error I get is:

$ flux get kustomizations -A 
NAMESPACE     	NAME          	READY	MESSAGE                                                  	REVISION                               	SUSPENDED 
flux-system   	apps          	True 	Applied revision: sample-01-variable-substitution/bca8929	sample-01-variable-substitution/bca8929	False    	
flux-system   	flux-system   	True 	Applied revision: sample-01-variable-substitution/bca8929	sample-01-variable-substitution/bca8929	False    	
flux-system   	infrastructure	True 	Applied revision: sample-01-variable-substitution/bca8929	sample-01-variable-substitution/bca8929	False    	
team-a-jenkins	flux-system   	False	Source 'GitRepository/flux-system' not found             	                                       	False    	
team-b-jenkins	flux-system   	False	Source 'GitRepository/flux-system' not found

using this configuration https://github.com/dev-samples/sample-flux/tree/sample-01-variable-substitution

But I need to add e.g.: namespace: team-a-jenkins to get the helm release to work. If I remove namespace: team-a-jenkins (from the vanilla kustomize object) the helm release then fails with:

flux-system apps False HelmRelease/jenkins/jenkins namespace not specified, error: namespaces "jenkins" not found sample-01-variable-substitution-debug/f00983103fc8e1971a667e025efa39f5738715f1 False

It seems that:

https://github.com/dev-samples/sample-flux/blob/sample-01-variable-substitution/apps/kind-dev/team-a-jenkins/kustomization.yaml#L12

patchesStrategicMerge:
  - jenkins-release.yaml #helm.toolkit.fluxcd.io/v2beta1
  - variables.yaml #kustomize.toolkit.fluxcd.io/v1beta2

cannot be mixed.

[kingdonb]

Thank you for providing a concrete example. I have to take some time to review it and make sure I understand the problem again.

But from the hip, I can try to add a few more things that might get you past it first:

• the Kustomization and HelmRelease objects are each a "nexus of control" for your Flux applier workflows. Whatever namespace that resource lives in, will be the namespace where secrets are drawn from and where any references that do not allow cross-namespaces will be targeting.
• You can set these variables in HelmRelease which make the deployment target a different location than the "nexus of control" location: spec.storageNamespace, spec.targetNamespace. They are usually set to the same namespace, but that is not strictly required (although I would recommend setting them to be the same always, unless you know exactly why they could be set different.)

I think what you want is for instances of the HelmRelease to patch storageNamespace and targetNamespace, or to use the variable substitution in those fields. I can see where that would go wrong though, now that I'm thinking about it, (the release itself should get a different name, and you cannot patch a resource's name, I think...)

I unfortunately don't think the advice I've provided so far is likely going to get you past this issue, although it might be adjacent to the thing that gets you unblocked. Let me have a think and some time to review your repo this week, and I'll see if I can suggest a change.

[dev-samples]

Hm maybe this vanilla kustomize "hack":

kubernetes-sigs/kustomize#388 (comment)

could work instead of using spec.postBuild.substitute from kustomize.toolkit.fluxcd.io/v1beta2.

On a side note. I am a bit concerned that such simple/classic build configuration management needs to be done in such a convoluted way - and potentially its not possible at all.

Yes I get that that kustomize is based on not doing envsubs/templating configuration but that view on reality does not match any projects I have/am working on.

[dev-samples]

Now works as expected/as described in the docs when having one kustomize.toolkit.fluxcd.io/v1beta2 per app registered in the flux-system namespace. Both sops, patching and separate namespace per app.

[joaqquin89]

Guys

I have the same problem .. but im trying to install flux using the community helm charts .. and my idea is to manage multiple cluster from 1 repo ... but when I tried to add my second cluster . the problems start ... because I have the nexts problems:

anyone knows what means this problem or what is the configuration for the helm chart?

thanks

[kingdonb]

may not add resource with an already registered id

This typically means that within the Flux Kustomization, there is more than one instance of the resource named (in this case a namespace flux-system.)

There must be only one instance of each resource, and any alternative versions should be created as patches instead. Please share the repository if you need help debugging, it won't be possible to say more than that without more concrete examples – if you share the repo, then someone may be able to point out specifically what the error is.

If you want to try to trace it down interactively, the best way is to install kustomize and run kustomize build in that path, or if you have many Flux features used, try flux build which can exercise most or all of the options that might be used in a Kustomization. It should get you the same error on the console, then you can try different versions of your config and see what works without waiting for the git push-reconcile interval each time you want to test a different version.

[joaqquin89]

King thanks for your response ..my idea is to install flux 2 only using helmcharts not bootstrap and the problem happened when i installed flux in the cluster A and B … but both are pointing to the same repo ( my idea is not to manage 1 repo per cluster and i want to manage 1 repo for both clusters

[kingdonb]

The Flux v2 Helm Charts from flux-community are not quite supported, as there are known issues related to Helm controller which signify that with a deadlock condition standing in the way as it is currently implemented, Helm controller cannot upgrade itself without failing. Bootstrap is the main officially supported method which we recommend for all users who can manage Flux that way.

If you want to use bootstrap, the configuration you're describing (one configuration repository from which two clusters are bootstrapped, and they both mirror the same configuration) is possible, it is called "Identical Environments" and is documented here:

https://github.com/fluxcd/flux2-kustomize-helm-example#identical-environments

[tspearconquest]

I ran into this error due to the Kustomize CLI's kustomize.yaml supporting a namespace: field which, apparently, will overwrite the name field of a namespace resource. I had tried to add a second namespace managed under the flux-system Kustomization, and one of the kustomization.yaml files that was nested in a subdirectory under flux-system had this field with the value set to flux-system. This forced my new namespace's name to be transformed to flux-system, causing Kustomize to spit out an error because there were 2 flux-system namespaces in the build. Removing this field solved the error and allowed this extra namespace to be created.

Check your nested kustomize.yaml files for any mentions of this field if you run into this error when trying to have the main flux-system kustomization create any extra namespaces. Note: flux team discourage managing namespaces in this way - but as this is a one-off namespace for a test, and I ran into this, it's possible others will too so I wanted to mention checking for this regardless.