ArgoCD fails on hook-finalizer (after upgrade to 3.x)

[ep4sh]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I have a Job (db migrations) with hook-weight == "-5" and another resource, let's say ConfigMap with hook-weight == "-10".
Job is long-running task, after the job finishes it seems that argocd can't properly finalize the sync process: it successfully removes CM but cant remove Job and eventually fails by timeout (context deadline).

This behavior is actual since I've upgraded to 3.x from 2.x

If I manually remove the finalizer from the Job, the synchronization process completes OK.

To Reproduce

Create Job with hook-weight == "-5", CM hook-weight == "-10".
Sync.

Expected behavior

ArgoCD should properly handle finalizer / helm-hooks

Screenshots

Version

v3.0.3+a14b012

[lonnix]

I have the same issue on 3.0.5. I have it on a ConfigMap that has helm.sh/hook-weight: "-100" but also on a Job that doesn't have hook-weight set but does have argocd.argoproj.io/hook-delete-policy: BeforeHookCreation

In both cases the finalizer is argocd.argoproj.io/hook-finalizer

[yunasc]

Any updates on this? I heavily rely on spinning up my environment with PreSync hooks to populate DB with data.

[kingbj940429]

I'll take a look at this

[isarns]

I'm encountering the same issue. I held off on upgrading to v3 for most clusters—only one of my clusters is currently running v3, and it's the only one affected. All clusters share the same configuration, so this seems specific to v3.

[kingbj940429]

I can't reproduce this issue

here yamls I used

apiVersion: batch/v1
kind: Job
metadata:
  name: test
  namespace: default
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-weight: "-5"
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: main
          image: busybox
          command: ["sh", "-c", "echo 'hello' && sleep 10"]
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: test
  namespace: default
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-weight: "-10"
data:
  MESSAGE: "hello-from-configmap"

finalizer was removed well after job is completed
(ignore createTimestamp)

[isarns]

Hi @kingbj940429, thanks for taking a look at this.

We're using the following Argo CD annotations:

argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/hook-delete-policy: HookSucceeded
argocd.argoproj.io/sync-options: Force=true,Replace=true
argocd.argoproj.io/sync-wave: '-200'

I’ve been able to consistently reproduce the issue. It appears to occur specifically when a sync is attempted after a job failure.

To reproduce:

1. Trigger a sync and let the job start.
2. Terminate the sync while the job is still running.
3. Attempt to sync again.

In v2.14, this would result in everything being deleted and re-synced. However, in the current behavior, the sync simply hangs and does not progress.

Let me know if I can provide more details!

[kingbj940429]

Hi @isarns thanks for your information

after pod by job is completed, job wasn't removed because of job's finalizer argocd.argoproj.io/hook-finalizer

so syncing status is continued

this Issue is right?

If right, this issue occurs when use replace/force together as sync option like below

apiVersion: batch/v1
kind: Job
metadata:
  name: test
  namespace: default
  annotations:
    helm.sh/hook: pre-upgrade
    argocd.argoproj.io/sync-options: Force=true,Replace=true #-- or check replace, force at Sync UI
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: main
          image: busybox
          command: ["sh", "-c", "echo 'hello' && sleep 3"]

[yunasc]

Yes, that's correct. We use Force and Replace for the hook.

[isarns]

@kingbj940429
yes this is the issue we encounter

[kingbj940429]

I took a close look at the code, and this looks more like a bug in how Helm Template works rather than anything related to argocd.argoproj.io/sync-options: Force=true,Replace=true

the same issue happens even when those sync-options aren’t set

Looking at the code, it uses the helm template command to render the YAML, and of course, the rendered YAML doesn’t include any finalizers

argo-cd/util/helm/cmd.go

Line 433 in 414d9eb

out, command, err := c.run(args...)

created like this, there is no finalizer field
because I didn't specify finalizer in helm

apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: test:batch/Job:default/job-test
    helm.sh/hook: pre-upgrade
  labels:
    batch.kubernetes.io/controller-uid: 98cc82ed-86f8-4e9d-93b2-79de6cfcfd9c
    batch.kubernetes.io/job-name: job-test
    controller-uid: 98cc82ed-86f8-4e9d-93b2-79de6cfcfd9c
    job-name: job-test
  name: job-test
  namespace: default

But when the code below runs, since the YAML rendered by helm template command doesn’t have the finalizer, it doesn’t remove it—instead, it ends up adding it.

https://github.com/argoproj/gitops-engine/blob/093aef0dad58015619479509f0d2ac71cc9cefd7/pkg/sync/sync_context.go#L789-L788

so a job created by helm is not deleted