CORE-1331: fix CI flake from webhook cert secret mount delay#5336
Conversation
Enable RestartOnSecretRefresh on autoscaler and instrumentor cert rotators so pods remount TLS certs immediately instead of waiting on slow kubelet secret volume sync during readiness. Co-authored-by: Cursor <cursoragent@cursor.com>
RestartOnSecretRefresh intentionally restarts autoscaler/instrumentor once after writing webhook certs; relax restartCount to <= 1. Co-authored-by: Cursor <cursoragent@cursor.com>
…to-webhook-cert-secret
…to-webhook-cert-secret
RonFed
left a comment
RonFed
left a comment
There was a problem hiding this comment.
not sure we should re-start the autoscaler to handle this.
The slowness is only happening in initial install (which is what we test all the time) - users' common use-case is upgrade, which doesn't have this slowness issue.
Having always restart count of 1, might raise some questions, and cause users to think something is wrong
|
Need to unblock the CI, lets put some fix in this PR which help us unblock the CI, also I agree with Ron about the autoscaler restart |
Avoid RestartOnSecretRefresh so install no longer bumps restartCount. Materialize TLS certs from the Secret into a writable local CertDir so readiness does not wait on kubelet secret volume projection. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@RonFed @shivanshuraj1333 Agreed on avoiding autoscaler restart. Dropped Also removed the unused Secret volume mounts from autoscaler/instrumentor deployments. |
|
LGTM, waiting for Ron's approval |
3 participants
Summary
RestartOnSecretRefresh), sync TLS material from the Secret into a writable localCertDirviaSecretDiskSyncFixes CORE-1331
Related: cert-controller#35
Test plan
restartCount: 0/tmp/odigos-webhook-certsshortly after Secret is written/readyzpasses without waiting on kubelet secret projectioncentral-install/ kubernetes-tests for the previous autoscaler Ready flake