Hot reload certificates after rotation

[jared-schmidt-niceincontact]

Is your feature request related to a problem? Please describe.
Right now, certificate rotation requires restarting the agent and principal pods. This requires installing, distributing, and maintaining additional software such as Stakater Reloader or some custom scripting. It would be much better and cleaner to simply support hot reloading of certificates.

Pardon the AI drop, but this seemed like a pretty good suggestion and considers a number of edge cases. Feel free to pick it apart, particularly if the error handling is not in line with your vision of how the overall system should normally work.

Problem

Certificate rotation currently requires a process restart. The documented procedure ends with:

kubectl rollout restart deployment argocd-agent-principal -n argocd

This is awkward for a few reasons:

• With cert-manager or a corporate PKI issuing short-lived certificates, rotation is frequent and automated, but the restart is not.
• Restarting the principal drops every agent's gRPC stream at once, causing a reconnect storm proportional to the number of connected agents.
• A restart is an all-or-nothing operation: if the newly issued certificate is malformed, the principal fails to start and enters CrashLoopBackOff rather than continuing to serve on the still-valid previous certificate.

Proposed mechanism

Go's crypto/tls supports per-handshake certificate selection, so no listener restart is needed. The suggestion is a small Provider type that holds validated TLS material behind an atomic pointer, plus a source-agnostic watcher that swaps it.

1. The holder

// Material is a validated, ready-to-serve set of TLS material.
type Material struct {
    Cert   *tls.Certificate
    CAPool *x509.CertPool
    hash   [32]byte // over the source PEM bytes; suppresses no-op reloads
}

type Provider struct {
    current atomic.Pointer[Material]
}

func (p *Provider) Load() *Material { return p.current.Load() }

Reads are lock-free, which matters because they sit on the TLS handshake path.

2. Server wiring — use GetConfigForClient, not just GetCertificate

This is the key detail. tls.Config.GetCertificate is the usual answer for hot-reloading, but it only supplies the leaf certificate. ClientCAs is read once when the config is built and is never consulted again, so a principal using only GetCertificate would hot-reload its server certificate while continuing to verify agent client certs against a stale CA bundle. Since the principal validates agents against argocd-agent-ca, rotating that CA needs to work too.

GetConfigForClient is invoked per handshake and returns a complete *tls.Config, so it covers both:

base := &tls.Config{
    MinVersion: tls.VersionTLS13,
    ClientAuth: tls.RequireAndVerifyClientCert,
}

srvTLS := &tls.Config{
    GetConfigForClient: func(*tls.ClientHelloInfo) (*tls.Config, error) {
        m := provider.Load()
        if m == nil {
            return nil, errors.New("no TLS material loaded")
        }
        c := base.Clone() // Go does not merge the returned config with the outer one
        c.Certificates = []tls.Certificate{*m.Cert}
        c.ClientCAs = m.CAPool
        return c, nil
    },
}

grpcServer := grpc.NewServer(grpc.Creds(credentials.NewTLS(srvTLS)))

The same wiring applies to the resource proxy's HTTPS server.

3. Validate before swapping

The swap should be fail-closed: bad material is rejected and the previous certificate keeps serving.

func Validate(certPEM, keyPEM, caPEM []byte) (*Material, error) {
    cert, err := tls.X509KeyPair(certPEM, keyPEM)
    if err != nil {
        return nil, fmt.Errorf("invalid keypair: %w", err)
    }
    leaf, err := x509.ParseCertificate(cert.Certificate[0])
    if err != nil {
        return nil, fmt.Errorf("cannot parse leaf: %w", err)
    }
    cert.Leaf = leaf // avoids re-parsing on every handshake
    if time.Now().After(leaf.NotAfter) {
        return nil, fmt.Errorf("certificate expired at %s", leaf.NotAfter)
    }
    pool := x509.NewCertPool()
    if !pool.AppendCertsFromPEM(caPEM) {
        return nil, errors.New("CA bundle contains no usable certificates")
    }
    return &Material{Cert: &cert, CAPool: pool}, nil
}

This is the main safety argument for the feature: a botched rotation becomes a logged error and a metric rather than a crash loop.

4. Sources

The principal reads certificates either from Secrets or from file paths (--tls-cert, --tls-key, --root-ca-path), so the watcher should be behind an interface with two implementations:

type Source interface {
    Load(ctx context.Context) (certPEM, keyPEM, caPEM []byte, err error)
    Watch(ctx context.Context, onChange func()) error
}

Secret source. A SharedInformer filtered to the specific secret names via fieldSelector=metadata.name=.... This is the natural fit — the principal already has a Kubernetes client and already resolves these secrets by name, and it needs no extra RBAC beyond a watch verb on the secrets it already reads.

File source. fsnotify on the containing directory, not the file. Kubernetes projects Secret volumes through an atomically swapped ..data symlink, so the mounted file never receives a WRITE event — the observable events are CREATE/REMOVE/RENAME on the directory. Re-read all material on any event and debounce for ~1s. A low-frequency fallback poll (compare the SHA-256 in Material.hash) is worth adding as well, since fsnotify silently misses events on some filesystems.

If controller-runtime is already in the dependency graph, sigs.k8s.io/controller-runtime/pkg/certwatcher implements the file-path half of this and could be used directly. It does not cover Secrets or the CA pool, so it would only replace part of the work.

5. Client side (agent)

tls.Config.GetClientCertificate is the per-handshake analogue for the agent's client certificate. Note the asymmetry: there is no per-handshake callback for RootCAs on the client side. Rotating the CA the agent trusts requires either rebuilding the tls.Config per dial via grpc.WithContextDialer, or InsecureSkipVerify combined with a manual VerifyPeerCertificate against the swappable pool. Worth deciding explicitly rather than by accident.

Also worth stating in the docs: because gRPC holds a long-lived HTTP/2 connection, a rotated agent certificate only takes effect on the next reconnect. That is fine as long as the old certificate has not expired, but an optional graceful reconnect on change may be desirable.

Observability

Cheap to add alongside, and useful independently:

• argocd_agent_tls_cert_not_after_seconds{component} — gauge, enables expiry alerting
• argocd_agent_tls_cert_reload_total{component,result} — counter, result in success|error
• argocd_agent_tls_cert_last_reload_timestamp_seconds{component} — gauge

Only log on an actual fingerprint change; informer resyncs will otherwise produce noise.

Suggested scoping

There are several independent consumers of TLS material (principal gRPC server certificate, root CA for client cert verification, resource proxy certificate and its CA, agent client certificate, JWT signing key). Rather than one large change, this could land as:

1. The Provider + Source abstraction with the Secret and file implementations, wired into the principal's gRPC server only.
2. Resource proxy.
3. Agent client certificate, including the RootCAs decision above.

The JWT signing key fits the same swappable-holder pattern but needs a grace period during which both old and new keys validate, since already-issued tokens remain in flight. Probably best treated as a separate issue.

Alternatives considered

• SIGHUP handler. Simple, but does not compose with cert-manager, which has no way to signal the pod.
• Sidecar that restarts the process on change. Still drops all agent connections and still crash-loops on a bad certificate.
• Rely on kubectl rollout restart. The status quo; acceptable for long-lived certificates, increasingly painful as issuance lifetimes shorten.

[jannfis]

Thanks for creating this issue. I agree that hot reloading of the TLS certificates is a valuable feature, and the solution proposed seems legit to me.