OCPBUGS-86789: Make LateConnections test outcome-based

[mkowalski]

What

Makes the [sig-api-machinery][Feature:APIServer][Late] API LBs follow /readyz of kube-apiserver and stop sending requests test outcome-based instead of failing on the mere presence of a LateConnections event.

When a LateConnections event is found, the test now inspects the kube-apiserver audit log of the affected node for the surrounding time window and:

• flakes (not fails) when all late requests were gracefully rejected with 429 + Retry-After — keeping a fingerprint of slow LB convergence without failing the run,
• fails when late requests were answered with 5xx (graceful rejection did not protect the client),
• fails when no graceful rejection is recorded at all — the late connections were then probably reset without a response, the worst client outcome,
• fails when audit logs cannot be inspected or the event cannot be attributed to a node (conservative fallback, preserving current behaviour).

Why

The LateConnections/NonReadyRequests events were silently swallowed for ~5 years due to a type assertion bug in the openshift/kubernetes carry patch (unexpectedRequestsEventf.Store(s.Eventf) stored a method value whose dynamic type never matched the eventfFunc assertion). The Kube 1.36 rebase fixed the conversion, the events started being emitted, and this test began failing across multiple platforms — see Component Readiness regressions 44896 (metal-ipi ovn ipv4/ipv6/dualstack) and 44949 (azure ovn serial), tracked in OCPBUGS-86789.

Analysis of all 12 failing job runs shows the event only proves that a load balancer routed a new connection to a terminating kube-apiserver late in the shutdown window. No LB implementation can be perfectly synchronized with /readyz:

• on metal, the in-cluster kubernetes.default/OVN path goes stale when ovn-kubernetes' EndpointSlice informers lose their API watches through the api-int VIP during the control-plane rollout itself (~50 watch failures observed in the minute preceding the event in periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-ovn-dualstack/2079715174809341952),
• on cloud, NLB target health-check convergence has the same inherent latency.

Meanwhile shutdown-send-retry-after=true is the OpenShift default: late requests get 429 + Retry-After + Connection: close and well-behaved clients transparently retry against a healthy backend. Routing late is a mechanism detail; harming a client is the outcome that matters. The test now distinguishes the two.

Notes

• Audit logs are only fetched when a LateConnections event exists, so healthy runs pay zero extra cost; this is a [Late] test so runtime pressure is low.
• 2xx responses inside the window are intentionally not flagged: they belong to in-flight requests started before the drain, which graceful termination is supposed to finish.
• Related server-side hardening: OCPBUGS-86789: Enable send-retry-after-while-not-ready-once on all topologies cluster-kube-apiserver-operator#2236 (enable send-retry-after-while-not-ready-once on all topologies).

---

🤖 This PR was generated by AI on behalf of @mkowalski, who has reviewed it.

Summary by CodeRabbit

• Tests
  • Improved graceful-termination testing for late connections during API server shutdown.
  • Classifies late requests by observed outcomes, distinguishing server errors and missing responses from graceful rejections.
  • Identifies expected transient failures caused by load-balancer convergence separately from genuine termination issues.
  • Adds validation using audit logs and node attribution for more accurate failure reporting.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

[openshift-ci-robot]

@mkowalski: This pull request references Jira Issue OCPBUGS-86789, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What

Makes the [sig-api-machinery][Feature:APIServer][Late] API LBs follow /readyz of kube-apiserver and stop sending requests test outcome-based instead of failing on the mere presence of a LateConnections event.

When a LateConnections event is found, the test now inspects the kube-apiserver audit log of the affected node for the surrounding time window and:

• flakes (not fails) when all late requests were gracefully rejected with 429 + Retry-After — keeping a fingerprint of slow LB convergence without failing the run,
• fails when late requests were answered with 5xx (graceful rejection did not protect the client),
• fails when no graceful rejection is recorded at all — the late connections were then probably reset without a response, the worst client outcome,
• fails when audit logs cannot be inspected or the event cannot be attributed to a node (conservative fallback, preserving current behaviour).

Why

The LateConnections/NonReadyRequests events were silently swallowed for ~5 years due to a type assertion bug in the openshift/kubernetes carry patch (unexpectedRequestsEventf.Store(s.Eventf) stored a method value whose dynamic type never matched the eventfFunc assertion). The Kube 1.36 rebase fixed the conversion, the events started being emitted, and this test began failing across multiple platforms — see Component Readiness regressions 44896 (metal-ipi ovn ipv4/ipv6/dualstack) and 44949 (azure ovn serial), tracked in OCPBUGS-86789.

Analysis of all 12 failing job runs shows the event only proves that a load balancer routed a new connection to a terminating kube-apiserver late in the shutdown window. No LB implementation can be perfectly synchronized with /readyz:

• on metal, the in-cluster kubernetes.default/OVN path goes stale when ovn-kubernetes' EndpointSlice informers lose their API watches through the api-int VIP during the control-plane rollout itself (~50 watch failures observed in the minute preceding the event in periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-ovn-dualstack/2079715174809341952),
• on cloud, NLB target health-check convergence has the same inherent latency.

Meanwhile shutdown-send-retry-after=true is the OpenShift default: late requests get 429 + Retry-After + Connection: close and well-behaved clients transparently retry against a healthy backend. Routing late is a mechanism detail; harming a client is the outcome that matters. The test now distinguishes the two.

Notes

• Audit logs are only fetched when a LateConnections event exists, so healthy runs pay zero extra cost; this is a [Late] test so runtime pressure is low.
• 2xx responses inside the window are intentionally not flagged: they belong to in-flight requests started before the drain, which graceful termination is supposed to finish.
• Related server-side hardening: OCPBUGS-86789: Enable send-retry-after-while-not-ready-once on all topologies cluster-kube-apiserver-operator#2236 (enable send-retry-after-while-not-ready-once on all topologies).

---

🤖 This PR was generated by AI on behalf of @mkowalski, who has reviewed it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mkowalski

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [mkowalski]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Walkthrough

The graceful termination test now correlates LateConnections events with node-specific kube-apiserver audit logs and classifies outcomes as graceful 429 flakes, hard failures, or missing/unattributed responses.

Changes

Graceful termination testing

Layer / File(s)	Summary
Late connection windows test/extended/apiserver/graceful_termination.go	The test documents outcome semantics, imports HTTP and audit types, and records node-specific time windows for late events.
Audit outcome classification test/extended/apiserver/graceful_termination.go	Late events are attributed to nodes, audit entries are filtered by timestamp, and 429, 5xx, missing, and unattributed outcomes produce flakes or errors.
Estimated code review effort: 3 (Moderate)	~30 minutes

Sequence Diagram(s)

sequenceDiagram
  participant LateConnectionsTest
  participant nodeNameFromKASEvent
  participant auditEntriesForNodeInWindow
  participant KubeAPIServerAuditLogs
  LateConnectionsTest->>nodeNameFromKASEvent: Attribute late event to a node
  nodeNameFromKASEvent-->>LateConnectionsTest: Return node name
  LateConnectionsTest->>auditEntriesForNodeInWindow: Query audit entries for the event window
  auditEntriesForNodeInWindow->>KubeAPIServerAuditLogs: List and parse audit logs
  KubeAPIServerAuditLogs-->>auditEntriesForNodeInWindow: Return filtered audit events
  auditEntriesForNodeInWindow-->>LateConnectionsTest: Return response statuses
  LateConnectionsTest->>LateConnectionsTest: Classify 429, 5xx, or missing responses

Loading

---

Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

• Ignore

❌ Failed checks (1 error, 2 warnings)

Check name	Status	Explanation	Resolution
No-Sensitive-Data-In-Logs	❌ Error	New t.Errorf/result messages print full Event objects and raw audit fields (URI, UA, source IPs), which can expose internal hostnames/IPs and request data.	Redact or omit raw Event/audit fields; log only minimal context (node, reason, timestamp, status code) and avoid printing full objects, URIs, UAs, or IPs.
Test Structure And Quality	⚠️ Warning	The new test still has several bare Expect(...).NotTo(HaveOccurred()) setup checks, so failures will lack context.	Add contextual messages to the setup assertions (e.g. listing events/logs) and keep the outcome checks equally specific.
Microshift Test Compatibility	⚠️ Warning	New/changed e2e coverage uses openshift-kube-apiserver and master-node kube-apiserver logs with no MicroShift skip or apigroup guard.	Add a MicroShift exclusion ([Skipped:MicroShift] or runtime skip) or tag/guard the test so MicroShift CI skips it.

✅ Passed checks (12 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly reflects the main change: making the LateConnections test evaluate outcomes instead of failing on any event.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The only changed test file uses static Ginkgo titles; no It/Describe/Context/When strings include dynamic values or unstable identifiers.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The new LateConnections test has no multi-node-only logic and doesn’t require distinct nodes, node scaling, or HA failover.
Topology-Aware Scheduling Compatibility	✅ Passed	The only change is test logic; it adds no pod specs, replicas, affinities, selectors, PDBs, or topology-aware scheduling constraints.
Ote Binary Stdout Contract	✅ Passed	No process-level stdout writes were added; the new code only uses g.It/g.By and helpers, with no init/TestMain/BeforeSuite/Print calls.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	New test uses only cluster API/node logs; no hardcoded IPv4s, ParseIP/JoinHostPort, or public internet hosts were found.
No-Weak-Crypto	✅ Passed	Only test code changed; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons were added.
Container-Privileges	✅ Passed	Only a Go test file changed; no container manifests or privilege-related settings (privileged, hostPID/Network/IPC, SYS_ADMIN, allowPrivilegeEscalation, runAsUser 0) were introduced.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

@mkowalski: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.