http: fix potential FD leak when zombie streams prevent connection close #42859
+219
−14
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I am chasing a case where a client sends POST messages on a HTTP1.1 connection, but is being systematically denied. The connection is immediately reused by the client. In some case we seem to trigger a racy case where the number of open file descriptor explodes. When a stream becomes a zombie (waiting for codec encode completion) and the connection is in DrainState::Closing, the connection is not closed because checkForDeferredClose() is not called after the zombie stream was finally destroyed. This commonly occurs with HTTP/1.1 connections when: - ext_authz (or similar filter) denies a request early, sending a 403 response before the full request body is received - The connection is (potentially) marked DrainState::Closing - The stream becomes a zombie waiting for the codec to finish encoding - When onCodecEncodeComplete() fires, doDeferredStreamDestroy() was called but checkForDeferredClose() was not, leaving the connection open In scenarios where clients try to reuse the same HTTP/1.1 connection and requests are systematically denied (e.g., by ext_authz), this causes rapid FD exhaustion as each denied request leaves an orphaned connection. The fix adds checkForDeferredClose() calls after zombie stream destruction in both onCodecEncodeComplete() and onCodecLowLevelReset(). Signed-off-by: William Dauchy <[email protected]>
Member
|
@KBaichoo, Kevin, do you want also to take a look for this? |
botengyao
reviewed
Jan 7, 2026
Member
botengyao
left a comment
botengyao
left a comment
There was a problem hiding this comment.
Thanks for fixing this, and left a high level question.
/wait
wdauchy
force-pushed
the
fd_leak
branch
6 times, most recently
from
495f026 to
c96a9be
Compare
Signed-off-by: William Dauchy <[email protected]>
Contributor
Author
|
/retest transients |
KBaichoo
previously approved these changes
Jan 9, 2026
Contributor
KBaichoo
left a comment
KBaichoo
left a comment
There was a problem hiding this comment.
Nice work, looks good to me
/wait
Signed-off-by: William Dauchy <[email protected]>
Contributor
Author
|
/retest transients |
botengyao
reviewed
Jan 9, 2026
Member
botengyao
left a comment
botengyao
left a comment
There was a problem hiding this comment.
Thanks for the fix!! LGTM, module 2 nits
/wait
| // This tests the fix for a potential connection leak where zombie streams | ||
| // were not triggering connection close when drain_state_ was Closing. | ||
| TEST_F(HttpConnectionManagerImplTest, ZombieStreamClosesConnectionOnCodecComplete) { | ||
| // Use HTTP/1.1 and set max_requests_per_connection to 1 to trigger |
Member
There was a problem hiding this comment.
For the test, can we explicitly set the runtime guard for true and false.
| // closed. doEndStream() call that created the zombie may have set | ||
| // drain_state_ to Closing, but checkForDeferredClose() couldn't close the | ||
| // connection at that time because streams_ wasn't empty yet. | ||
| if (Runtime::runtimeFeatureEnabled( |
Member
There was a problem hiding this comment.
Could we latch the runtime value as well to improve efficiency?
Signed-off-by: William Dauchy <[email protected]>
botengyao
approved these changes
Jan 10, 2026
Member
botengyao
left a comment
botengyao
left a comment
There was a problem hiding this comment.
lgtm, thanks for the fix!
Will merge it after the release early next week.
Contributor
Author
|
/retest transients |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Commit Message:
I am chasing a case where a client sends POST messages on a HTTP1.1 connection, but is being systematically denied. The connection is immediately reused by the client. In some case we seem to trigger a racy case where the number of open file descriptor explodes.
When a stream becomes a zombie (waiting for codec encode completion) and the connection is in DrainState::Closing, the connection is not closed because checkForDeferredClose() is not called after the zombie stream was finally destroyed.
This commonly occurs with HTTP/1.1 connections when:
In scenarios where clients try to reuse the same HTTP/1.1 connection and requests are systematically denied (e.g., by ext_authz), this causes rapid FD exhaustion as each denied request leaves an orphaned connection.
The fix adds checkForDeferredClose() calls after zombie stream destruction in both onCodecEncodeComplete() and onCodecLowLevelReset().
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]