fix: wait for an in flight heartbeat before deleting the message - #848
Open
marianorodriguez wants to merge 1 commit into
Open
fix: wait for an in flight heartbeat before deleting the message#848marianorodriguez wants to merge 1 commit into
marianorodriguez wants to merge 1 commit into
Conversation
The visibility timeout heartbeat runs on a setInterval and its changeMessageVisibility call is not awaited, so a tick that fires just before the handler resolves can still be on its way to SQS while the message is being deleted. When it lands after the delete, SQS rejects it and the consumer emits an error for a message that was processed and deleted correctly: Error changing visibility timeout: Value <handle> for parameter ReceiptHandle is invalid. Reason: Message does not exist or is not available for visibility timeout change. clearInterval cannot help here because it only cancels future ticks. Keep a reference to the tick that is in flight instead, and clear the interval and await it before deleting.
marianorodriguez
requested review from
a team and
nicholasgriffintn
as code owners
July 31, 2026 09:02
Member
|
Thanks for raising this PR, I appreciate it a lot. I will take a look into this when I have time to, I presume there is a reason we didn't await before, I just want to make sure we're not going to step on any landmines. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
When
heartbeatIntervalis set, the consumer occasionally emits anerrorfor a message that wasprocessed and deleted perfectly fine:
This makes the heartbeat wait for its in-flight request before the message is deleted, so the
spurious error no longer happens.
Type of change:
Why is this change required?:
startHeartbeatruns on asetIntervaland does not await itschangeMessageVisibilitycall, so atick that fires shortly before the handler resolves can still be travelling to SQS while
deleteMessageruns. If it arrives after the delete, the receipt handle is already gone and SQSrejects it, which surfaces as an
errorevent even though the message was handled exactly once.clearIntervalin thefinallyblock cannot prevent this, because it only cancels future ticks —the request already in flight is beyond its reach.
It only shows up when a handler happens to finish within a few milliseconds before a tick, so it is
rare but persistent. The affected messages are handled correctly throughout: the error is emitted
right after a successful delete, and they are never redelivered — the only symptom is the misleading
error itself.
Code changes:
startHeartbeatnow stores the promise of the visibility timeout change it has in flight, so thecaller can wait for it. It takes a small mutable
Heartbeatholder rather than returning it,because
processMessageruns concurrently for each message in a batch and the interval callbackneeds somewhere per-message to record it.
stopHeartbeat, which clears the interval and awaits that in-flight tick. Awaiting it iswhat closes the window: once it has settled, no heartbeat request can still be on its way to SQS.
processMessageandprocessMessageBatchcallstopHeartbeatright before deleting. Thefinallyblocks are left untouched so the error paths keep clearing the interval exactly asbefore.
heartbeat.inFlightis either unset or alreadysettled, so the
awaitresolves immediately. It only ever waits in the rare case that iscurrently producing the error.
Checklist:
The new test models SQS's actual behaviour — the visibility timeout change only fails once the
message has been deleted — so it fails without this change and passes with it. The existing 121 unit
tests pass unmodified.
npm run typecheck,npm run lintandnpm run format:checkare all clean.I was not able to run the integration suite locally, so that one is down to CI.