Skip to content

fix: wait for an in flight heartbeat before deleting the message - #848

Open
marianorodriguez wants to merge 1 commit into
bbc:mainfrom
marianorodriguez:wait-for-in-flight-heartbeat
Open

fix: wait for an in flight heartbeat before deleting the message#848
marianorodriguez wants to merge 1 commit into
bbc:mainfrom
marianorodriguez:wait-for-in-flight-heartbeat

Conversation

@marianorodriguez

Copy link
Copy Markdown

Description:

When heartbeatInterval is set, the consumer occasionally emits an error for a message that was
processed and deleted perfectly fine:

Error changing visibility timeout: Value <receipt-handle> for parameter ReceiptHandle is invalid.
Reason: Message does not exist or is not available for visibility timeout change.

This makes the heartbeat wait for its in-flight request before the message is deleted, so the
spurious error no longer happens.

Type of change:

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Why is this change required?:

startHeartbeat runs on a setInterval and does not await its changeMessageVisibility call, so a
tick that fires shortly before the handler resolves can still be travelling to SQS while
deleteMessage runs. If it arrives after the delete, the receipt handle is already gone and SQS
rejects it, which surfaces as an error event even though the message was handled exactly once.

clearInterval in the finally block cannot prevent this, because it only cancels future ticks —
the request already in flight is beyond its reach.

It only shows up when a handler happens to finish within a few milliseconds before a tick, so it is
rare but persistent. The affected messages are handled correctly throughout: the error is emitted
right after a successful delete, and they are never redelivered — the only symptom is the misleading
error itself.

Code changes:

  • startHeartbeat now stores the promise of the visibility timeout change it has in flight, so the
    caller can wait for it. It takes a small mutable Heartbeat holder rather than returning it,
    because processMessage runs concurrently for each message in a batch and the interval callback
    needs somewhere per-message to record it.
  • Added stopHeartbeat, which clears the interval and awaits that in-flight tick. Awaiting it is
    what closes the window: once it has settled, no heartbeat request can still be on its way to SQS.
  • processMessage and processMessageBatch call stopHeartbeat right before deleting. The
    finally blocks are left untouched so the error paths keep clearing the interval exactly as
    before.
  • Waiting adds no latency in the normal case: heartbeat.inFlight is either unset or already
    settled, so the await resolves immediately. It only ever waits in the rare case that is
    currently producing the error.

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

The new test models SQS's actual behaviour — the visibility timeout change only fails once the
message has been deleted — so it fails without this change and passes with it. The existing 121 unit
tests pass unmodified. npm run typecheck, npm run lint and npm run format:check are all clean.
I was not able to run the integration suite locally, so that one is down to CI.

The visibility timeout heartbeat runs on a setInterval and its
changeMessageVisibility call is not awaited, so a tick that fires just
before the handler resolves can still be on its way to SQS while the
message is being deleted. When it lands after the delete, SQS rejects it
and the consumer emits an error for a message that was processed and
deleted correctly:

  Error changing visibility timeout: Value <handle> for parameter
  ReceiptHandle is invalid. Reason: Message does not exist or is not
  available for visibility timeout change.

clearInterval cannot help here because it only cancels future ticks.
Keep a reference to the tick that is in flight instead, and clear the
interval and await it before deleting.
@nicholasgriffintn

nicholasgriffintn commented Jul 31, 2026

Copy link
Copy Markdown
Member

Thanks for raising this PR, I appreciate it a lot.

I will take a look into this when I have time to, I presume there is a reason we didn't await before, I just want to make sure we're not going to step on any landmines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants