fix: close resumed streams when the DONE pub/sub message is lost - #47
fix: close resumed streams when the DONE pub/sub message is lost#47mdnanocom wants to merge 2 commits into
Conversation
The DONE control message travels over Redis pub/sub, which is fire-and-forget. If it is lost (subscriber reconnect, producer death between the sentinel SET and the publish), a resumed stream stays open forever even though the durable sentinel already says DONE — every chunk delivered, consumer hanging. Add a watchdog to resumed streams that periodically re-checks the durable sentinel and closes the stream after two consecutive checks observe a finished (or expired) sentinel. The two-check rule gives in-flight tail messages a full interval to drain before giving up. Interval is configurable via doneWatchdogIntervalMs (default 10s). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@cramforce hi sorry to bother you, any chance this pr gets merged or should i close it? |
|
I'm slightly sus of this PR but open to it. Is polling really the only option? |
|
(And thanks for the ping!) |
|
@cramforce |
|
The fix makes sense, but the watchdog currently leaks if subscription setup fails: after resumeStream() rejects, the interval keeps polling Redis indefinitely. Async setInterval can also overlap when Redis is slow. Could you clean it up in the outer catch/stream cancellation path and use non-overlapping polling (e.g. recursive setTimeout), with regression tests? |
|
@cramforce |
|
Alright, thanks so much and so for this being a slow process. One last thing: I need you to sign the commits |
Problem
The DONE control message is delivered to resumed streams over Redis pub/sub, which is fire-and-forget. If that single message is lost — subscriber connection dropped and reconnected at the wrong moment, or the producer died between writing the DONE sentinel and publishing — a resumed stream stays open forever, even though:
DONE.We hit this in production miltiple times behind an AI SDK chat UI: the answer rendered fully, but the SSE response never closed, so the client stayed in
streamingstate indefinitely (the AI SDK only leavesstreamingwhen the connection closes). A page refresh recovered —resumeExistingStreamreads the sentinel and returnsnull— which shows the durable state was correct; only the live consumer never re-checks it.Related but distinct from #42 / #43, which cover the initial ack phase. This is about a consumer that attached successfully and then never learns the stream ended.
Fix
Add a watchdog to
resumeStream: periodically re-read the durable sentinel and close the stream once the producer is finished or the sentinel expired.doneWatchdogIntervalMscontext option (default 10 000 ms) — oneGETper interval per attached consumer.Test
New test in
generic.test.tsusing the in-memory pub/sub: a wrapper publisher drops the DONE control message (durableSETstill goes through), a resumed consumer reads the stream. Without the watchdog the test hangs forever; with it, the consumer receives all chunks and closes within two intervals.