[Bug/错误]: Synthetic `[DONE]` bypasses pre-content retry for incomplete Responses streams

[diegonix]

Pre-submission Checklist / 提交前检查

• I have searched the existing issues and this bug has not been reported / 我已搜索现有 issues，此 Bug 尚未被报告
• I have read the documentation / 我已阅读文档
• I am using the latest version of AxonHub / 我正在使用 AxonHub 的最新版本

FAQ Check / FAQ 检查

• I have checked the FAQ and this bug is not covered there / 我已查阅 FAQ，此 Bug 未被覆盖

Bug Description / Bug 描述

When a Codex/OpenAI Responses upstream emits response.created and then reaches a clean transport EOF without any real Responses terminal event (response.completed, response.failed, response.incomplete, or response.cancelled), AxonHub appends a synthetic [DONE] before reporting ErrStreamIncomplete.

That synthetic terminal marker crosses the pre-content retry boundary first. With retry budget available and emptyResponseDetection=false, the pre-reader treats llm.DoneResponse as a valid terminal event and returns the stream as successful even though no meaningful content was produced. ErrStreamIncomplete appears only on the next iteration, after retry is no longer possible.

For a downstream /v1/chat/completions request, the same synthetic DoneResponse is also converted to client-visible [DONE] and can set the shared persistence state to completed before the late stream error is observed. In the affected request, AxonHub logged stream ended without terminal event, but both the request and its only execution were stored as completed with no error message.

This differs from an ordinary mid-content stream failure: the affected attempt delivered no text, reasoning, or tool call to OpenCode. Its persisted message had zero input/output/reasoning tokens and only a step-start part. Retrying or failing over was still safe at that point.

Environment

• AxonHub runtime: v1.0.0-beta5 (d061ac7df6aef0c5ec6cdfa9dc5002546a1c5a57)
• Also confirmed by source inspection in v1.0.0-beta6 (439e115ec74eda938c345a662cba700c94aefab4)
• Downstream API: POST /v1/chat/completions, streaming
• Channel: built-in ChatGPT/Codex Responses transport
• Client: OpenCode
• Models observed: gpt-5.4; the same incomplete-stream error has also occurred with gpt-5.6-sol
• Retry policy: enabled, maxChannelRetries=3, maxSingleChannelRetries=2
• emptyResponseDetection=false

Deterministic Stream Sequence

The minimal upstream sequence is:

response.created
clean source EOF (stream.Err() == nil)
no response.completed|failed|incomplete|cancelled

The current iterator order is:

1. Responses TransformStream wraps the raw stream with AppendStream(..., [DONE]).
2. response.created records responseID and produces only a role delta.
3. The raw stream reaches clean EOF.
4. AppendStream emits the synthetic [DONE].
5. responsesOutboundStream converts it to llm.DoneResponse without setting responseCompleted.
6. preReadLlmStream sees DoneResponse as terminal and returns success because emptyResponseDetection=false.
7. On the next iteration, the wrapper is exhausted and responsesOutboundStream sets ErrStreamIncomplete.
8. The retry loop has already returned the stream, and /v1/chat/completions may already have emitted [DONE] and marked persistence completed.

This sequence is deterministic at the stream-iterator level; it does not require a timing race.

Runtime Evidence

One affected attempt:

trace_id:   at-74579add-ff60-4ea9-93e4-b3f40f166ea5
request_id: ar-0aa6066a-ec15-42cc-8005-222fda2a6800
AxonHub request/execution IDs: 11086 / 11090
AxonHub log: Error in stream: stream ended without terminal event
internal AxonHub retries: 0
persisted request status: completed
persisted execution status: completed
persisted error_message: empty
OpenCode visible content: none
OpenCode input/output/reasoning tokens: 0 / 0 / 0

The client later made a separate request. It was not an AxonHub retry: each inbound request had exactly one execution and a distinct trace/request ID.

Source Evidence (v1.0.0-beta6)

• Responses adds a synthetic [DONE]: llm/transformer/openai/responses/outbound_stream.go#L24-L35
• AppendStream emits appended items after clean EOF: llm/streams/append.go#L21-L45
• [DONE] becomes llm.DoneResponse without validating responseCompleted: outbound_stream.go#L133-L141
• Missing-terminal validation runs only after the wrapped stream is exhausted: outbound_stream.go#L93-L114
• Pre-read returns on DoneResponse when empty-response detection is disabled: llm/pipeline/stream.go#L174-L246
• Chat Completions converts DoneResponse to [DONE]: llm/transformer/openai/inbound.go#L103-L124
• Persistence marks terminal events completed and checks that state before a later stream error: internal/server/orchestrator/inbound.go#L69-L115, internal/server/orchestrator/inbound.go#L135-L171

The beta5 implementation has the same ordering. PR #2090 in beta6 adds retry classification for real transport errors such as io.EOF, io.ErrUnexpectedEOF, and temporary/timeout net.Error, but this exact error is a synthetic sentinel created only when the underlying stream reports no error. It is therefore not covered by that classification change.

Steps to Reproduce / 复现步骤

1. Configure streaming through a Codex/Responses upstream with retry enabled and emptyResponseDetection=false.
2. Make the upstream emit response.created and then close cleanly without a Responses terminal event.
3. Call AxonHub through POST /v1/chat/completions.
4. Observe that no internal retry occurs, [DONE] precedes ErrStreamIncomplete, and persistence may record completed.

Expected Behavior / 期望行为

• A synthetic [DONE] must not be emitted until a real Responses terminal event has been validated.
• If the upstream ends after response.created but before meaningful content and without a real terminal event, the incomplete-stream error should reach the pipeline pre-reader and use the configured retry/failover budget.
• If recovery is unavailable, the client should receive a failure without a preceding success terminal marker.
• Persistence should record the request/execution as failed, not completed.

Error Message / 错误消息

Error in stream: stream ended without terminal event

Operating System / 操作系统

Linux (Ubuntu/Debian) / Linux（Ubuntu/Debian）

AxonHub Version / AxonHub 版本

v1.0.0-beta5

Usage Scenario / 使用场景

Other / 其他

API Format / API 格式

OpenAI - Chat Completions (/v1/chat/completions) / OpenAI 聊天补全

Channel Type / 渠道类型

OpenAI / OpenAI

Logs / 日志

Additional Context / 其他补充信息

Suggested Minimal Fix

Validate [DONE] inside responsesOutboundStream before enqueueing llm.DoneResponse. Conceptually:

if string(event.Data) == "[DONE]" {
    if s.state.responseID != "" && !s.responseCompleted {
        return ErrStreamIncomplete
    }

    s.enqueue(llm.DoneResponse)
    return nil
}

This makes the existing error visible during pre-read, before the stream crosses the retry/client/persistence boundary. A real terminal Responses event sets responseCompleted=true, so its trailing provider or synthetic [DONE] remains valid.

As defense in depth, persistence finalization could also prioritize a genuine non-cancellation stream error over StreamCompleted, but preventing an invalid synthetic terminal marker is the root fix.

Suggested Regression Tests

1. Responses outbound: response.created -> clean EOF returns ErrStreamIncomplete and never returns llm.DoneResponse.
2. Pipeline: first attempt response.created -> clean EOF, second attempt valid content and response.completed; assert two attempts and only second-attempt output reaches the client.
3. Chat Completions/persistence: assert no [DONE] precedes the error and the failed attempt is not stored as completed.
4. Control: response.created -> response.completed -> clean EOF still produces normal completion.

Related Issues and PRs

• [Bug/错误]: Codex /v1/responses stream errors become non-retryable after passing through AxonHub (OC direct->CPA can retry, via AxonHub fails with context canceled / TypeValidationError)` #1421 covers the broader family where AxonHub changes Responses stream failures into a non-retryable client-visible shape.
• [Bug/错误]: 错误 "stream ended without terminal event or completed response" 应该触发失败重试而不是直接失败 #1248 reports the same error text, but its closure discussion concerns streams that have already delivered chunks. This case has no meaningful content and should be recoverable by the newer pre-content retry path.
• [Bug/错误]: 上游中断导致的失败在请求中被记录为已完成 #1444 / fix: should not mark as completed if upstream error, close #1444 #1488 concerns failed streams recorded as completed, but this synthetic terminal ordering still reproduces that outcome in beta5 and remains present in beta6 source.
• opt: responses stream failed handle & request executions index, close #1473 #1585 introduced ErrStreamIncomplete and Responses failure conversion.
• fix: retry stream errors before first content #1944 introduced retry before first meaningful stream content.
• fix(orchestrator): retry transport-level errors via channel failover #2090 broadens transport-error retry in beta6 but does not classify the synthetic ErrStreamIncomplete sentinel.