disclaimer: issue drafted by Claude Opus, with better understanding of Rust than I. As per #44, I'm trying to find the secret sauce for nested ESI and streaming output.
drain_queue: nested dca="esi" includes are appended out of document order
Summary
When a dca="esi" include contains further <esi:include> tags, the nested includes are emitted after all sibling content in the parent document rather than inline at the position of the original include. This breaks document order for any nesting depth > 1.
Minimal reproduction
Setup — three fragments served by a backend called "origin":
/page (the outer ESI document):
<html>
<body>
<header>
<esi:include src="/header" dca="esi" />
</header>
<main>Main content</main>
</body>
</html>/header (fragment, included with dca="esi"):
<h1>Site Title</h1>
<nav>
<esi:include src="/menu" dca="esi" />
</nav>
/menu (fragment, included with dca="esi"):
<ul><li>Home</li><li>About</li></ul>
Expected output
<html>
<body>
<header>
<h1>Site Title</h1>
<nav>
<ul><li>Home</li><li>About</li></ul>
</nav>
</header>
<main>Main content</main>
</body>
</html>The <ul> menu appears inside <nav>, inside <header>, before <main>.
Actual output
<html>
<body>
<header>
<h1>Site Title</h1>
<nav>
</nav>
</header>
<main>Main content</main>
</body>
</html><ul><li>Home</li><li>About</li></ul>The <ul> menu appears after </html>, completely outside the document structure. The <nav> is empty.
Root cause
The bug is in drain_queue in esi/src/lib.rs — specifically the interaction between Step 1 (slot assignment) and Step 5 (processing completed includes).
How slots work
drain_queue maintains a buf: Vec<Option<Bytes>> where each queued element gets a sequential slot index. Content is flushed in order from next_out forward. The sequence for the example above after initial parsing:
| Slot |
Content |
| 0 |
Content("<html>\n<body>\n <header>\n ") |
| 1 |
Include(/header) — pending |
| 2 |
Content("\n </header>\n <main>Main content</main>\n</body>\n</html>") |
What happens when /header completes (Step 5)
process_include is called with fragment for /header, which has dca="esi". This invokes process_fragment_body, which:
- Parses the
/header response body as ESI via parse_complete
- Creates a
DocumentHandler writing to a local slot_buf: Vec<u8>
- Iterates over the parsed elements
When it hits <h1>Site Title</h1> — DocumentHandler::write_bytes checks self.processor.queue.is_empty(). The queue is empty (Step 1 drained it), so it writes directly to slot_buf. ✅
When it hits <esi:include src="/menu" dca="esi" /> — DocumentHandler::on_include dispatches the request and pushes QueuedElement::Include onto self.processor.queue. ✅
When it hits the trailing text \n</nav>\n — DocumentHandler::write_bytes sees the queue is non-empty (the /menu include is queued), so it pushes QueuedElement::Content onto the queue instead of writing to slot_buf. This is correct behaviour for maintaining document order within the fragment.
After process_include returns, slot_buf contains only <h1>Site Title</h1>\n<nav>\n (everything before the nested include). This is assigned to buf[1].
The ordering problem
Back in drain_queue's main loop, we return to Step 1, which drains self.queue. The nested /menu include and the trailing </nav> content get assigned new slots at the end of buf:
| Slot |
Content |
| 0 |
Content("<html>...<header>\n ") ✅ already flushed |
| 1 |
Content("<h1>Site Title</h1>\n<nav>\n ") ← partial, just assigned |
| 2 |
Content("\n </header>...\n</html>") ← sibling content |
| 3 |
Include(/menu) — pending ← nested include, appended after slot 2 |
| 4 |
Content("\n</nav>\n") ← trailing content from /header, also after slot 2 |
Because flushing is sequential (next_out walks forward), slot 2 (containing </header>...\n</html>) is flushed before slots 3–4. The nested menu and the </nav> close tag end up after </html>.
The invariant violation
The comment on Step 1 states:
After this inner loop self.queue is guaranteed empty. That invariant means DocumentHandler::write_bytes() called from within process_include writes directly to the caller-supplied slot_buf rather than re-queuing.
This invariant holds when process_include is called from Step 1 for already-completed requests. But when called from Step 5 (after select() returns), self.queue is also empty at that point — however, the issue isn't about writes going to the queue vs slot_buf. The issue is that any new includes discovered during dca="esi" parsing get pushed to self.queue, and when Step 1 next runs, they are assigned slots that come after all previously-assigned sibling slots.
Why on_eval doesn't have this bug
on_eval (for <esi:eval>) handles dca="esi" by creating an isolated processor and calling drain_queue on it before returning. This resolves all nested includes inline, so the parent's slot only sees the fully-resolved output. on_include/process_fragment_body does not do this — it creates a DocumentHandler that shares the parent processor's queue, so nested includes escape into the parent's slot-assignment scope.
Suggested fix
When process_fragment_body processes a dca="esi" fragment and the fragment contains <esi:include> tags, the nested includes need to be fully resolved before the parent's slot_buf is finalised. One approach:
After the element processing loop in process_fragment_body, drain any items that were pushed to self.queue during processing — similar to how on_eval calls isolated_processor.drain_queue():
// In process_fragment_body, after the element processing loop for dca="esi":
if dca_mode == DcaMode::Esi {
// ... existing parse + process loop ...
// Drain any nested includes that were queued during dca="esi" processing.
// Without this, nested includes escape to the parent's drain_queue scope
// and get assigned slots after sibling content, breaking document order.
self.drain_queue(output_writer, dispatcher, process_fragment_response)?;
}This ensures that the slot_buf written back to buf[slot] in drain_queue contains the fully-resolved fragment content, including all nested includes, preserving document order.
An alternative approach would be to splice new queue items into buf at buf_slot + 1 rather than appending them, but this would require shifting all subsequent slot indices and updating url_map entries.
Environment
esi crate: git main branch (commit 3ade5ef)- Tested via Viceroy local development with
process_stream API
- The
process_response API is equally affected (it delegates to process_stream internally)
- Only manifests with nesting depth ≥ 2 where an intermediate include uses
dca="esi" and itself contains <esi:include> tags
disclaimer: issue drafted by Claude Opus, with better understanding of Rust than I. As per #44, I'm trying to find the secret sauce for nested ESI and streaming output.
drain_queue: nesteddca="esi"includes are appended out of document orderSummary
When a
dca="esi"include contains further<esi:include>tags, the nested includes are emitted after all sibling content in the parent document rather than inline at the position of the original include. This breaks document order for any nesting depth > 1.Minimal reproduction
Setup — three fragments served by a backend called
"origin":/page(the outer ESI document):/header(fragment, included withdca="esi"):/menu(fragment, included withdca="esi"):Expected output
The
<ul>menu appears inside<nav>, inside<header>, before<main>.Actual output
The
<ul>menu appears after</html>, completely outside the document structure. The<nav>is empty.Root cause
The bug is in
drain_queueinesi/src/lib.rs— specifically the interaction between Step 1 (slot assignment) and Step 5 (processing completed includes).How slots work
drain_queuemaintains abuf: Vec<Option<Bytes>>where each queued element gets a sequential slot index. Content is flushed in order fromnext_outforward. The sequence for the example above after initial parsing:Content("<html>\n<body>\n <header>\n ")Include(/header)— pendingContent("\n </header>\n <main>Main content</main>\n</body>\n</html>")What happens when
/headercompletes (Step 5)process_includeis called withfragmentfor/header, which hasdca="esi". This invokesprocess_fragment_body, which:/headerresponse body as ESI viaparse_completeDocumentHandlerwriting to a localslot_buf: Vec<u8>When it hits
<h1>Site Title</h1>—DocumentHandler::write_byteschecksself.processor.queue.is_empty(). The queue is empty (Step 1 drained it), so it writes directly toslot_buf. ✅When it hits
<esi:include src="/menu" dca="esi" />—DocumentHandler::on_includedispatches the request and pushesQueuedElement::Includeontoself.processor.queue. ✅When it hits the trailing text
\n</nav>\n—DocumentHandler::write_bytessees the queue is non-empty (the/menuinclude is queued), so it pushesQueuedElement::Contentonto the queue instead of writing toslot_buf. This is correct behaviour for maintaining document order within the fragment.After
process_includereturns,slot_bufcontains only<h1>Site Title</h1>\n<nav>\n(everything before the nested include). This is assigned tobuf[1].The ordering problem
Back in
drain_queue's main loop, we return to Step 1, which drainsself.queue. The nested/menuinclude and the trailing</nav>content get assigned new slots at the end ofbuf:Content("<html>...<header>\n ")✅ already flushedContent("<h1>Site Title</h1>\n<nav>\n ")← partial, just assignedContent("\n </header>...\n</html>")← sibling contentInclude(/menu)— pending ← nested include, appended after slot 2Content("\n</nav>\n")← trailing content from /header, also after slot 2Because flushing is sequential (
next_outwalks forward), slot 2 (containing</header>...\n</html>) is flushed before slots 3–4. The nested menu and the</nav>close tag end up after</html>.The invariant violation
The comment on Step 1 states:
This invariant holds when
process_includeis called from Step 1 for already-completed requests. But when called from Step 5 (afterselect()returns),self.queueis also empty at that point — however, the issue isn't about writes going to the queue vsslot_buf. The issue is that any new includes discovered duringdca="esi"parsing get pushed toself.queue, and when Step 1 next runs, they are assigned slots that come after all previously-assigned sibling slots.Why
on_evaldoesn't have this bugon_eval(for<esi:eval>) handlesdca="esi"by creating an isolated processor and callingdrain_queueon it before returning. This resolves all nested includes inline, so the parent's slot only sees the fully-resolved output.on_include/process_fragment_bodydoes not do this — it creates aDocumentHandlerthat shares the parent processor's queue, so nested includes escape into the parent's slot-assignment scope.Suggested fix
When
process_fragment_bodyprocesses adca="esi"fragment and the fragment contains<esi:include>tags, the nested includes need to be fully resolved before the parent'sslot_bufis finalised. One approach:After the element processing loop in
process_fragment_body, drain any items that were pushed toself.queueduring processing — similar to howon_evalcallsisolated_processor.drain_queue():This ensures that the
slot_bufwritten back tobuf[slot]indrain_queuecontains the fully-resolved fragment content, including all nested includes, preserving document order.An alternative approach would be to splice new queue items into
bufatbuf_slot + 1rather than appending them, but this would require shifting all subsequent slot indices and updatingurl_mapentries.Environment
esicrate: gitmainbranch (commit3ade5ef)process_streamAPIprocess_responseAPI is equally affected (it delegates toprocess_streaminternally)dca="esi"and itself contains<esi:include>tags