Skip to content

Missing ability to pause/resume the parser to prevent race condition in async event handlers & manage backpressure #6

Description

@pqnet

The event based parser lacks the ability to pause and resume parsing.
This is a problem because if processing of an event takes some time (for example, it requires doing some asynchronous i/o or network request) the parser will continue to parse the document and emit more events.
This has two unfortunate effects:

  • it will quickly fill up memory if the input file is significantly large
  • event handler will be executed in a non predictable order, causing race conditions.

This completely negate the main advantage of having an event based parser, that is the ability to deal with large files in a streaming fashion.

Example:

const parser = new SAXParser();
const writer = await Deno.open("data.jsonl", { write: true });
const encoder = new TextEncoder()
parser.on("end_element", async(e) => {
    const row = JSON.stringify(Object.fromEntries(e.attributes.map(a => [a.qName, a.value])) + "\n";
    let buffer = encoder.encode(row);
    while (buffer.length > 0) {
        const written = await writer.write(buffer);
        buffer = buffer.slice(written);
    }
})

const reader = await Deno.open("bigfile.xml");
await parser.parse(reader, "utf-8");

In this case, if the element being parsed is large, writer.write(buffer) won't be able to write the entire buffer in a single call, and other elements may begin being written on the file before the previous elements have finished writing.

A normal way to handle this is to pause parsing when an event handler starts, resuming it when the event handler ends.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions