The event based parser lacks the ability to pause and resume parsing.
This is a problem because if processing of an event takes some time (for example, it requires doing some asynchronous i/o or network request) the parser will continue to parse the document and emit more events.
This has two unfortunate effects:
- it will quickly fill up memory if the input file is significantly large
- event handler will be executed in a non predictable order, causing race conditions.
This completely negate the main advantage of having an event based parser, that is the ability to deal with large files in a streaming fashion.
Example:
const parser = new SAXParser();
const writer = await Deno.open("data.jsonl", { write: true });
const encoder = new TextEncoder()
parser.on("end_element", async(e) => {
const row = JSON.stringify(Object.fromEntries(e.attributes.map(a => [a.qName, a.value])) + "\n";
let buffer = encoder.encode(row);
while (buffer.length > 0) {
const written = await writer.write(buffer);
buffer = buffer.slice(written);
}
})
const reader = await Deno.open("bigfile.xml");
await parser.parse(reader, "utf-8");
In this case, if the element being parsed is large, writer.write(buffer) won't be able to write the entire buffer in a single call, and other elements may begin being written on the file before the previous elements have finished writing.
A normal way to handle this is to pause parsing when an event handler starts, resuming it when the event handler ends.
The event based parser lacks the ability to pause and resume parsing.
This is a problem because if processing of an event takes some time (for example, it requires doing some asynchronous i/o or network request) the parser will continue to parse the document and emit more events.
This has two unfortunate effects:
This completely negate the main advantage of having an event based parser, that is the ability to deal with large files in a streaming fashion.
Example:
In this case, if the element being parsed is large,
writer.write(buffer)won't be able to write the entire buffer in a single call, and other elements may begin being written on the file before the previous elements have finished writing.A normal way to handle this is to pause parsing when an event handler starts, resuming it when the event handler ends.