GH-50944: [C++] Replace RapidJSON with simdjson in JSON chunker by Reranko05 · Pull Request #50945 · apache/arrow

Reranko05 · 2026-08-21T13:39:30Z

Rationale for this change

This PR continues the simdjson migration by replacing the RapidJSON-based JSON boundary detection used by the JSON chunker.

The existing implementation uses RapidJSON's streaming parser to identify complete JSON values. This change replaces that logic with structural boundary detection and simdjson validation.

Changes

Replace the RapidJSON custom stream and boundary detection.
Detect complete JSON values with structural scanning.
Validate complete candidates with simdjson::dom::parser.
Preserve incomplete-value, block-boundary, and error handling behavior.
Update the affected error expectation.

Fixes: #50944

GitHub Issue: [C++] Replace RapidJSON with simdjson in JSON chunker #50944

Reranko05 · 2026-08-21T14:05:22Z

While working on this migration, @rok's earlier implementation rok#47 of the simdjson-based JSON chunker. It was very helpful. Thanks :)

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

pitrou

I don't understand why this is parsing JSON by hand?

It seems that we might be able to use simdjson::ondemand::parser::iterate_many.

Reranko05 · 2026-08-24T10:05:59Z

I don't understand why this is parsing JSON by hand?
It seems that we might be able to use simdjson::ondemand::parser::iterate_many.

I initially tried using simdjson::ondemand::parser::iterate_many, but after a few attempts I ran into boundary-handling issues with the chunker semantics. I then referred to an earlier implementation by @rok as a reference and followed that approach.

That said, I agree that parsing JSON manually here is not ideal. I'll revisit this using iterate_many and address the other review comments as well.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Reranko05 · 2026-08-25T12:46:28Z

@pitrou I tried using simdjson::ondemand::parser::iterate_many(). Most tests pass, but PropagateErrorsNonLinewiseChunker behaves differently: malformed input is split into a separate document and the error is then reported by the JSON parser, whereas the current implementation reports a chunker error.

Manual structural parsing approach preserved the existing test behavior. @rok, since your earlier implementation was helpful here, do you have any suggestions on how to preserve the current error behavior with iterate_many()?

pitrou · 2026-08-25T12:51:17Z

Most tests pass, but PropagateErrorsNonLinewiseChunker behaves differently: malformed input is split into a separate document and the error is then reported by the JSON parser, whereas the current implementation reports a chunker error.

As long as an error is reported while reading the JSON stream, I don't think we care if it's reported by the chunker or the parser.

Reranko05 · 2026-08-26T11:33:30Z

@pitrou I gave iterate_many() more that a few attempts, but I am running into semantic differences with the previous RapidJSON implementation, especially around stopping after a complete value when it is followed by a partial or malformed value. At this point, I think manually finding the boundary of the first complete object/array and then validating that slice with simdjson may be simpler and closer to the existing behavior. Do you think that would be a reasonable approach?

pitrou · 2026-08-26T11:43:13Z

@pitrou I gave iterate_many() more that a few attempts, but I am running into semantic differences with the previous RapidJSON implementation, especially around stopping after a complete value when it is followed by a partial or malformed value.

Hmm, I see. Thanks for trying anyway :-)

At this point, I think manually finding the boundary of the first complete object/array and then validating that slice with simdjson may be simpler and closer to the existing behavior. Do you think that would be a reasonable approach?

Well, as a last resort, yes. The problem:

We're writing our own JSON parser, which means we must careful test it.
We're losing performance unless we implement our own SIMD optimizations.

Thoughts @HuaHuaY @cyb70289 ?

cyb70289 · 2026-08-27T02:47:23Z

Try to understand the issue. Is it that json strings legal for rapidjson may fail on simdjson, makes future Arrow release potentially incompatible to old version?

Writing our own optimized version looks not ideal. Can we just use simdjson? It's state-of-the-art, and even with self written object delimiter, there's still incompatibility risk I'm afraid.

Reranko05 · 2026-08-27T08:18:07Z

@cyb70289 I don't think the issue is JSON compatibility between RapidJSON and simdjson. The main issue I ran into is the boundary/streaming semantics.

The previous RapidJSON implementation uses kParseStopWhenDoneFlag, so it stops as soon as the first complete top-level value is parsed, even if the remaining input contains a partial or malformed value.

With iterate_many(), I wasn't able to reproduce that behavior: errors in subsequent/incomplete data can affect whether we successfully identify the preceding complete value.

I agree that writing our own optimized JSON parser would not be ideal. The manual approach I used, and which @rok also implemented in rok#47, only scans for the boundary of the first complete object or array while respecting strings and escapes, and then lets simdjson perform the actual JSON validation. But I agree this still introduces complexity and needs careful testing.

If there is a way to use simdjson directly while preserving the old stop-after-one-value semantics, that would definitely be preferable.

pitrou · 2026-08-27T09:06:21Z

With iterate_many(), I wasn't able to reproduce that behavior: errors in subsequent/incomplete data can affect whether we successfully identify the preceding complete value.

Is that a problem? We want to keep compatibility when parsing valid JSON streams. The failure mode for an invalid JSON stream can change.

cyb70289 · 2026-08-27T09:27:33Z

A discussion about ignoring trailing garbage in simdjson. Looks there're real use cases lenient parsing can be useful.
simdjson/simdjson#2502

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

python/pyarrow/tests/test_json.py:155

The test now accepts any ValueError, which weakens coverage for the expected failure mode when block_size is too small. Other tests in this file still assert the specific "straddling object ..." error; keeping a message match here helps ensure regressions don’t silently change the error raised for undersized blocks.

                with pytest.raises(ValueError):
                    self.read_bytes(data, read_options=read_options,
                                    parse_options=parse_options)

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Reranko05 · 2026-09-01T19:07:15Z

+            pass
+        else:
+            with pytest.raises(pa.ArrowInvalid):
+                reader.read_next_batch()


Updated for readahead, since the error may be surfaced by either the chunker or parser.

Can we use this?

if self.use_threads: # TODO: Add a comment why the first read_next_batch() may not raise an error with threads. with pytest.raises(pa.ArrowInvalid, match="JSON (parse|chunk) error"): reader.read_next_batch() reader.read_next_batch() else: with pytest.raises(pa.ArrowInvalid, match="JSON parse error"): reader.read_next_batch()

kou · 2026-09-01T21:58:24Z

+            pass
+        else:
+            with pytest.raises(pa.ArrowInvalid):
+                reader.read_next_batch()


Can we use this?

if self.use_threads: # TODO: Add a comment why the first read_next_batch() may not raise an error with threads. with pytest.raises(pa.ArrowInvalid, match="JSON (parse|chunk) error"): reader.read_next_batch() reader.read_next_batch() else: with pytest.raises(pa.ArrowInvalid, match="JSON parse error"): reader.read_next_batch()

kou · 2026-09-02T00:50:05Z

-        // found incomplete object or block is empty
+
+    simdjson::padded_string padded(block);
+    simdjson::ondemand::parser parser;


Cam we reuse simdjson::ondemand::parser in FindLast() and ConsumeWholeObject() for performance?

See also: https://github.com/simdjson/simdjson/blob/master/doc/performance.md#reusing-the-parser-for-maximum-efficiency

Done. Reused a single simdjson::ondemand::parser in both FindLast() and ConsumeWholeObject().

kou · 2026-09-02T00:54:20Z

-      if (length == string_view::npos || length == 0) {
-        // found incomplete object or block is empty
+
+    simdjson::padded_string padded(block);


Can we use simdjson::padded_string_view if possible by introducing our internal buffer?

See also: https://github.com/simdjson/simdjson/blob/master/doc/performance.md#reusing-string-buffers

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

rok · 2026-09-02T12:59:35Z

+
+    simdjson::ondemand::document_stream stream;
+
+    if (parser_.iterate_many(GetPaddedStringView(block)).get(stream) !=


iterate_many rejects strings larger than 1mb. We could:

Suggested change

if (parser_.iterate_many(GetPaddedStringView(block)).get(stream) !=

if (parser_.iterate_many(GetPaddedStringView(block), block.size()).get(stream) !=

It says that each document must not be larger than 1MB, not that the whole string is limited.

Also, I don't understand why the suggested fix would change anything?

rok · 2026-09-02T13:00:06Z

+
+    simdjson::ondemand::document_stream stream;
+
+    if (parser_.iterate_many(GetPaddedStringView(input)).get(stream) !=


See comment above

Suggested change

if (parser_.iterate_many(GetPaddedStringView(input)).get(stream) !=

if (parser_.iterate_many(GetPaddedStringView(input), input.size()).get(stream) !=

rok · 2026-09-02T13:29:12Z

+    } else if (block.empty()) {
+      input = simdjson::padded_string(partial);
+    } else {
+      simdjson::padded_string_builder builder(partial.size() + block.size());


padded_string_builder was introduced in v4.3.0 so builds against 4.0-4.2 will likely fail.

Pity. Either we can backport it, emulate it, or use #if -based version checks.

github-actions Bot added Component: C++ awaiting review Awaiting review labels Aug 21, 2026

Reranko05 added CI: Extra: C++ Run extra C++ CI and removed awaiting review Awaiting review labels Aug 21, 2026

Reranko05 marked this pull request as ready for review August 21, 2026 16:33

Reranko05 requested a review from pitrou as a code owner August 21, 2026 16:33

Copilot AI lite review requested due to automatic review settings August 21, 2026 16:33

Copilot AI reviewed Aug 21, 2026

Reranko05 requested review from kou and rok August 21, 2026 16:33

github-actions Bot added the awaiting review Awaiting review label Aug 21, 2026

Replace RapidJSON with simdjson in JSON chunker

be4c2a1

Reranko05 force-pushed the gh-35460-chunker branch from e9fc9fe to be4c2a1 Compare August 24, 2026 07:52

pitrou requested changes Aug 24, 2026

View reviewed changes

Comment thread cpp/src/arrow/json/chunker.cc Outdated

Comment thread cpp/src/arrow/json/chunker.cc Outdated

Comment thread cpp/src/arrow/json/chunker.cc Outdated

Comment thread cpp/src/arrow/json/chunker.cc Outdated

Comment thread cpp/src/arrow/json/chunker.cc Outdated

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Aug 24, 2026

WIP: preserve chunker behavior before iterate_many

aa6e20d

Copilot AI review requested due to automatic review settings August 24, 2026 20:06

Copilot AI reviewed Aug 24, 2026

WIP: Iterate Many

a5b89a7

Address Feedback

ce19130

Reranko05 force-pushed the gh-35460-chunker branch from 3c934c5 to ce19130 Compare September 1, 2026 18:04

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Comment thread cpp/src/arrow/json/reader_test.cc Outdated

Copilot AI review requested due to automatic review settings September 1, 2026 18:06

Copilot AI reviewed Sep 1, 2026

Address Copilot Feedback

b64f91c

Copilot AI review requested due to automatic review settings September 1, 2026 18:16

Copilot AI reviewed Sep 1, 2026

Python test fix

bdc7f56

Copilot AI review requested due to automatic review settings September 1, 2026 18:42

Copilot AI reviewed Sep 1, 2026

Reranko05 commented Sep 1, 2026

View reviewed changes

kou reviewed Sep 2, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 2, 2026

Test fixing

e7d2595

Copilot AI review requested due to automatic review settings September 2, 2026 11:15

Copilot AI reviewed Sep 2, 2026

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 2, 2026

Reranko05 force-pushed the gh-35460-chunker branch from 2b91c84 to 729a6a8 Compare September 2, 2026 11:45

Copilot AI review requested due to automatic review settings September 2, 2026 11:45

Copilot AI reviewed Sep 2, 2026

Address Feedback

05033bf

Copilot AI review requested due to automatic review settings September 2, 2026 11:54

Reranko05 force-pushed the gh-35460-chunker branch from 729a6a8 to 05033bf Compare September 2, 2026 11:54

Copilot AI reviewed Sep 2, 2026

rok reviewed Sep 2, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 2, 2026


		simdjson::ondemand::document_stream stream;

		if (parser_.iterate_many(GetPaddedStringView(block)).get(stream) !=

Uh oh!

Conversation

Reranko05 commented Aug 21, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Changes

Uh oh!

Reranko05 commented Aug 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reranko05 commented Aug 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Aug 25, 2026

Uh oh!

pitrou commented Aug 25, 2026

Uh oh!

Reranko05 commented Aug 26, 2026

Uh oh!

pitrou commented Aug 26, 2026

Uh oh!

cyb70289 commented Aug 27, 2026

Uh oh!

Reranko05 commented Aug 27, 2026

Uh oh!

pitrou commented Aug 27, 2026

Uh oh!

cyb70289 commented Aug 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Reranko05 commented Aug 21, 2026 •

edited by github-actions Bot

Loading