POC: Switch from ujson to orjson/stdlib - #67203
Draft
rhshadrach wants to merge 10 commits into
Draft
Conversation
# Conflicts: # doc/source/whatsnew/v3.1.0.rst # pandas/tests/io/json/test_ujson.py
Member
Author
|
cc @pandas-dev/pandas-core If I can get some head nods, I'll start the deprecation process for the three arguments above as part of 3.x. The plan would then be to add orjson and stdlib alongside ujson, and deprecate ujson all in 3.x. |
Member
|
+1 on deprecating those precision keywords Only question that comes to mind is how this handles pyarrow/nullable cases |
Member
Author
Same as main, via |
mroeschke
reviewed
Aug 30, 2026
| * ``encoding`` : The encoding to use to decode py3 bytes. | ||
| * ``chunksize`` : when used in combination with ``lines=True``, return a ``pandas.api.typing.JsonReader`` which reads in ``chunksize`` lines per iteration. | ||
| * ``engine``: Either ``"ujson"``, the built-in JSON parser, or ``"pyarrow"`` which dispatches to pyarrow's ``pyarrow.json.read_json``. | ||
| * ``engine``: Either ``"json"``, the default parser (`orjson <https://github.com/ijl/orjson>`__ when installed, the standard library ``json`` module otherwise), or ``"pyarrow"`` which dispatches to pyarrow's ``pyarrow.json.read_json``. |
Member
There was a problem hiding this comment.
Is there a performance difference between the stdlib json and the current ujson parser, given orjson is current optional in this PR?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR shows what the final result would roughly look like if we were to remove ujson and replace with orjson (opitonal) and stdlib (used when orjson is not available). Written by Claude Fable 5, not (thoroughly) reviewed or refined by me - I don't intend to do so for this POC but will when we go to actually make the switch.
tl;dr: With a few deprecations that I think we should do anyways, we can do a pretty good swap out with orjson. Benchmarks below show this is by-and-large positive on performance.
Notable API decisions:
double_precisionhas been removed. Floats are written with shortest round-trip repr instead of being rounded to 10 decimals. Users who want fewer digits can round the data first.force_asciihas been removed.indentrestricted to{0, 2}, and indented output uses": "between key and value (matching json.dumps(indent=2)). We can implement our own indentation for use with orjson, it'd likely be slow and not often used.precise_floatofread_jsonhas been removed; floats are always parsed with full precision.date_formatdeprecation and theread_jsonconvert_datesdeprecation are not enforced in this PR.NaNreading behavior preserved.NaN/±infcontinue to write as null (both engines).This PR also fixes one bug (couldn't find an issue):
Benchmarks comparing
orjsonagainst main,new / old, so< 1means "got faster".Details