refactor(lexer): split classify pipeline stage into multiple files - #26303
refactor(lexer): split classify pipeline stage into multiple files#26303overlookmotel wants to merge 1 commit into
classify pipeline stage into multiple files#26303Conversation
How to use the Graphite Merge QueueAdd either label to this PR to merge it via the merge queue:
You must have a Graphite account in order to use the merge queue. Sign up using this link. An organization admin has enabled the Graphite Merge Queue in this repository. Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue. This stack of pull requests is managed by Graphite. Learn more about stacking. |
There was a problem hiding this comment.
🟢 Approval recommended
The changes appear to be a straightforward module/file split with preserved call sites and conditional compilation behavior, and I did not find any functional or API-affecting issues within the changed regions.
Pull request overview
Refactors the pipeline::classify stage in oxc_lexer by splitting the previous single-module implementation into multiple files while keeping the same compile-time AVX2 vs generic selection and shared helpers.
Changes:
- Introduces a new
pipeline/classify/module withmod.rsas a minimal entry point that conditionally selects AVX2 or genericclassify. - Moves the AVX2 implementation into
avx2.rsand the non-SIMD implementation intogeneric.rs. - Extracts shared logic (
misc_pre/misc_postand UTF-8 / Unicode whitespace helpers) intocommon.rsand re-exports it from theclassifymodule.
File summaries
| File | Description |
|---|---|
| crates/oxc_lexer/src/pipeline/classify/mod.rs | New module entry point selecting AVX2 vs generic implementation and re-exporting shared helpers. |
| crates/oxc_lexer/src/pipeline/classify/avx2.rs | Contains the AVX2/BMI2 SIMD classify implementation moved from the previous monolithic module. |
| crates/oxc_lexer/src/pipeline/classify/generic.rs | Contains the non-SIMD classify implementation and lookup-table construction moved from the previous monolithic module. |
| crates/oxc_lexer/src/pipeline/classify/common.rs | Contains shared misc_pre / misc_post logic and helper routines extracted from the previous module. |
| crates/oxc_lexer/src/pipeline/classify.rs | Removes the old monolithic classify module file after splitting into the new directory-based module. |
Review details
- Files reviewed: 5/5 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
|
This stack marked as draft as expecting a PR from @ShanonJackson which plan to merge first, and then will rebase this on top. |

Pure refactor. No code changes, just moving code around.
Split
classifypipeline stage into multiple files.avx2.rs/generic.rs- SIMD and generic impls.common.rs- Code shared between both impls.mod.rs- Minimal entry point, chooses which impl to load dependent on target.Additionally, a few stylistic changes in Oxc "house style":
usestatements.The split of SIMD/generic into separate files is the main point. This is a precursor to switching over to runtime feature detection, instead of compile-time.