Skip to content

refactor(lexer): split classify pipeline stage into multiple files - #26303

Draft
overlookmotel wants to merge 1 commit into
mainfrom
om/09-03-refactor_lexer_split_classify_pipeline_stage_into_multiple_files
Draft

refactor(lexer): split classify pipeline stage into multiple files#26303
overlookmotel wants to merge 1 commit into
mainfrom
om/09-03-refactor_lexer_split_classify_pipeline_stage_into_multiple_files

Conversation

@overlookmotel

@overlookmotel overlookmotel commented Sep 3, 2026

Copy link
Copy Markdown
Member

Pure refactor. No code changes, just moving code around.

Split classify pipeline stage into multiple files.

  • avx2.rs / generic.rs - SIMD and generic impls.
  • common.rs - Code shared between both impls.
  • mod.rs - Minimal entry point, chooses which impl to load dependent on target.

Additionally, a few stylistic changes in Oxc "house style":

  • Re-order code so it reads top-to-bottom - imports and consts first, then entry points, private fns/helpers lower down.
  • Combine imports into a single use statements.
  • Line breaks between functions.

The split of SIMD/generic into separate files is the main point. This is a precursor to switching over to runtime feature detection, instead of compile-time.

Copy link
Copy Markdown
Member Author

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

  • 0-merge - adds this PR to the back of the merge queue
  • hotfix - for urgent changes, fast-track this PR to the front of the merge queue

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The changes appear to be a straightforward module/file split with preserved call sites and conditional compilation behavior, and I did not find any functional or API-affecting issues within the changed regions.

Pull request overview

Refactors the pipeline::classify stage in oxc_lexer by splitting the previous single-module implementation into multiple files while keeping the same compile-time AVX2 vs generic selection and shared helpers.

Changes:

  • Introduces a new pipeline/classify/ module with mod.rs as a minimal entry point that conditionally selects AVX2 or generic classify.
  • Moves the AVX2 implementation into avx2.rs and the non-SIMD implementation into generic.rs.
  • Extracts shared logic (misc_pre / misc_post and UTF-8 / Unicode whitespace helpers) into common.rs and re-exports it from the classify module.
File summaries
File Description
crates/oxc_lexer/src/pipeline/classify/mod.rs New module entry point selecting AVX2 vs generic implementation and re-exporting shared helpers.
crates/oxc_lexer/src/pipeline/classify/avx2.rs Contains the AVX2/BMI2 SIMD classify implementation moved from the previous monolithic module.
crates/oxc_lexer/src/pipeline/classify/generic.rs Contains the non-SIMD classify implementation and lookup-table construction moved from the previous monolithic module.
crates/oxc_lexer/src/pipeline/classify/common.rs Contains shared misc_pre / misc_post logic and helper routines extracted from the previous module.
crates/oxc_lexer/src/pipeline/classify.rs Removes the old monolithic classify module file after splitting into the new directory-based module.
Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

@overlookmotel overlookmotel self-assigned this Sep 3, 2026
@overlookmotel
overlookmotel marked this pull request as draft September 3, 2026 20:05
@overlookmotel

Copy link
Copy Markdown
Member Author

This stack marked as draft as expecting a PR from @ShanonJackson which plan to merge first, and then will rebase this on top.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-lexer Area - Lexer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants