Implementation for enabling intra-segment search #19704
Merged
+1,116
−117
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Prudhvi Godithi <[email protected]>
Signed-off-by: Prudhvi Godithi <[email protected]>
Signed-off-by: Prudhvi Godithi <[email protected]>
Signed-off-by: Prudhvi Godithi <[email protected]>
Contributor
|
❌ Gradle check result for 73fba5f: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Member
Author
big5 data distributionThis is on the big5 workload when configured/default slice count is less than partitions requestedShow logs 1Creation of slices with partitions and distributed meeting Lucene constrains (no fallback as we have enough slices to take all the partitions)Show logs 2Parallel execution of slices with segments and partitionsShow logs 3 |
Member
Author
|
IMO this should be a good start for testing the queries and bechmarks. In parallel I will update the PR once I have this sorted #18851. Either way as posted in description even though we have auto partition and slice logic with #18851 we still honor user configured settings. |
Contributor
|
I ended up using the noaa dataset to work on running the term query Weirdly enough, I did not notice much of an improvement. Without intrasegment concurrent search: With intra segment concurrent search: |
Signed-off-by: Prudhvi Godithi <[email protected]>
github-actions
bot
added
enhancement
Contributor
|
❌ Gradle check result for 5ffd097: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Prudhvi Godithi <[email protected]>
Contributor
|
❌ Gradle check result for 019470b: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Prudhvi Godithi <[email protected]>
Member
Author
|
{"run-benchmark-test": "id_3"} |
Contributor
|
The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/4847/ . Final results will be published once the job is completed. |
Member
Author
|
{"run-benchmark-test": "id_3"} |
Contributor
|
The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/4848/ . Final results will be published once the job is completed. |
Contributor
|
❌ Gradle check result for 49fd872: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Collaborator
Benchmark ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-pull-request/4848/
|
Collaborator
Benchmark Baseline Comparison ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-compare/171/
|
Signed-off-by: Prudhvi Godithi <[email protected]>
Member
Author
|
{"run-benchmark-test": "id_6"} |
Contributor
|
The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/4859/ . Final results will be published once the job is completed. |
Contributor
|
The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/5859/ . Final results will be published once the job is completed. |
jainankitk
reviewed
Jan 28, 2026
Contributor
jainankitk
left a comment
jainankitk
left a comment
There was a problem hiding this comment.
Few more comments, otherwise the change mostly looks good!
server/src/main/java/org/opensearch/search/deciders/IntraSegmentSearchDecider.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/search/internal/MaxTargetSliceSupplier.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/search/internal/MaxTargetSliceSupplier.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/search/internal/MaxTargetSliceSupplier.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/search/internal/MaxTargetSliceSupplier.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/search/internal/MaxTargetSliceSupplier.java
Show resolved
Hide resolved
Signed-off-by: Prudhvi Godithi <[email protected]>
Contributor
|
❌ Gradle check result for 3fd164c: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Prudhvi Godithi <[email protected]>
Member
Author
|
Thank @expani, I dont deny your concern #19704 (comment) priority queue works for simple LPT, but with the |
Signed-off-by: Prudhvi Godithi <[email protected]>
Contributor
|
❌ Gradle check result for 677683c: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Collaborator
Benchmark ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-pull-request/5859/
|
Contributor
|
❌ Gradle check result for 677683c: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Collaborator
Benchmark Baseline Comparison ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-compare/234/
|
atris
requested changes
Jan 28, 2026
Contributor
atris
left a comment
atris
left a comment
There was a problem hiding this comment.
Mostly looks good. Left some comments
server/src/main/java/org/opensearch/search/aggregations/AggregatorFactory.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/search/deciders/IntraSegmentSearchDecider.java
Show resolved
Hide resolved
Signed-off-by: Prudhvi Godithi <[email protected]>
Member
Author
Thanks @atris addressed your comments. |
atris
approved these changes
Jan 28, 2026
Contributor
|
❌ Gradle check result for 31bdfdd: null Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Contributor
|
❌ Gradle check result for 31bdfdd: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
jainankitk
approved these changes
Jan 28, 2026
Contributor
|
❌ Gradle check result for 31bdfdd: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Prudhvi Godithi <[email protected]>
Contributor
|
❌ Gradle check result for 6bf2c10: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Contributor
|
❌ Gradle check result for 6bf2c10: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Contributor
github-project-automation
bot
moved this from In Progress
to Done
in Performance Roadmap
Collaborator
Benchmark ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-pull-request/5875/
|
Collaborator
Benchmark Baseline Comparison ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-compare/238/
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Initial WIP implementation
Creates segment partitions and distributes partitions across slices. Ensures Lucene constraint is satisfied. Each partition from a segment goes to a different slice with round robin assignment
Partitions are based on
min_segment_sizeandpartitions_per_segmentsettings. I'm working closely on implementing better auto partitions to slice mechanism [Intra-SegmentConcurrentSearch] Slicing mechanism #18851. Even with this auto partition logic the idea is to still honor user passedmin_segment_sizeandpartitions_per_segmentsettings.Falls back to default behavior (search by full segment) when there insufficient slices for the
partitions_per_segment. This is to ensure same segment partitions go to different slice. In this the idea is to increase themax_slice_countand do not rely oncomputeDefaultSliceCount()method during cluster start up.More details here on default Lucene partition to slice mechanism [Intra-SegmentConcurrentSearch] Slicing mechanism #18851 (comment).
To test the benchmarks we can disable the
logger.infolines. This is just to print the partition and assigned slice distribution. See Implementation for enabling intra-segment search #19704 (comment).Dec 18 2025 (Latest Updated changes)
I have updated the PR with the following changes. This PR introduces configuration settings and a decision framework for intra-segment search.
Updated settings:
Prerequisites:
supportsIntraSegmentSearchto returntrue.If either is disabled → intra-segment disabled, skip further evaluation.
Query Evaluation: Walks query tree, checks support for intra-segment and concurrent segment search:
Aggregation Evaluation: Checks AggregatorFactory for intra segment search support:
Final Decision:
Related Issues
Related to #19694 and part of #18851.
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.
Summary by CodeRabbit
New Features
New Settings
Changes
✏️ Tip: You can customize this high-level summary in your review settings.