handled empty results in terms aggs #20451

animodak7 · 2026-01-21T05:38:30Z

Solves -
Empty aggs cols returned from opensearch failing sql plugin while generating results

java.util.NoSuchElementException: No value present
        at java.base/java.util.Optional.get(Optional.java:143) ~[?:?]
        at org.opensearch.search.aggregations.bucket.terms.AbstractStringTermsAggregator.convert(AbstractStringTermsAggregator.java:124) ~[opensearch-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
        at org.opensearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.convert(GlobalOrdinalsStringTermsAggregator.java:100) ~[opensearch-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
        at org.opensearch.search.query.SearchEngineResultConversionUtils.lambda$convertDFResultGeneric$1(SearchEngineResultConversionUtils.java:67) ~[opensearch-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722) ~[?:?]
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at org.opensearch.search.query.SearchEngineResultConversionUtils.convertDFResultGeneric(SearchEngineResultConversionUtils.java:67) ~[opensearch-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
        at org.opensearch.search.query.QueryPhase.execute(QueryPhase.java:169) ~[opensearch-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
        at org.opensearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:743) ~[opensearch-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
        at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:901) ~[opensearch-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
        at org.opensearch.search.SearchService$3.onResponse(SearchService.java:967) ~[opensearch-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]

Replication steps -

curl --location --request DELETE 'localhost:9200/test-grab-index'

curl --location --request PUT 'http://localhost:9200/test-grab-index' \
--header 'Content-Type: application/json' \
--data-raw '{
    "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 0,
        "refresh_interval": -1,
        "optimized.enabled": true,
        "index.mapping.total_fields.limit": 1000,
        "index.analysis.filter.custom_word_delimiter.split_on_numerics": "false",
        "index.analysis.filter.custom_word_delimiter.split_on_case_change": "false",
        "index.analysis.filter.custom_word_delimiter.generate_word_parts": "true",
        "index.analysis.filter.custom_word_delimiter.type": "word_delimiter",
        "index.analysis.filter.custom_word_delimiter.preserve_original": "false",
        "index.analysis.analyzer.custom_message_analyzer.filter": ["custom_word_delimiter", "lowercase"],
        "index.analysis.analyzer.custom_message_analyzer.type": "custom",
        "index.analysis.analyzer.custom_message_analyzer.tokenizer": "whitespace"
    },
    "mappings": {
        "dynamic": "false",
        
        "properties": {
            "@timestamp": {
                "type": "date"
            },
            "additional_data": {
                "type": "text",
                "analyzer": "standard"
            },
            "app_name": {
                "type": "keyword"
            },
            "availability_zone": {
                "type": "keyword"
            },
            "class": {
                "type": "keyword"
            },
            "common_booking_code": {
                "type": "keyword"
            },
            "common_client": {
                "type": "keyword"
            },
            "common_client_ip": {
                "type": "keyword"
            },
            "common_driver_id": {
                "type": "long"
            },
            "common_endpoint": {
                "type": "keyword"
            },
            "common_endpoint_elapsed": {
                "type": "long"
            },
            "common_error": {
                "type": "text",
                "analyzer": "standard"
            },
            "common_error_type": {
                "type": "keyword"
            },
            "common_eta": {
                "type": "text",
                "analyzer": "standard"
            },
            "common_http_url": {
                "type": "keyword"
            },
            "common_merchant_id": {
                "type": "keyword"
            },
            "common_order_id": {
                "type": "keyword"
            },
            "common_passenger_id": {
                "type": "long"
            },
            "common_provider": {
                "type": "keyword"
            },
            "common_reference": {
                "type": "keyword"
            },
            "common_region": {
                "type": "keyword"
            },
            "common_request": {
                "type": "text",
                "analyzer": "standard"
            },
            "common_request_id": {
                "type": "keyword"
            },
            "common_response": {
                "type": "text",
                "analyzer": "standard"
            },
            "common_service": {
                "type": "keyword"
            },
            "common_trace_id": {
                "type": "keyword"
            },
            "common_user": {
                "type": "keyword"
            },
            "common_vehicle_type": {
                "type": "long"
            },
            "event_type": {
                "type": "keyword"
            },
            "gopoi_accuracy": {
                "type": "double"
            },
            "gopoi_keyword": {
                "type": "keyword"
            },
            "gopoi_type": {
                "type": "keyword"
            },
            "gopoi_xcountry": {
                "type": "keyword"
            },
            "hostname": {
                "type": "keyword"
            },
            "invalid_data": {
                "type": "text",
                "analyzer": "standard"
            },
            "jaya_rental_agreement_id": {
                "type": "long"
            },
            "jaya_user": {
                "type": "keyword"
            },
            "log_level": {
                "type": "keyword"
            },
            "log_write_epoch": {
                "type": "long"
            },
            "message": {
                "type": "text",
                "analyzer": "custom_message_analyzer"
            },
            "orbital_location_id": {
                "type": "long"
            },
            "pipeline_error": {
                "type": "text",
                "index": false
            },
  
            "source": {
                "type": "text",
                "fields": {
                    "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                    }
                }
            },
            "stacktrace": {
                "type": "text",
                "analyzer": "standard"
            },
            "tags": {
                "type": "text",
                "fields": {
                    "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                    }
                }
            },
            "ucm_config_key": {
                "type": "keyword"
            },
            "ucm_operation": {
                "type": "keyword"
            },
            "ucm_repository": {
                "type": "keyword"
            },
            "ucm_running_env": {
                "type": "keyword"
            },
            "vpc_name": {
                "type": "keyword"
            }
        }
    }
}'

curl --location --request POST 'http://localhost:9200/_bulk' \
--header 'Content-Type: application/json' \
--data-raw '{"index":{"_index":"test-grab-index"}}
{"@timestamp":"2024-01-15T10:00:00","additional_data":"data1","app_name":"booking-service","auth":{"metadata":{"instance_id":"i-001","role":"service"}},"availability_zone":"eu-west-1a","class":"BookingController","common_booking_code":"BC001","common_client":"mobile-app","common_client_ip":"192.168.1.1","common_driver_id":5001,"common_endpoint":"/api/v1/bookings","common_endpoint_elapsed":100,"common_error":"none","common_eta":"5min","common_http_url":"https://api.example.com/bookings","common_merchant_id":"MERCH001","common_order_id":"ORD001","common_passenger_id":1001,"common_provider":"grab","common_reference":"REF001","common_region":"SEA","common_request":"request_body","common_request_id":"REQ001","common_response":"response_body","common_service":"booking","common_trace_id":"TRACE001","common_user":"user001","common_vehicle_type":1,"event_type":"order_created","gopoi_accuracy":10,"gopoi_keyword":"airport","gopoi_type":"pickup","gopoi_xcountry":"SG","hostname":"host1","invalid_data":"none","jaya_rental_agreement_id":9001,"jaya_user":"jaya001","log_level":"INFO","log_write_epoch":1705315200,"message":"Order created successfully","orbital_location_id":7001,"pipeline_error":"none","request":{"headers":{"x-forwarded-for":"10.0.0.1"},"operation":"POST","path":"/bookings"},"source":"application","stacktrace":"none","tags":"production","ucm_config_key":"feature.flag","ucm_operation":"read","ucm_repository":"config-repo","ucm_running_env":"production","vpc_name":"vpc-prod"}
{"index":{"_index":"test-grab-index"}}
{"@timestamp":"2024-01-15T10:05:00","additional_data":"data2","app_name":"driver-service","auth":{"metadata":{"instance_id":"i-002","role":"service"}},"availability_zone":"eu-west-1b","class":"DriverController","common_booking_code":"BC002","common_client":"driver-app","common_client_ip":"192.168.1.2","common_driver_id":5002,"common_endpoint":"/api/v1/drivers","common_endpoint_elapsed":150,"common_error":"none","common_eta":"3min","common_http_url":"https://api.example.com/drivers","common_merchant_id":"MERCH002","common_order_id":"ORD002","common_passenger_id":1002,"common_provider":"grab","common_reference":"REF002","common_region":"SEA","common_request":"request_body","common_request_id":"REQ002","common_response":"response_body","common_service":"driver","common_trace_id":"TRACE002","common_user":"user002","common_vehicle_type":2,"event_type":"driver_assigned","gopoi_accuracy":15,"gopoi_keyword":"mall","gopoi_type":"dropoff","gopoi_xcountry":"SG","hostname":"host2","invalid_data":"none","jaya_rental_agreement_id":9002,"jaya_user":"jaya002","log_level":"INFO","log_write_epoch":1705315500,"message":"Driver assigned","orbital_location_id":7002,"pipeline_error":"none","request":{"headers":{"x-forwarded-for":"10.0.0.2"},"operation":"PUT","path":"/drivers"},"source":"application","stacktrace":"none","tags":"production","ucm_config_key":"feature.toggle","ucm_operation":"write","ucm_repository":"config-repo","ucm_running_env":"production","vpc_name":"vpc-prod"}
{"index":{"_index":"test-grab-index"}}
{"@timestamp":"2024-01-15T10:10:00","additional_data":"data3","app_name":"payment-service","auth":{"metadata":{"instance_id":"i-003","role":"service"}},"availability_zone":"eu-west-1c","class":"PaymentController","common_booking_code":"BC003","common_client":"web-app","common_client_ip":"192.168.1.3","common_driver_id":5003,"common_endpoint":"/api/v1/payments","common_endpoint_elapsed":200,"common_error":"none","common_eta":"1min","common_http_url":"https://api.example.com/payments","common_merchant_id":"MERCH003","common_order_id":"ORD003","common_passenger_id":1003,"common_provider":"grab","common_reference":"REF003","common_region":"SEA","common_request":"request_body","common_request_id":"REQ003","common_response":"response_body","common_service":"payment","common_trace_id":"TRACE003","common_user":"user003","common_vehicle_type":3,"event_type":"payment_processed","gopoi_accuracy":20,"gopoi_keyword":"hotel","gopoi_type":"destination","gopoi_xcountry":"SG","hostname":"host3","invalid_data":"none","jaya_rental_agreement_id":9003,"jaya_user":"jaya003","log_level":"INFO","log_write_epoch":1705315800,"message":"Payment processed","orbital_location_id":7003,"pipeline_error":"none","request":{"headers":{"x-forwarded-for":"10.0.0.3"},"operation":"POST","path":"/payments"},"source":"application","stacktrace":"none","tags":"production","ucm_config_key":"payment.enabled","ucm_operation":"read","ucm_repository":"config-repo","ucm_running_env":"production","vpc_name":"vpc-prod"}
{"index":{"_index":"test-grab-index"}}
{"@timestamp":"2024-01-15T10:15:00","additional_data":"data4","app_name":"location-service","auth":{"metadata":{"instance_id":"i-004","role":"service"}},"availability_zone":"eu-west-1a","class":"LocationController","common_booking_code":"BC004","common_client":"mobile-app","common_client_ip":"192.168.1.4","common_driver_id":5004,"common_endpoint":"/api/v1/locations","common_endpoint_elapsed":50,"common_error":"none","common_eta":"2min","common_http_url":"https://api.example.com/locations","common_merchant_id":"MERCH004","common_order_id":"ORD004","common_passenger_id":1004,"common_provider":"grab","common_reference":"REF004","common_region":"SEA","common_request":"request_body","common_request_id":"REQ004","common_response":"response_body","common_service":"location","common_trace_id":"TRACE004","common_user":"user004","common_vehicle_type":1,"event_type":"location_updated","gopoi_accuracy":12,"gopoi_keyword":"office","gopoi_type":"pickup","gopoi_xcountry":"SG","hostname":"host4","invalid_data":"none","jaya_rental_agreement_id":9004,"jaya_user":"jaya004","log_level":"DEBUG","log_write_epoch":1705316100,"message":"Location updated","orbital_location_id":7004,"pipeline_error":"none","request":{"headers":{"x-forwarded-for":"10.0.0.4"},"operation":"PATCH","path":"/locations"},"source":"application","stacktrace":"none","tags":"production","ucm_config_key":"location.tracking","ucm_operation":"write","ucm_repository":"config-repo","ucm_running_env":"production","vpc_name":"vpc-prod"}
{"index":{"_index":"test-grab-index"}}
{"@timestamp":"2024-01-15T10:20:00","additional_data":"data5","app_name":"notification-service","auth":{"metadata":{"instance_id":"i-005","role":"service"}},"availability_zone":"eu-west-1b","class":"NotificationController","common_booking_code":"BC005","common_client":"mobile-app","common_client_ip":"192.168.1.5","common_driver_id":5005,"common_endpoint":"/api/v1/notifications","common_endpoint_elapsed":75,"common_error":"none","common_eta":"4min","common_http_url":"https://api.example.com/notifications","common_merchant_id":"MERCH005","common_order_id":"ORD005","common_passenger_id":1005,"common_provider":"grab","common_reference":"REF005","common_region":"SEA","common_request":"request_body","common_request_id":"REQ005","common_response":"response_body","common_service":"notification","common_trace_id":"TRACE005","common_user":"user005","common_vehicle_type":2,"event_type":"notification_sent","gopoi_accuracy":18,"gopoi_keyword":"restaurant","gopoi_type":"dropoff","gopoi_xcountry":"SG","hostname":"host5","invalid_data":"none","jaya_rental_agreement_id":9005,"jaya_user":"jaya005","log_level":"INFO","log_write_epoch":1705316400,"message":"Notification sent","orbital_location_id":7005,"pipeline_error":"none","request":{"headers":{"x-forwarded-for":"10.0.0.5"},"operation":"POST","path":"/notifications"},"source":"application","stacktrace":"none","tags":"production","ucm_config_key":"notification.enabled","ucm_operation":"read","ucm_repository":"config-repo","ucm_running_env":"production","vpc_name":"vpc-prod"}
{"index":{"_index":"test-grab-index"}}
{"@timestamp":"2024-01-15T10:25:00","additional_data":"data6","app_name":"rating-service","auth":{"metadata":{"instance_id":"i-006","role":"service"}},"availability_zone":"eu-west-1c","class":"RatingController","common_booking_code":"BC006","common_client":"mobile-app","common_client_ip":"192.168.1.6","common_driver_id":5006,"common_endpoint":"/api/v1/ratings","common_endpoint_elapsed":120,"common_error":"none","common_eta":"6min","common_http_url":"https://api.example.com/ratings","common_merchant_id":"MERCH006","common_order_id":"ORD006","common_passenger_id":1006,"common_provider":"grab","common_reference":"REF006","common_region":"SEA","common_request":"request_body","common_request_id":"REQ006","common_response":"response_body","common_service":"rating","common_trace_id":"TRACE006","common_user":"user006","common_vehicle_type":3,"event_type":"rating_submitted","gopoi_accuracy":25,"gopoi_keyword":"station","gopoi_type":"pickup","gopoi_xcountry":"SG","hostname":"host6","invalid_data":"none","jaya_rental_agreement_id":9006,"jaya_user":"jaya006","log_level":"INFO","log_write_epoch":1705316700,"message":"Rating submitted","orbital_location_id":7006,"pipeline_error":"none","request":{"headers":{"x-forwarded-for":"10.0.0.6"},"operation":"POST","path":"/ratings"},"source":"application","stacktrace":"none","tags":"production","ucm_config_key":"rating.system","ucm_operation":"write","ucm_repository":"config-repo","ucm_running_env":"production","vpc_name":"vpc-prod"}
'

curl localhost:9200/test-grab-index/_refresh

curl -X POST "http://localhost:9200/_plugins/_ppl" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "source=test-grab-index | where common_error_type != \"\" | stats count() as c by common_error_type | sort - c | head 10"
  }'


### Description
[Describe what this change achieves]

### Related Issues
Resolves #[Issue number to be closed when this PR is merged]
<!-- List any other related issues here -->

### Check List
- [ ] Functionality includes testing.
- [ ] API changes companion pull request [created](https://github.com/opensearch-project/opensearch-api-specification/blob/main/DEVELOPER_GUIDE.md), if applicable.
- [ ] Public documentation issue/PR [created](https://github.com/opensearch-project/documentation-website/issues/new/choose), if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

coderabbitai · 2026-01-21T05:38:41Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-01-21T05:44:32Z

❌ Gradle check result for 8b77350: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2026-01-21T09:43:47Z

❌ Gradle check result for ee8e2f5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Co-authored-by: Aniket Modak <[email protected]>

animodak7 requested review from a team, Bukhtawar, CEHENKLE, Rishikesh1159, anasalkouz, andrross, ashking94, cwperks, dbwiddis, gbbafna, jed326, kotwanikunal, mch2, msfroh, owaiskazi19, reta, sachinpkale, saratvemulapalli, shwetathareja and sohami as code owners January 21, 2026 05:38

sandeshkr419 approved these changes Jan 21, 2026

View reviewed changes

handled empty results in terms aggs

ee8e2f5

animodak7 force-pushed the feature/datafusion branch from 8b77350 to ee8e2f5 Compare January 21, 2026 09:37

bharath-techie merged commit bede4c9 into opensearch-project:feature/datafusion Jan 21, 2026
7 of 31 checks passed

abhita pushed a commit to abhita/OpenSearch that referenced this pull request Jan 26, 2026

handled empty results in terms aggs (opensearch-project#20451)

1ff0d50

Co-authored-by: Aniket Modak <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

handled empty results in terms aggs #20451

handled empty results in terms aggs #20451

animodak7 commented Jan 21, 2026

Uh oh!

coderabbitai bot commented Jan 21, 2026

Review skipped

Uh oh!

github-actions bot commented Jan 21, 2026

Uh oh!

github-actions bot commented Jan 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

handled empty results in terms aggs #20451

handled empty results in terms aggs #20451

Conversation

animodak7 commented Jan 21, 2026

Uh oh!

coderabbitai bot commented Jan 21, 2026

Review skipped

Uh oh!

github-actions bot commented Jan 21, 2026

Uh oh!

github-actions bot commented Jan 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants