S3 GRIB2 dataset OpenDAP service failing

[mylesmc123]

I'm running THREDDS Data Server (TDS) 5.8 with datasets backed by an S3-compatible object store (CEPH). Catalog generation and browsing work correctly, all objects are visible in the TDS catalog, and downloadable via HTTP Service.

However, the OPeNDAP access fails with HTTP 500 errors:

Error { code = 500; message = "software.amazon.awssdk.services.s3.model.S3Exception: (Service: S3, Status Code: 403, Request ID: H7G8DREC04NQ36S9, Extended Request ID: F061xKMl5ueqxuA+Q2ftcauC6EymJ+5WGTSTf9XBmCWTsg0TnK3KMQ07kp8pYZh2UX84/h+W/+s=) (SDK Attempt Count: 1)"; };

My catalog is setup to use the following datasetScan using the cdms3 protocol:

     <datasetScan
        name="Monthly Means"
        ID="cfs_reanl_mm"
        path="monthly-means"
        location="cdms3://object.ncei.noaa.gov/datapub-prod-cfs-reanalysis?monthly-means/#delimiter=/">
        <metadata inherited="true">
          <serviceName>fileServices</serviceName>
        </metadata>
        <sort>
          <lexigraphicByName increasing="false" />
        </sort>
        <filter>
          <exclude wildcard="*.md5"/>
          <exclude wildcard="*.inv"/>
        </filter>
        <addDatasetSize />
      </datasetScan>

I have included in my log4j.xml some additonal loggers to help troubleshoot the issue:

<Logger name="ucar.unidata.io.s3" level="DEBUG" additivity="false">
      <AppenderRef ref="threddsServlet"/>
    </Logger>

    <Logger name="software.amazon.awssdk" level="DEBUG" additivity="false">
      <AppenderRef ref="threddsServlet"/>
    </Logger>

    <Logger name="thredds.server.opendap" level="DEBUG" additivity="false">
      <AppenderRef ref="threddsServlet"/>
    </Logger>

The threddsServlet.log then includes a lot of output when attempting to use the OpenDAP service with a relevant snippet being:

2026-01-26T12:34:35.181 -0500 [    182090][      21] INFO  - threddsServlet - Remote host: 127.0.0.1 - Request: "GET /thredds/catalog/monthly-means/201103/catalog.html?dataset=cfs_reanl_mm/201103/pgblnl.gdas.201103.grb2 HTTP/1.1"
2026-01-26T12:34:35.213 -0500 [    182122][      21] DEBUG - software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Creating an interceptor chain that will apply interceptors in the following order: [software.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@188faf5b, software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@3ac5302c, software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@11550dfd, software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@77f3cf02, software.amazon.awssdk.services.s3.internal.handlers.EnableChunkedEncodingInterceptor@126353d0, software.amazon.awssdk.services.s3.internal.handlers.DisableDoubleUrlEncodingInterceptor@13455e0, software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor@38b79ade, software.amazon.awssdk.services.s3.internal.handlers.CreateMultipartUploadRequestInterceptor@5155c915, software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor@5920ed3c, software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor@7a9838ad, software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInterceptor@7e7d2c4a, software.amazon.awssdk.services.s3.internal.handlers.EndpointAddressInterceptor@19f5ff1, software.amazon.awssdk.services.s3.internal.handlers.SignerOverrideInterceptor@5f080bf3, software.amazon.awssdk.services.s3.internal.handlers.GetBucketPolicyInterceptor@646e0def, software.amazon.awssdk.services.s3.internal.handlers.PutObjectInterceptor@73e66254, software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor@8fdcb0e, software.amazon.awssdk.services.s3.internal.handlers.DecodeUrlEncodedResponseInterceptor@17b51e71, software.amazon.awssdk.services.s3.internal.handlers.CreateBucketInterceptor@3a0781d, software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor@43945eff]
2026-01-26T12:34:35.214 -0500 [    182123][      21] DEBUG - software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.services.s3.internal.handlers.EndpointAddressInterceptor@19f5ff1' modified the message with its modifyHttpRequest method.
2026-01-26T12:34:35.214 -0500 [    182123][      21] DEBUG - software.amazon.awssdk.retries.LegacyRetryStrategy - Request attempt 1 token acquired (backoff: 0ms, cost: 0, capacity: 500/500)
2026-01-26T12:34:35.214 -0500 [    182123][      21] DEBUG - software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=GET, protocol=https, host=object.ncei.noaa.gov, port=443, encodedPath=/datapub-prod-cfs-reanalysis, headers=[amz-sdk-invocation-id, User-Agent], queryParameters=[list-type, prefix])
2026-01-26T12:34:35.214 -0500 [    182123][      21] DEBUG - software.amazon.awssdk.auth.signer.Aws4Signer - AWS4 Canonical Request: GET
/datapub-prod-cfs-reanalysis
list-type=2&prefix=monthly-means%2F201103%2F
amz-sdk-invocation-id:38dc0d18-9b98-d8d7-3ed9-a911c5971471
amz-sdk-request:attempt=1; max=4
host:object.ncei.noaa.gov
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20260126T173435Z

amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
2026-01-26T12:34:35.214 -0500 [    182123][      21] DEBUG - software.amazon.awssdk.auth.signer.Aws4Signer - AWS4 String to sign: AWS4-HMAC-SHA256
20260126T173435Z
20260126/us-east-1/s3/aws4_request
b3728c1e2aae070dea78592dd0b063082fbd5799b7a207628fee80a739457f44
2026-01-26T12:34:35.216 -0500 [    182125][      21] DEBUG - software.amazon.awssdk.http.apache.internal.conn.SdkTlsSocketFactory - Connecting socket to object.ncei.noaa.gov/10.254.11.44:443 with timeout 10000
2026-01-26T12:34:35.218 -0500 [    182127][      21] DEBUG - software.amazon.awssdk.http.apache.internal.conn.SdkTlsSocketFactory - Enabled protocols: [TLSv1.3, TLSv1.2]
2026-01-26T12:34:35.218 -0500 [    182127][      21] DEBUG - software.amazon.awssdk.http.apache.internal.conn.SdkTlsSocketFactory - Enabled cipher suites:[TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2026-01-26T12:34:35.218 -0500 [    182127][      21] DEBUG - software.amazon.awssdk.http.apache.internal.conn.SdkTlsSocketFactory - socket.getSupportedProtocols(): [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1, SSLv3, SSLv2Hello], socket.getEnabledProtocols(): [TLSv1.3, TLSv1.2]
2026-01-26T12:34:35.218 -0500 [    182127][      21] DEBUG - software.amazon.awssdk.http.apache.internal.conn.SdkTlsSocketFactory - Starting handshake
2026-01-26T12:34:35.230 -0500 [    182139][      21] DEBUG - software.amazon.awssdk.http.apache.internal.conn.SdkTlsSocketFactory - Secure session established
2026-01-26T12:34:35.230 -0500 [    182139][      21] DEBUG - software.amazon.awssdk.http.apache.internal.conn.SdkTlsSocketFactory -  negotiated protocol: TLSv1.3
2026-01-26T12:34:35.230 -0500 [    182139][      21] DEBUG - software.amazon.awssdk.http.apache.internal.conn.SdkTlsSocketFactory -  negotiated cipher suite: TLS_AES_128_GCM_SHA256
2026-01-26T12:34:35.230 -0500 [    182139][      21] DEBUG - software.amazon.awssdk.http.apache.internal.conn.SdkTlsSocketFactory -  peer principal: CN=object.ncei.noaa.gov, O=NCEI.NOAA.GOV
2026-01-26T12:34:35.230 -0500 [    182139][      21] DEBUG - software.amazon.awssdk.http.apache.internal.net.SdkSslSocket - created: object.ncei.noaa.gov/10.254.11.44:443
2026-01-26T12:34:35.255 -0500 [    182164][      21] DEBUG - software.amazon.awssdk.requestId - Received successful response: 200, Request ID: tx00000b5ccff2af3d78b19-006977a5ab-2113c3e-ncei-nc-1, Extended Request ID: not available
2026-01-26T12:34:35.255 -0500 [    182164][      21] DEBUG - software.amazon.awssdk.request - Received successful response: 200, Request ID: tx00000b5ccff2af3d78b19-006977a5ab-2113c3e-ncei-nc-1, Extended Request ID: not available
2026-01-26T12:34:35.272 -0500 [    182181][      21] DEBUG - software.amazon.awssdk.retries.LegacyRetryStrategy - Request attempt 1 succeeded (cost: -1, capacity: 500/500)
2026-01-26T12:34:35.273 -0500 [    182182][      21] DEBUG - software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Creating an interceptor chain that will apply interceptors in the following order: [software.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@188faf5b, software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@3ac5302c, software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@11550dfd, software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@77f3cf02, software.amazon.awssdk.services.s3.internal.handlers.EnableChunkedEncodingInterceptor@126353d0, software.amazon.awssdk.services.s3.internal.handlers.DisableDoubleUrlEncodingInterceptor@13455e0, software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor@38b79ade, software.amazon.awssdk.services.s3.internal.handlers.CreateMultipartUploadRequestInterceptor@5155c915, software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor@5920ed3c, software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor@7a9838ad, software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInterceptor@7e7d2c4a, software.amazon.awssdk.services.s3.internal.handlers.EndpointAddressInterceptor@19f5ff1, software.amazon.awssdk.services.s3.internal.handlers.SignerOverrideInterceptor@5f080bf3, software.amazon.awssdk.services.s3.internal.handlers.GetBucketPolicyInterceptor@646e0def, software.amazon.awssdk.services.s3.internal.handlers.PutObjectInterceptor@73e66254, software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor@8fdcb0e, software.amazon.awssdk.services.s3.internal.handlers.DecodeUrlEncodedResponseInterceptor@17b51e71, software.amazon.awssdk.services.s3.internal.handlers.CreateBucketInterceptor@3a0781d, software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor@43945eff]
2026-01-26T12:34:35.274 -0500 [    182183][      21] DEBUG - software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.services.s3.internal.handlers.EndpointAddressInterceptor@19f5ff1' modified the message with its modifyHttpRequest method.
2026-01-26T12:34:35.275 -0500 [    182184][      21] DEBUG - software.amazon.awssdk.retries.LegacyRetryStrategy - Request attempt 1 token acquired (backoff: 0ms, cost: 0, capacity: 500/500)
2026-01-26T12:34:35.275 -0500 [    182184][      21] DEBUG - software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=GET, protocol=https, host=object.ncei.noaa.gov, port=443, encodedPath=/datapub-prod-cfs-reanalysis, headers=[amz-sdk-invocation-id, User-Agent], queryParameters=[delimiter, prefix])
2026-01-26T12:34:35.275 -0500 [    182184][      21] DEBUG - software.amazon.awssdk.auth.signer.Aws4Signer - AWS4 Canonical Request: GET
/datapub-prod-cfs-reanalysis
delimiter=%2F&prefix=monthly-means%2F201103%2F
amz-sdk-invocation-id:905f516b-2de9-a45d-b42e-f855e285937f
amz-sdk-request:attempt=1; max=4
host:object.ncei.noaa.gov
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20260126T173435Z

amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
2026-01-26T12:34:35.275 -0500 [    182184][      21] DEBUG - software.amazon.awssdk.auth.signer.Aws4Signer - AWS4 String to sign: AWS4-HMAC-SHA256
20260126T173435Z
20260126/us-east-1/s3/aws4_request
eb201b466a8a7cb7054a031c4a22fe55f69a0f4f7df6d4981c437d852d236ca2
2026-01-26T12:34:35.304 -0500 [    182213][      21] DEBUG - software.amazon.awssdk.requestId - Received successful response: 200, Request ID: tx000001f3d29ec5422858e-006977a5ab-2113c53-ncei-nc-1, Extended Request ID: not available
2026-01-26T12:34:35.304 -0500 [    182213][      21] DEBUG - software.amazon.awssdk.request - Received successful response: 200, Request ID: tx000001f3d29ec5422858e-006977a5ab-2113c53-ncei-nc-1, Extended Request ID: not available
2026-01-26T12:34:35.315 -0500 [    182224][      21] DEBUG - software.amazon.awssdk.retries.LegacyRetryStrategy - Request attempt 1 succeeded (cost: -1, capacity: 500/500)
2026-01-26T12:34:35.332 -0500 [    182241][      21] DEBUG - software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Creating an interceptor chain that will apply interceptors in the following order: [software.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@188faf5b, software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@3ac5302c, software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@11550dfd, software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@77f3cf02, software.amazon.awssdk.services.s3.internal.handlers.EnableChunkedEncodingInterceptor@126353d0, software.amazon.awssdk.services.s3.internal.handlers.DisableDoubleUrlEncodingInterceptor@13455e0, software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor@38b79ade, software.amazon.awssdk.services.s3.internal.handlers.CreateMultipartUploadRequestInterceptor@5155c915, software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor@5920ed3c, software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor@7a9838ad, software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInterceptor@7e7d2c4a, software.amazon.awssdk.services.s3.internal.handlers.EndpointAddressInterceptor@19f5ff1, software.amazon.awssdk.services.s3.internal.handlers.SignerOverrideInterceptor@5f080bf3, software.amazon.awssdk.services.s3.internal.handlers.GetBucketPolicyInterceptor@646e0def, software.amazon.awssdk.services.s3.internal.handlers.PutObjectInterceptor@73e66254, software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor@8fdcb0e, software.amazon.awssdk.services.s3.internal.handlers.DecodeUrlEncodedResponseInterceptor@17b51e71, software.amazon.awssdk.services.s3.internal.handlers.CreateBucketInterceptor@3a0781d, software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor@43945eff]
2026-01-26T12:34:35.333 -0500 [    182242][      21] DEBUG - software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.services.s3.internal.handlers.EndpointAddressInterceptor@19f5ff1' modified the message with its modifyHttpRequest method.
2026-01-26T12:34:35.333 -0500 [    182242][      21] DEBUG - software.amazon.awssdk.retries.LegacyRetryStrategy - Request attempt 1 token acquired (backoff: 0ms, cost: 0, capacity: 500/500)
2026-01-26T12:34:35.333 -0500 [    182242][      21] DEBUG - software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=GET, protocol=https, host=object.ncei.noaa.gov, port=443, encodedPath=/datapub-prod-cfs-reanalysis, headers=[amz-sdk-invocation-id, User-Agent], queryParameters=[delimiter, prefix])
2026-01-26T12:34:35.334 -0500 [    182243][      21] DEBUG - software.amazon.awssdk.auth.signer.Aws4Signer - AWS4 Canonical Request: GET
/datapub-prod-cfs-reanalysis
delimiter=%2F&prefix=monthly-means%2F201103%2F
amz-sdk-invocation-id:3f624c42-8908-e123-b01a-e41ff810c859
amz-sdk-request:attempt=1; max=4
host:object.ncei.noaa.gov
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20260126T173435Z

amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
2026-01-26T12:34:35.334 -0500 [    182243][      21] DEBUG - software.amazon.awssdk.auth.signer.Aws4Signer - AWS4 String to sign: AWS4-HMAC-SHA256
20260126T173435Z
20260126/us-east-1/s3/aws4_request
a42c263f12a24cf439862546b01a655eecbd2c979fbaffd17671864b92d23476
2026-01-26T12:34:35.373 -0500 [    182282][      21] DEBUG - software.amazon.awssdk.requestId - Received successful response: 200, Request ID: tx000001b26a29c1e61743d-006977a5ab-2113c5f-ncei-nc-1, Extended Request ID: not available
2026-01-26T12:34:35.373 -0500 [    182282][      21] DEBUG - software.amazon.awssdk.request - Received successful response: 200, Request ID: tx000001b26a29c1e61743d-006977a5ab-2113c5f-ncei-nc-1, Extended Request ID: not available
2026-01-26T12:34:35.383 -0500 [    182292][      21] DEBUG - software.amazon.awssdk.retries.LegacyRetryStrategy - Request attempt 1 succeeded (cost: -1, capacity: 500/500)
2026-01-26T12:34:35.384 -0500 [    182293][      21] DEBUG - software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Creating an interceptor chain that will apply interceptors in the following order: [software.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@188faf5b, software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@3ac5302c, software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@11550dfd, software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@77f3cf02, software.amazon.awssdk.services.s3.internal.handlers.EnableChunkedEncodingInterceptor@126353d0, software.amazon.awssdk.services.s3.internal.handlers.DisableDoubleUrlEncodingInterceptor@13455e0, software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor@38b79ade, software.amazon.awssdk.services.s3.internal.handlers.CreateMultipartUploadRequestInterceptor@5155c915, software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor@5920ed3c, software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor@7a9838ad, software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInterceptor@7e7d2c4a, software.amazon.awssdk.services.s3.internal.handlers.EndpointAddressInterceptor@19f5ff1, software.amazon.awssdk.services.s3.internal.handlers.SignerOverrideInterceptor@5f080bf3, software.amazon.awssdk.services.s3.internal.handlers.GetBucketPolicyInterceptor@646e0def, software.amazon.awssdk.services.s3.internal.handlers.PutObjectInterceptor@73e66254, software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor@8fdcb0e, software.amazon.awssdk.services.s3.internal.handlers.DecodeUrlEncodedResponseInterceptor@17b51e71, software.amazon.awssdk.services.s3.internal.handlers.CreateBucketInterceptor@3a0781d, software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor@43945eff]
2026-01-26T12:34:35.385 -0500 [    182294][      21] DEBUG - software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.services.s3.internal.handlers.EndpointAddressInterceptor@19f5ff1' modified the message with its modifyHttpRequest method.
2026-01-26T12:34:35.385 -0500 [    182294][      21] DEBUG - software.amazon.awssdk.retries.LegacyRetryStrategy - Request attempt 1 token acquired (backoff: 0ms, cost: 0, capacity: 500/500)
2026-01-26T12:34:35.385 -0500 [    182294][      21] DEBUG - software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=HEAD, protocol=https, host=object.ncei.noaa.gov, port=443, encodedPath=/datapub-prod-cfs-reanalysis/monthly-means/201103/.zarray, headers=[amz-sdk-invocation-id, User-Agent], queryParameters=[])
2026-01-26T12:34:35.386 -0500 [    182295][      21] DEBUG - software.amazon.awssdk.auth.signer.Aws4Signer - AWS4 Canonical Request: HEAD
/datapub-prod-cfs-reanalysis/monthly-means/201103/.zarray

amz-sdk-invocation-id:0e885f48-1124-6504-2270-b1da9fbe30e9
amz-sdk-request:attempt=1; max=4
host:object.ncei.noaa.gov
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20260126T173435Z

amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
2026-01-26T12:34:35.386 -0500 [    182295][      21] DEBUG - software.amazon.awssdk.auth.signer.Aws4Signer - AWS4 String to sign: AWS4-HMAC-SHA256
20260126T173435Z
20260126/us-east-1/s3/aws4_request
0051e2b532f9bc37df6ce8bd4eb9002969b8e34b5a1acb871af9fd2846ee6445
2026-01-26T12:34:35.389 -0500 [    182298][      21] DEBUG - software.amazon.awssdk.requestId - Received failed response: 404, Request ID: tx000001151e804eb67f3dc-006977a5ab-2113c02-ncei-nc-1, Extended Request ID: not available
2026-01-26T12:34:35.389 -0500 [    182298][      21] DEBUG - software.amazon.awssdk.request - Received failed response: 404, Request ID: tx000001151e804eb67f3dc-006977a5ab-2113c02-ncei-nc-1, Extended Request ID: not available
2026-01-26T12:34:35.389 -0500 [    182298][      21] DEBUG - software.amazon.awssdk.retries.LegacyRetryStrategy - Request attempt 1 encountered non-retryable failure
software.amazon.awssdk.services.s3.model.S3Exception: (Service: S3, Status Code: 404, Request ID: tx000001151e804eb67f3dc-006977a5ab-2113c02-ncei-nc-1)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:155)
2026-01-26T12:34:35.390 -0500 [    182299][      21] DEBUG - software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor@7a9838ad' modified the message with its modifyException method.
2026-01-26T12:34:35.390 -0500 [    182299][      21] DEBUG - software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Creating an interceptor chain that will apply interceptors in the following order: [software.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@188faf5b, software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@3ac5302c, software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@11550dfd, software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@77f3cf02, software.amazon.awssdk.services.s3.internal.handlers.EnableChunkedEncodingInterceptor@126353d0, software.amazon.awssdk.services.s3.internal.handlers.DisableDoubleUrlEncodingInterceptor@13455e0, software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor@38b79ade, software.amazon.awssdk.services.s3.internal.handlers.CreateMultipartUploadRequestInterceptor@5155c915, software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor@5920ed3c, software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor@7a9838ad, software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInterceptor@7e7d2c4a, software.amazon.awssdk.services.s3.internal.handlers.EndpointAddressInterceptor@19f5ff1, software.amazon.awssdk.services.s3.internal.handlers.SignerOverrideInterceptor@5f080bf3, software.amazon.awssdk.services.s3.internal.handlers.GetBucketPolicyInterceptor@646e0def, software.amazon.awssdk.services.s3.internal.handlers.PutObjectInterceptor@73e66254, software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor@8fdcb0e, software.amazon.awssdk.services.s3.internal.handlers.DecodeUrlEncodedResponseInterceptor@17b51e71, software.amazon.awssdk.services.s3.internal.handlers.CreateBucketInterceptor@3a0781d, software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor@43945eff]
2026-01-26T12:34:35.391 -0500 [    182300][      21] DEBUG - software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.services.s3.internal.handlers.EndpointAddressInterceptor@19f5ff1' modified the message with its modifyHttpRequest method.
2026-01-26T12:34:35.391 -0500 [    182300][      21] DEBUG - software.amazon.awssdk.retries.LegacyRetryStrategy - Request attempt 1 token acquired (backoff: 0ms, cost: 0, capacity: 500/500)
2026-01-26T12:34:35.391 -0500 [    182300][      21] DEBUG - software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=HEAD, protocol=https, host=object.ncei.noaa.gov, port=443, encodedPath=/datapub-prod-cfs-reanalysis/monthly-means/201103/.zgroup, headers=[amz-sdk-invocation-id, User-Agent], queryParameters=[])
2026-01-26T12:34:35.391 -0500 [    182300][      21] DEBUG - software.amazon.awssdk.auth.signer.Aws4Signer - AWS4 Canonical Request: HEAD
/datapub-prod-cfs-reanalysis/monthly-means/201103/.zgroup

amz-sdk-invocation-id:0152edf4-a128-5390-c27a-01e267d69e04
amz-sdk-request:attempt=1; max=4
host:object.ncei.noaa.gov
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20260126T173435Z

What I find odd about this log snippet is that it is referencing .zarray and .zgroup files even though this dataset is purely GRIB2 data. Any help is appreciated in further troubleshooting this issue.

[lesserwhirls]

So a couple of things. In general, we do not support serving GRIB using datasetScan. That said, it's probably the only way for the TDS to access GRIB from an object store currently, as the featureCollection does not work with object storage quite yet (it is on the to-do list, to be sure). However, in order for netCDF-Java to work with a GRIB file, it has to create gbx9 and ncx4 index files somewhere (or they need to already exist). If the index files do not exist, then by default, the TDS will likely try to create them in the same location as the grib file itself, which in this case would be using the same prefix on the object store. As far as I am aware, that does not work at this point, even if your access key allows the TDS to create objects in the host bucket (and you might not want that to happen silently by default). There is a way to force the TDS to create the index files locally, which you may want to try. You would need to edit threddsConfig.xml to use:

<GribIndex>
  <neverUse>false</neverUse>
  <alwaysUse>true</alwaysUse>
  <dir>/custom/path</dir>
</GribIndex>

where /custom/path is a local directory within which the index files can be generated. While I have not tried this myself, based on older netCDF-Java issues like this one, it should allow this to work.

The messages you see about .zarray and .zgroup is likely from datasetScan doing checks to see out if the location is a zarr store so that it can create an accessible dataset while still being able to generate a navigable catalog structure further within the zarr key structure (ongoing work at this point).

[mylesmc123]

@lesserwhirls , Hey thanks for the quick response. I am using a local directory for the cache in my threddsConfig.xml:

<GribIndex>
    <neverUse>false</neverUse>
    <alwaysUse>true</alwaysUse>
    <dir>/var/lib/thredds/cache/GribIndex</dir>
    <policy>nestedDirectory</policy>
</GribIndex>

However I do notice that the temp gbx9 files in the local cache directory, that are created when I attempt to access the OpenDAP service for a GRIB file, have filename patterns that include the #delimeter string from the catalog datasetScan location. For example, the cache files in the local directory /var/lib/thredds/cache/GribIndex/cdms3/object.ncei.noaa.gov/datapub-prod-cfs-reanalysismonthly-means/201103 include file that look like:

ipvl05.gdas.201103.18Z.grb2#delimiter.gbx9.tmp
ipvl05.gdas.201103.grb2#delimiter.gbx9.tmp
pgblnl.gdas.201103.grb2#delimiter.gbx9.tmp

I think its picking up the #delimeter from the catalog entries location attribute:

<datasetScan
        name="Monthly Means"
        ID="cfs_reanl_mm"
        path="monthly-means"
        location="cdms3://object.ncei.noaa.gov/datapub-prod-cfs-reanalysis?monthly-means/#delimiter=/">
        <metadata inherited="true">
          <serviceName>fileServices</serviceName>
        </metadata>
        <sort>
          <lexigraphicByName increasing="false" />
        </sort>
        <filter>
          <exclude wildcard="*.md5"/>
          <exclude wildcard="*.inv"/>
        </filter>
        <addDatasetSize />
</datasetScan>

[lesserwhirls]

What are the sizes of the .tmp files that are created?

[mylesmc123]

tiny, 11 bytes. Whereas the GRIB files are around 17Mb each.

[lesserwhirls]

I'd expect them to be ~100 KiB. This is definitely a bug that happens early on in the index file generation, so I'll need to take a look locally to see what I can find. I'm guessing object.ncei.noaa.gov is internal-only access?

[mylesmc123]

Yes, unfortunately that's correct it's an internal only CEPH store for now, it will made public at some point. However, this particular GRIB data is downloadable via the httpServer service in THREDDS here for example: https://www.ncei.noaa.gov/thredds/catalog/model-cfs_reanl_mm/201103/catalog.html

[lesserwhirls]

It will take me a bit to get to it, but I will see what I can do about reproducing a small collection for local testing.