Doris 3.0.7 disaggregate cluster BE node encountered AWSXmlClient404 error

[whg517]

env：

• doris-operator deploy disaggregatecluster
• doris version 3.0.7
• minio as storage vault

the be pod log：

│ W20260326 14:25:16.643488 285 aws_logger.h:53] [AWSClient] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
│ E20260326 14:25:18.835106 285 aws_logger.h:50] [AWSXmlClient] HTTP response code: 404
│ Resolved remote host IP address: 10.16.2.146
│ Request ID:
│ Exception name:
│ Error message: No response body.
│ 14 response headers:
│ accept-ranges : bytes
│ content-length : 0
│ date : Thu, 26 Mar 2026 14:25:18 GMT
│ server : MinIO
│ strict-transport-security : max-age=31536000; includeSubDomains
│ vary : Origin
│ x-amz-id-2 : 6eca0391ce9176c9fc0b26ef30b9ff589a8ac06ef349b5109438331097525715
│ x-amz-request-id : 18A06A49E782FD18
│ x-content-type-options : nosniff
│ x-minio-error-code : NoSuchKey
│ x-minio-error-desc : "The specified key does not exist."
│ x-ratelimit-limit : 2490
│ x-ratelimit-remaining : 2490
│ x-xss-protection : 1; mode=block
│ W20260326 14:25:18.835165 285 aws_logger.h:53] [AWSClient] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
│ E20260326 14:25:19.760339 285 aws_logger.h:50] [AWSXmlClient] HTTP response code: 404
│ Resolved remote host IP address: 10.16.2.146
│ Request ID:
│ Exception name:
│ Error message: No response body.
│ 14 response headers:
│ accept-ranges : bytes
│ content-length : 0
│ date : Thu, 26 Mar 2026 14:25:19 GMT
│ server : MinIO
│ strict-transport-security : max-age=31536000; includeSubDomains
│ vary : Origin
│ x-amz-id-2 : 6eca0391ce9176c9fc0b26ef30b9ff589a8ac06ef349b5109438331097525715
│ x-amz-request-id : 18A06A4A1EA0906F
│ x-content-type-options : nosniff
│ x-minio-error-code : NoSuchKey
│ x-minio-error-desc : "The specified key does not exist."
│ x-ratelimit-limit : 2490
│ x-ratelimit-remaining : 2490
│ x-xss-protection : 1; mode=block
│ W20260326 14:25:19.760408 285 aws_logger.h:53] [AWSClient] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.

[Nickalus12]

To resolve the AWSXmlClient404 (NoSuchKey) error when using MinIO with Doris 3.0.7, ensure your StorageVault configuration explicitly enables path-style access and that your system clocks are synchronized.

Recommended Storage Vault Configuration

If you are configuring the vault via SQL, use the following template. Note the s3.use_path_style property, which is mandatory for most MinIO deployments:

CREATE STORAGE VAULT IF NOT EXISTS minio_vault
PROPERTIES (
    "type" = "S3",
    "s3.endpoint" = "http://minio-internal-service:9000",
    "s3.region" = "us-east-1",
    "s3.access_key" = "your_access_key",
    "s3.secret_key" = "your_secret_key",
    "s3.root_path" = "doris_data",
    "s3.bucket" = "my-bucket",
    "s3.use_path_style" = "true" -- Required for MinIO
);

-- Set as default vault for the disaggregated cluster
SET minio_vault AS DEFAULT STORAGE VAULT;

Why this error occurs

The NoSuchKey error (404) in a disaggregated architecture typically stems from three root causes:

1. Virtual-Hosted vs. Path-Style Access: By default, S3 clients attempt to connect via bucket.host. MinIO usually requires path-style access (host/bucket). If s3.use_path_style is false, the client may construct an incorrect URL, leading the MinIO server to return a 404 because it cannot find the object in the "default" or misinterpreted bucket path.
2. Time Skew: Your logs explicitly mention: This could be because of a time skew. AWS/S3 signatures are valid only within a small time window (usually 5–15 minutes). If the clock on your BE pods differs significantly from the MinIO server, the signature check fails. While this often results in a 403, it can trigger retries and inconsistent error reporting (like 404) depending on how the SDK handles the fallback.
3. Meta Service Inconsistency: In a disaggregated cluster, the Meta Service (backed by FoundationDB) maintains the mapping of data to S3 objects. If the bucket was manually cleaned or if the s3.root_path was changed after data was already written, the BE will attempt to fetch keys that no longer exist in MinIO.

Important

Verify that the Kubernetes nodes hosting your BE pods and your MinIO instance are all synchronized via NTP. A discrepancy of more than a few minutes will cause the AWSClient to fail signature adjustments.

Troubleshooting Steps

• Check Connectivity: Run mc ls (MinIO Client) from inside a BE pod to verify it can see the objects at the s3.root_path.
• Verify Permissions: Ensure the access_key has s3:GetBucketLocation, s3:ListBucket, s3:GetObject, and s3:PutObject permissions.
• Check Operator CR: If using the DorisDisaggregatedCluster CRD, ensure the objectStore section includes usePathStyle: true.

Are you seeing this error during initial cluster startup, or does it happen during specific query operations?