aws s3 v2 is adding the header content-encoding twice and it breaks some signature verifications

[peczenyj]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

Hello

By using the aws s3 v2 sdk latest version (v1.79.4) we note an issue when we try to use another service (yandex s3). We follow the documentation and we set the region and base endpoint and we can upload files to yandex s3 with success EXCEPT if we try to set the content encoding of the object.

For instance, if we try to put an object and specify the content encoding as "gzip", in the end it generates an http request like this

SDK 2025/12/19 11:31:14 DEBUG Request
PUT /idsync/0/FR/2025121811/windtrap-idsync-preprod-2766057041572339330.jsonl.gz?x-id=PutObject HTTP/1.1
Host: dx-to-bigsea-preprod-ru.storage.yandexcloud.net
User-Agent: aws-sdk-go-v2/1.36.3 ua/2.1 os/linux lang/go#1.25.5 md/GOOS#linux md/GOARCH#amd64 api/s3#1.79.4 m/E,Z,g
Content-Length: 179
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 9bdaa2d2-d8e9-493d-8fdb-627b4ec155ab
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=YCAJEcTiAlLnai6gQmjJYqJy8/20251219/ru-central1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-trailer, Signature=ced777c4703d01b9b9fd68277f5c33617fb2523921766a111bc197a4d93e3437
Content-Encoding: gzip
Content-Encoding: aws-chunked
Content-Type: application/jsonl+json
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20251219T103114Z
X-Amz-Decoded-Content-Length: 137
X-Amz-Trailer: x-amz-checksum-crc32

as you can see, we have twice the header Content-Encoding

Content-Encoding: gzip
Content-Encoding: aws-chunked

However, this is not correct. We expect the following header

Content-Encoding: gzip, aws-chunked

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The upload to yandex s3 works with success on

github.com/aws/aws-sdk-go-v2/service/s3 v1.58.3

if we dump the request, we see only one header

Current Behavior

yandex s3 was not able to calculate the right signature and return the following response

client put object error: operation error S3: PutObject, https response error StatusCode: 403, RequestID: b024ff32a48a7f0c, HostID: , api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.

Reproduction Steps

This program can trigger the issue

package main

import (
	"context"
	"flag"
	"log"
	"os"
	"path/filepath"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
	// Getting the bucket name from the command line argument
	bucketName := flag.String("b", "", "The name of the bucket")
	endpoint := flag.String("e", "https://storage.yandexcloud.net", "yandex aws s3 endpoint")
	region := flag.String("r", "ru-central1", "yandex aws s3 region")
	fileName := flag.String("f", "", "file to upload")
	objectPath := flag.String("p", "path/to/object", "object path")
	contentEncoding := flag.String("content-encoding", "", "if present, it specify the content encoding")

	flag.Parse()

	if *bucketName == "" {
		log.Fatal("You must supply the name of a bucket (-b BUCKET)")
	}

	if *fileName == "" {
		log.Fatal("You must supply the name of a file (-f FILENAME)")
	}

	ctx := context.Background()

	cfg, err := config.LoadDefaultConfig(ctx,
		config.WithClientLogMode(aws.LogRequestWithBody|aws.LogResponseWithBody),
		func(lo *config.LoadOptions) error {
			lo.BaseEndpoint = *endpoint
			lo.Region = *region

			return nil
		})
	if err != nil {
		log.Fatalf("config.LoadDefaultConfig(ctx): %v", err)
	}

	// cfg.Region = *region
	// cfg.BaseEndpoint = endpoint

	// Creating a client to access the S3 storage
	client := s3.NewFromConfig(cfg, func(o *s3.Options) {
		o.Region = *region
		o.BaseEndpoint = endpoint
	})

	file, err := os.Open(*fileName)
	if err != nil {
		log.Fatalf(" os.Open(%v): %v", *fileName, err)
	}

	defer file.Close()

	objectKey := filepath.Join(*objectPath, "filename.json.gz")

	output, err := client.PutObject(ctx, &s3.PutObjectInput{
		Bucket:          bucketName,
		Key:             aws.String(objectKey),
		Body:            file,
		ContentType:     aws.String("application/jsonl+json"),
		ContentEncoding: contentEncoding,
	})
	if err != nil {
		log.Printf("client put object error: %v", err)

		return
	}
	log.Printf("upload success %+v", output)

	foo, err := client.GetObject(ctx, &s3.GetObjectInput{
		Bucket: bucketName,
		Key:    aws.String(objectKey),
	})
	if err != nil {
		log.Printf("client get object error: %v", err)

		return
	}

	log.Printf("all %+v (content type %v, content encoding %v, metadata %+v)",
		foo, *foo.ContentType, foo.ContentEncoding, foo.Metadata)
}

$ go run main.go -f filename.json.gz -content-encoding "gzip"

Possible Solution

I have an workaround that is not set the content encoding or using and old version that I know that works properly.

A better solution is sending a single Content-Encoding header like it is specified on

https://httpwg.org/specs/rfc9110.html#field.content-encoding

Additional Information/Context

I did not test with minio

I found this issue by using redpanda-data/connect

redpanda-data/connect#3864 (comment)

AWS Go SDK V2 Module Versions Used

I used the following versions

github.com/aws/aws-sdk-go-v2 v1.36.3
github.com/aws/aws-sdk-go-v2/config v1.29.14
github.com/aws/aws-sdk-go-v2/service/s3 v1.79.4

Compiler and Version used

go version go1.25.5 linux/amd64

Operating System and version

Linux Ubuntu 24.04.3 LTS

[Madrigal]

Looked into this

as you can see, we have twice the header Content-Encoding

However, this is not correct. We expect the following header

This is not an SDK-specific behavior but something that Go does. You can check how an HTTP request gets dumped as a string here

https://go.dev/play/p/_oqIwfo5stk

The header is sent down the wire as separate values rather than being combined as a CSV, which is valid behavior as mentioned by the RFC

5.3. Field Order

A recipient MAY combine multiple field lines within a field section that have the same field name into one field line, without changing the semantics of the message, by appending each subsequent field line value to the initial field line value in order, separated by a comma (",") and optional whitespace

This has been discussed in other places, like on this thread or this other proposal to normalize headers golang/go#62471

It's in theory possible for the SDK to massage headers before sending them down the wire and combine multiple values, but implementing custom behavior for sending headers deviating from go standard library is likely going to bring us headaches in the future, so I'd be inclined to not change it