Upgrade to AWS SDK V2

[jtuglu1]

Closes #16903.

Description

Upgrade to AWS SDK v2.40.0. Looks like dependencies hdfs-storage + ranger have transitive dependencies on V1. Those will need to be excluded or upgraded as well. Core modules/extensions required for runtime are not reliant on V1.

Migrated to strictly synchronous APIs (to maintain backwards-compatibility with V1). A follow-up PR should be able to make use of async APIs where suitable. S3TransferManager is the only case where an async client may be used.

Hiccups

1. KinesisAggregatorV2 not available in Maven – Maven via JitPack: amazon-kinesis-aggregator-2.0.2 has v1 code, amazon-kinesis-deaggregator-2.0.2 unavailable awslabs/kinesis-aggregation#120. Doesn't look like anything uses aggregation=true, so added a warning message. Seems like [FLINK-32097][Connectors/Kinesis] Implement support for Kinesis deaggregation flink-connector-aws#188 also ran into similar issues, but implemented their own unmerged approach. Per the warning on the https://github.com/awslabs/kinesis-aggregation repo, looks like data loss can occur anyways when using this mode: https://github.com/awslabs/kinesis-aggregation/blob/master/potential_data_loss.md so don't think that's really a desired feature anyways.

Release note

Upgrade to AWS SDK v2.40.0

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[cryptoe]

Should we upgrade to the latest aws v2 sdk version ?

[jtuglu1]

Yes I will bump it to 2.40 minor release. Wanted to minimize chances of upstream bugs by using a stable version for testing.

[Fly-Style]

Very good job @jtuglu1! I have no objections, PR looks good from my perspective!

[jtuglu1]

👍 I'd still like to add 1-2 ITs that fully cover the newer S3 changes, but overall think it's basically there.

[cryptoe]

Due to the criticality of the change, I think we should get 2 approvals for this PR.
cc @jtuglu1

[cryptoe]

Left a partial review.
Wanted to ask the following questions:
Did we test this patch on any cluster with the following things with we have s3 as the deep storage.

• Kinesis Ingestion.
• MSQ with Durable storage.
• MSQ select with durable storage as the destination.
• Segment getting nuked out.
• Task logs being viewable and getting purged out on S3.
• Segment ingestion.

Also have you tried this patch in any of the production clusters ?

[cryptoe]

Who calls the refresh method now ?

[jtuglu1]

This refresh() is done now solely through a preexisting scheduled on the class. This is equivalent behavior as before, where FileSessionCredentialsProvider handles its own refresh internally on its scheduler. I don't believe the standalone method was used explicitly either; the v1 AWSCredentialsProviderChain didn't proactively call refresh() on providers during getCredentials() resolution — it just iterated through providers catching exceptions (from what I can tell).

[cryptoe]

Feels a bit weird that in the constructor we are calling these methods.
Should we call these in hasNext() ?

[jtuglu1]

I've moved out of ctor and into an initialize method. Iterators are expected to be single-threaded access, I'll make the initialized indicator thread-safe just in case.

[cryptoe]

Could you please share relevant docs for these placeholders ?

[jtuglu1]

Sure, I believe Gian added these a while back: #9098 (comment)

[jtuglu1]

Hi @cryptoe. Yes, I'm testing this on our production clusters running Kafka + S3. We don't have production clusters using Kinesis. I've only been able to run tests locally. I wonder if you folks might have some Kinesis clusters to test the new logic out on?

[cryptoe]

Hi @cryptoe. Yes, I'm testing this on our production clusters running Kafka + S3. We don't have production clusters using Kinesis. I've only been able to run tests locally.

Thanks for this. Do keep us posted about the findings.

I wonder if you folks might have some Kinesis clusters to test the new logic out on?

We also don't have active kinesis clusters but let me get back to you on this.

[jtuglu1]

Hi, update here:

I've left this internally on a few of our clusters and things are working fine. Confirmed I've tested a superset of the following:

#	Test	Status
1	Native Batch Ingestion (index_parallel)	DONE
2	Streaming Ingestion (index_kafka)	DONE
3	Streaming Ingestion (index_kinesis)	TODO
4	MSQ SELECT with S3 Durable Storage	DONE
5	MSQ INSERT with S3 Shuffle Storage	DONE
6	Segment Mark as Unused + Kill Task + Nuke Segments	DONE
7	Task Log Deletion from S3	DONE

I will see if I can get the kinesis tested, but otherwise things are looking good. I will address rest of comments and then should be ready to go.