CloudControl waiter generation should handle throttling error

[psantus]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

The generated resourceRequestSuccessStateRetryable function treats all OperationStatus: "FAILED" as terminal, but should retry when ErrorCode: "Throttling".

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Retry on throttling errors

Current Behavior

Doesn't retry for GetResourceRequestStatus.go‎

Reproduction Steps

Using Terraform AWSCC provider, trigger multiple resource creation / deletion at once.

Possible Solution

Fix needed in: Smithy model or waiter generation templates to add throttling-aware retry logic.

I believe resulting generated code should look like suggested fix

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2 v1.41.0

Compiler and Version used

go1.24.11 darwin/arm64

Operating System and version

MacOS

[lucix-aws]

The waiter is behaving correctly on spec. If we don't match any of the waiter's acceptors, and we hit an error, we transition to the failure state. That's step 4 of the linked waiter workflow.

Totally independently of the waiter's retry loop, he SDK should be swallowing throttling errors in the operation retry loop. Is that not the case here? That could be happening for any number of reasons:

1. the underlying client configured by Terraform has been altered to not retry
2. the client is retrying and we're just hitting too many throttling errors
3. the API signature of the throttling error is different in some way (we'd need to see what the response looks like, ideally with aws.LogResponseWithBody

[psantus]

Hi @lucix-aws,

I pushed log level to trace and I have this

http.response.body=

  | {"ProgressEvent":{"ErrorCode":"Throttling","EventTime":1.765450096027E9,"Operation":"CREATE","OperationStatus":"FAILED","RequestToken":"706b3d4e-5d20-4d25-afed-225dad80333d","StatusMessage":"Rate exceeded for operation 'CREATE'.","TypeName":"AWS::SageMaker::Image"}}
   http.response.header.date="Thu, 11 Dec 2025 10:48:20 GMT" rpc.method=GetResourceRequestStatus rpc.service=CloudControl @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.68/logging/tf_logger.go:45 http.response.header.x_amzn_requestid=741ab480-b962-42ac-92bb-8d34f385558b http.status_code=200 rpc.system=aws-api tf_req_id=be90bae1-13a4-bf43-c8cd-4c2973de2c48 tf_rpc=ApplyResourceChange http.response.header.content_type=application/x-amz-json-1.0 tf_aws.signing_region="" timestamp="2025-12-11T11:48:20.080+0100"
...
 

I can see it only once in the logs, suggesting there is no retry involved (or maybe I'm wrong and only the last, definitely failing call is logged??)

[psantus]

@lucix-aws I'm assuming you've had a look at e830742 ?

[lucix-aws]

Yes but per what I said before about expected waiter behavior that patch isn't right (or at a minimum it does not consider the whole picture here)

[lucix-aws]

So that API response is actually a 200 which I didn't expect. All of the stuff I was saying above about the SDK retry loop handling "throttles" applies to error-level status codes i.e. 4xx and up.

Since this is a 2xx-level error, i.e. it's something that the service apparently should be returning in normal behavior, and since they have this 2xx-level throttling error for this operation - it looks like the service has not modeled their waiter correctly to account for this. Basically they would need to add a new output-based "retry" acceptor to this waiter that matches on ProgressEvent.ErrorCode equal to Throttling. That's something I can upstream to them which I will try to do today.

[psantus]

Great. Indeed 200 + FAILURE is kinda disturbing ;)

[lucix-aws]

V2047161098