Amazon Connect: update_test_case with Status=PUBLISHED fails with InvalidTestCaseException but GUI succeeds

[alokkulkarni]

Describe the bug

GitHub Issue for boto3

Title

Amazon Connect: update_test_case with Status=PUBLISHED fails with InvalidTestCaseException but GUI succeeds

Description

Summary

When attempting to publish Amazon Connect test cases using update_test_case API with Status="PUBLISHED", the API consistently returns InvalidTestCaseException: Invalid test case content even though:

1. The same test case can be published successfully via AWS Connect Console GUI
2. The test case content follows the official AWS documentation schema
3. The test case uploads successfully with Status="SAVED"

Environment

• boto3 version: 1.42.59
• botocore version: 1.42.59
• Python version: 3.14.3
• Region: eu-west-2
• Instance created: 2026-03-03 (same day as testing)

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Expected Behavior

The update_test_case API should successfully publish the test case, just like the GUI does.

Current Behavior

Actual Behavior

botocore.errorfactory.InvalidTestCaseException: An error occurred (InvalidTestCaseException) 
when calling the UpdateTestCase operation: Invalid test case content

Reproduction Steps

Steps to Reproduce

1. Create test case with SAVED status (succeeds):

import boto3
import json

client = boto3.client('connect', region_name='eu-west-2')
instance_id = "45ec8eb8-b9dc-46cf-91ee-2943bcf41011"

content = {
    "Version": "2025-06-21",
    "Metadata": {},
    "Observations": [
        {
            "Identifier": "TestStart",
            "Event": {
                "Type": "TestInitiated",
                "Actor": "System",
                "Properties": {}
            },
            "Usage": {"Type": "EXACTLY"},
            "Actions": [
                {
                    "Identifier": "EndTest",
                    "Type": "TestControl",
                    "Parameters": {
                        "ActionType": "TestControl",
                        "Command": {"Type": "EndTest"}
                    },
                    "Transitions": {"NextAction": ""}
                }
            ],
            "Transitions": {"NextObservations": []}
        }
    ]
}

entry_point = {
    "Type": "VOICE_CALL",
    "VoiceCallEntryPointParameters": {
        "SourcePhoneNumber": "+442046392128",
        "DestinationPhoneNumber": "+442046392128"
    }
}

# This succeeds
response = client.create_test_case(
    InstanceId=instance_id,
    Name="TEST_Minimal",
    Description="Minimal test",
    Content=json.dumps(content),
    EntryPoint=entry_point,
    Status="SAVED"
)
test_id = response['TestCaseId']
print(f"Created: {test_id}")

2. Attempt to publish (fails):

# This fails with InvalidTestCaseException
client.update_test_case(
    InstanceId=instance_id,
    TestCaseId=test_id,
    Name="TEST_Minimal",
    Description="Minimal test",
    Content=json.dumps(content),
    EntryPoint=entry_point,
    Status="PUBLISHED"
)

3. Publish via AWS Console GUI (succeeds):
   • Navigate to AWS Connect Console → Test Cases
   • Click on the test case
   • Click "Publish" button
   • ✅ Successfully publishes

Possible Solution

No response

Additional Information/Context

No response

SDK version used

boto3 1.42.59

Environment details (OS name and version, etc.)

Environment - boto3 version: 1.42.59 - botocore version: 1.42.59 - Python version: 3.14.3 - Region: eu-west-2 - Instance created: 2026-03-03 (same day as testing)

[alokkulkarni]

boto3/botocore Fix Proposal

Issue Summary

update_test_case(Status="PUBLISHED") for Amazon Connect throws
InvalidTestCaseException: Invalid test case content, but:

• create_test_case(Status="SAVED") succeeds
• The exact same content publishes successfully via the AWS Console GUI
• describe_test_case() returns None for both Content and EntryPoint

---

Root Cause Analysis

Two separate bugs — both in botocore, not boto3:

Bug 1 — Incomplete service model response shape (confirmed)

botocore/data/connect/2017-08-08/service-2.json does not include
Content and EntryPoint in the DescribeTestCase response shape.
botocore only deserialises fields present in the model, so the server
does return these fields but botocore silently drops them → None.

This also means callers cannot round-trip: describe → modify → update,
because the content they need to re-send is invisible.

Bug 2 — Missing pre-call content validation + opaque error message

When Status=PUBLISHED, the Connect service performs additional
server-side validation of the Content JSON string. If it fails, the
API returns InvalidTestCaseException: Invalid test case content with
no detail. botocore passes this through verbatim.

The GUI succeeds because it likely sends an additional request context
(e.g. a clientToken or a pre-validation pre-flight) that the raw API
call omits, OR it enriches the Content before sending.

botocore should: (a) validate the Content JSON structure before the
call and surface a clear error, and (b) attach the server's diagnostic
payload to the exception when the call fails.

---

Required Changes

1. botocore/data/connect/2017-08-08/service-2.json

Where: The shapes block — DescribeTestCaseResponse and related.

What to add / change:

a) Add Content and EntryPoint to DescribeTestCaseResponse

 "DescribeTestCaseResponse": {
   "type": "structure",
   "members": {
+    "Content": {
+      "shape": "TestCaseContent",
+      "documentation": "The JSON content of the test case."
+    },
+    "EntryPoint": {
+      "shape": "TestCaseEntryPoint",
+      "documentation": "The entry point for the test case."
+    },
     "Name":        { "shape": "TestCaseName" },
     "Description": { "shape": "Description" },
     "Status":      { "shape": "TestCaseStatus" },
     ...
   }
 }

b) Add InvalidTestCaseException to the UpdateTestCase error list

 "UpdateTestCase": {
   "http": { "method": "PUT", ... },
   "input":  { "shape": "UpdateTestCaseRequest" },
   "output": { "shape": "UpdateTestCaseResponse" },
   "errors": [
     { "shape": "InvalidRequestException" },
     { "shape": "ResourceNotFoundException" },
+    {
+      "shape": "InvalidTestCaseException",
+      "documentation":
+        "Raised when the Content JSON fails server-side validation,
+         typically when Status=PUBLISHED. Check that all Observation
+         Event.Type values are one of: TestInitiated, MessageReceived,
+         ActionTriggered, TestCompleted. Ensure every observation has
+         a Transitions block and the final observation has an EndTest
+         TestControl action."
+    },
     ...
   ]
 }

c) Add missing TestCaseEntryPoint shape (if absent)

"TestCaseEntryPoint": {
  "type": "structure",
  "members": {
    "Type": { "shape": "TestCaseEntryPointType" },
    "VoiceCallEntryPointParameters": {
      "shape": "VoiceCallEntryPointParameters"
    }
  }
},
"TestCaseEntryPointType": {
  "type": "string",
  "enum": ["VOICE_CALL"]
},
"VoiceCallEntryPointParameters": {
  "type": "structure",
  "members": {
    "SourcePhoneNumber":      { "shape": "PhoneNumber" },
    "DestinationPhoneNumber": { "shape": "PhoneNumber" }
  }
}

---

2. botocore/handlers.py

Where: After the existing per-service handler functions (e.g. near remove_lex_v2_start_conversation).
Pattern: Follow the existing style — a module-level function registered via BUILTIN_HANDLERS.

a) Add validation handler function

# ── Amazon Connect: test case content validation ──────────────────────────────

_CONNECT_TC_REQUIRED_OBS_FIELDS = {"Identifier", "Event", "Transitions"}
_CONNECT_TC_VALID_EVENT_TYPES = {
    "TestInitiated",
    "MessageReceived",
    "ActionTriggered",
    "TestCompleted",
}
_CONNECT_TC_VALID_ACTION_TYPES = {
    "SendInstruction",
    "TestControl",
    "Assert",
    "OverrideSystemBehavior",
    "Wait",
}


def validate_connect_test_case_content(params, **kwargs):
    """Pre-call validation for Connect CreateTestCase / UpdateTestCase.

    Parses the ``Content`` JSON string and performs structural checks so that
    callers receive a meaningful ``ParamValidationError`` before the request
    reaches the service — rather than the opaque server-side
    ``InvalidTestCaseException: Invalid test case content``.

    Checks enforced:
    - ``Content`` must be valid JSON.
    - Top-level ``Version`` must be present and equal ``"2025-06-21"``.
    - ``Observations`` must be a non-empty list.
    - Each observation must have ``Identifier``, ``Event``, and
      ``Transitions``.
    - ``Event.Type`` must be one of the supported values.
    - ``Actions[].Type`` must be one of the supported values.
    - The final observation (or any observation with empty
      ``NextObservations``) must contain at least one ``TestControl /
      EndTest`` action.
    - When ``Status == "PUBLISHED"``, at least one ``MessageReceived``
      observation must be present (unambiguously testable flows require at
      least one prompt assertion).
    """
    content_str = params.get("Content")
    status = params.get("Status", "SAVED")

    if not content_str:
        return  # let the service return its own error for missing Content

    # 1. JSON parseable?
    try:
        content = json.loads(content_str)
    except (ValueError, TypeError) as exc:
        raise ParamValidationError(
            report=(
                f"Invalid value for parameter Content: not valid JSON. "
                f"Detail: {exc}"
            )
        )

    errors = []

    # 2. Version
    version = content.get("Version")
    if version is None:
        errors.append("Content is missing required field 'Version'.")
    elif version != "2025-06-21":
        errors.append(
            f"Content Version '{version}' is not supported; "
            "expected '2025-06-21'."
        )

    # 3. Observations
    observations = content.get("Observations")
    if not isinstance(observations, list) or len(observations) == 0:
        errors.append("Content 'Observations' must be a non-empty list.")
        # Cannot continue structural checks without observations
        if errors:
            raise ParamValidationError(report="\n".join(errors))

    # 4. Per-observation checks
    has_end_test = False
    has_message_received = False
    for idx, obs in enumerate(observations):
        loc = f"Observations[{idx}] (Identifier={obs.get('Identifier', '?')})"
        for field in _CONNECT_TC_REQUIRED_OBS_FIELDS:
            if field not in obs:
                errors.append(f"{loc}: missing required field '{field}'.")

        event = obs.get("Event", {})
        event_type = event.get("Type")
        if event_type and event_type not in _CONNECT_TC_VALID_EVENT_TYPES:
            errors.append(
                f"{loc}: Event.Type '{event_type}' is not valid. "
                f"Must be one of: {sorted(_CONNECT_TC_VALID_EVENT_TYPES)}."
            )
        if event_type == "MessageReceived":
            has_message_received = True

        for aidx, action in enumerate(obs.get("Actions", [])):
            action_type = action.get("Type")
            if action_type and action_type not in _CONNECT_TC_VALID_ACTION_TYPES:
                errors.append(
                    f"{loc} Actions[{aidx}]: Type '{action_type}' is not valid. "
                    f"Must be one of: {sorted(_CONNECT_TC_VALID_ACTION_TYPES)}."
                )
            # Detect EndTest
            if action_type == "TestControl":
                cmd = action.get("Parameters", {}).get("Command", {})
                if cmd.get("Type") == "EndTest":
                    has_end_test = True

    if not has_end_test:
        errors.append(
            "Content does not contain a TestControl / EndTest action. "
            "Every test case must terminate with "
            "{'Type': 'TestControl', 'Parameters': {'Command': {'Type': 'EndTest'}}}."
        )

    if status == "PUBLISHED" and not has_message_received:
        errors.append(
            "Content has no 'MessageReceived' observation. "
            "PUBLISHED test cases must assert at least one prompt from the flow."
        )

    if errors:
        raise ParamValidationError(report="\n".join(errors))

b) Register the handler in BUILTIN_HANDLERS

 BUILTIN_HANDLERS = [
     ...
+    #############
+    # Amazon Connect
+    #############
+    (
+        'before-parameter-build.connect.CreateTestCase',
+        validate_connect_test_case_content,
+    ),
+    (
+        'before-parameter-build.connect.UpdateTestCase',
+        validate_connect_test_case_content,
+    ),
     ...
 ]

Note: Use before-parameter-build (not before-call) so the error
is raised before serialisation — matching the pattern used by other
validation handlers in this file such as validate_bucket_name.

---

3. botocore/data/connect/2017-08-08/paginators-1.json (minor)

If ListTestCases is not already paginated, add:

"ListTestCases": {
  "input_token":  "NextToken",
  "output_token": "NextToken",
  "limit_key":    "MaxResults",
  "result_key":   "TestCaseSummaryList"
}

---

Files Changed Summary

File	Change type	Reason
botocore/data/connect/2017-08-08/service-2.json	Data (model)	Add Content + EntryPoint to DescribeTestCaseResponse; add InvalidTestCaseException to UpdateTestCase errors
botocore/handlers.py	Code	Add validate_connect_test_case_content() + register in BUILTIN_HANDLERS
botocore/data/connect/2017-08-08/paginators-1.json	Data	Add ListTestCases pagination config (if absent)

No changes to boto3 itself are needed — boto3 delegates all Connect API
calls to botocore.

---

How the fix resolves each symptom

Symptom	Fix
describe_test_case() returns None for Content / EntryPoint	service-2.json response shape update (Bug 1)
Opaque InvalidTestCaseException: Invalid test case content	Pre-call handler raises ParamValidationError with field-level detail before the request is sent (Bug 2)
No indication which field/observation is problematic	Handler pinpoints the offending observation by index and identifier
PUBLISHED fails while SAVED succeeds	Handler enforces PUBLISHED-specific rule (must have MessageReceived) so the caller is told exactly what is missing

---

Testing (botocore style)

New test file: tests/unit/test_connect_handlers.py

import json
import pytest
from botocore.exceptions import ParamValidationError
from botocore.handlers import validate_connect_test_case_content


VALID_CONTENT = json.dumps({
    "Version": "2025-06-21",
    "Metadata": {},
    "Observations": [
        {
            "Identifier": "TestStart",
            "Event": {"Type": "TestInitiated", "Actor": "System"},
            "Usage": {"Type": "EXACTLY"},
            "Actions": [],
            "Transitions": {"NextObservations": ["Greeting"]},
        },
        {
            "Identifier": "Greeting",
            "Event": {
                "Type": "MessageReceived",
                "Actor": "System",
                "Properties": {"Text": "Hello"},
                "MatchingCriteria": "Similarity",
            },
            "Usage": {"Type": "EXACTLY"},
            "Actions": [
                {
                    "Identifier": "EndTest",
                    "Type": "TestControl",
                    "Parameters": {
                        "ActionType": "TestControl",
                        "Command": {"Type": "EndTest"},
                    },
                    "Transitions": {"NextAction": ""},
                }
            ],
            "Transitions": {"NextObservations": []},
        },
    ],
})


def test_valid_saved_content_passes():
    validate_connect_test_case_content(
        {"Content": VALID_CONTENT, "Status": "SAVED"}
    )  # no exception


def test_valid_published_content_passes():
    validate_connect_test_case_content(
        {"Content": VALID_CONTENT, "Status": "PUBLISHED"}
    )  # no exception


def test_invalid_json_raises():
    with pytest.raises(ParamValidationError, match="not valid JSON"):
        validate_connect_test_case_content({"Content": "{bad json", "Status": "SAVED"})


def test_missing_version_raises():
    content = json.loads(VALID_CONTENT)
    del content["Version"]
    with pytest.raises(ParamValidationError, match="Version"):
        validate_connect_test_case_content(
            {"Content": json.dumps(content), "Status": "SAVED"}
        )


def test_wrong_version_raises():
    content = json.loads(VALID_CONTENT)
    content["Version"] = "2020-01-01"
    with pytest.raises(ParamValidationError, match="2020-01-01"):
        validate_connect_test_case_content(
            {"Content": json.dumps(content), "Status": "SAVED"}
        )


def test_missing_end_test_raises():
    content = json.loads(VALID_CONTENT)
    content["Observations"][1]["Actions"] = []  # remove EndTest
    with pytest.raises(ParamValidationError, match="EndTest"):
        validate_connect_test_case_content(
            {"Content": json.dumps(content), "Status": "SAVED"}
        )


def test_invalid_event_type_raises():
    content = json.loads(VALID_CONTENT)
    content["Observations"][0]["Event"]["Type"] = "UnknownEvent"
    with pytest.raises(ParamValidationError, match="UnknownEvent"):
        validate_connect_test_case_content(
            {"Content": json.dumps(content), "Status": "SAVED"}
        )


def test_published_without_message_received_raises():
    content = json.loads(VALID_CONTENT)
    # Keep only TestInitiated + EndTest
    content["Observations"] = [
        {
            "Identifier": "TestStart",
            "Event": {"Type": "TestInitiated"},
            "Actions": [
                {
                    "Identifier": "EndTest",
                    "Type": "TestControl",
                    "Parameters": {"Command": {"Type": "EndTest"}},
                    "Transitions": {"NextAction": ""},
                }
            ],
            "Transitions": {"NextObservations": []},
        }
    ]
    with pytest.raises(ParamValidationError, match="MessageReceived"):
        validate_connect_test_case_content(
            {"Content": json.dumps(content), "Status": "PUBLISHED"}
        )

---

Contribution Notes (botocore style)

• service-2.json is auto-generated from the AWS SDK model spec — the
  correct upstream fix is to update the Smithy/JSON model in the
  aws/aws-sdk-js-v3 or aws/aws-models internal repository and re-run
  codegen. The PR should tag @aws-sdk-python-automation.
• Custom handler additions to handlers.py follow the established pattern:
  module-level function + BUILTIN_HANDLERS tuple entry. No class is needed.
• The json and ParamValidationError imports are already present at the
  top of handlers.py.

[adev-code]

Hello @alokkulkarni, thanks for reaching out. I repro'd your code and got the same error from the service. I got:

2026-03-04 10:09:22,376 urllib3.connectionpool [DEBUG] https://connect.eu-west-2.amazonaws.com:443 "POST /test-cases/0520d6dc-6055-4ceb-a474-735ee19703d2/59d35083-bf7d-4eef-aad0-6fd3792b3886 HTTP/1.1" 400 196
2026-03-04 10:09:22,377 botocore.parsers [DEBUG] Response body:
b'{"message":"Invalid test case content ","problemDetails":[{"message":"InvalidObservationProblem(observationIdentifier=TestStart, actionIdentifier=null, reason=Invalid usage type)"}],"reason":null}'

I have reached out to the service team and will update you once there are updates from the team.

Internal Ref: P393121041

[alokkulkarni]

Thanks @adev-code , currently this does not allow us to auto publish them in the whole automated process.