[BUG]: Enabling Capture Requests and Responses From AWS Services adds material latency to spans

[miketheman]

Tracer Version(s)

4.5.3

Python Version(s)

Python 3.13.12

Pip Version(s)

pip 26.0.1

Bug Report

Setting DD_TRACE_CLOUD_REQUEST_PAYLOAD_TAGGING=all and DD_TRACE_CLOUD_RESPONSE_PAYLOAD_TAGGING=all adds material latency to S3 calls, to the point that Watchdog even flagged it.

Setting both variables to "" and redeploying drops levels down to prior latencies

Original feature implementation: #10642

Reproduction Code

• Set DD_TRACE_CLOUD_REQUEST_PAYLOAD_TAGGING=all and DD_TRACE_CLOUD_RESPONSE_PAYLOAD_TAGGING=all
• Run warehouse, connect to S3
• See latencies spike

Error Logs

No response

Libraries in Use

Details

alembic==1.18.4
alembic-postgresql-enum==1.10.0
amqp==5.3.1
annotated-types==0.7.0
argon2-cffi==25.1.0
argon2-cffi-bindings==25.1.0
attrs==25.4.0
Automat==25.4.16
b2sdk==2.10.4
babel==2.18.0
bcrypt==4.3.0
billiard==4.2.4
boto3==1.42.29
botocore==1.42.29
bytecode==0.17.0
cachetools==7.0.2
cattrs==26.1.0
cbor2==5.8.0
celery==5.6.2
celery-redbeat==2.3.3
certifi==2026.2.25
cffi==2.0.0
charset-normalizer==3.4.4
click==8.3.1
click-didyoumean==0.3.1
click-plugins==1.1.1.2
click-repl==0.3.0
cmarkgfm==2025.10.22
cryptography==46.0.5
cssselect==1.4.0
cssutils==2.11.1
datadog==0.52.1
ddtrace==4.5.3
Deprecated==1.3.1
disposable-email-domains==0.0.167
dnspython==2.8.0
docutils==0.22.4
email-validator==2.3.0
envier==0.6.1
Events==0.5
filelock==3.25.0
forcediphttpsadapter==1.1.0
github_reserved_names==2026.3.1
google-api-core==2.29.0
google-auth==2.48.0
google-cloud-bigquery==3.40.1
google-cloud-core==2.5.0
google-cloud-storage==3.9.0
google-crc32c==1.8.0
google-resumable-media==2.8.0
googleapis-common-protos==1.72.0
greenlet==3.3.1
grpcio==1.78.1
grpcio-status==1.78.1
gunicorn==23.0.0
hiredis==3.3.0
html5lib==1.1
humanize==4.15.0
hupper==1.12.1
id==1.6.1
idna==3.11
importlib_metadata==8.7.1
isodate==0.7.2
itsdangerous==2.2.0
Jinja2==3.1.6
jmespath==1.1.0
jsonschema==4.26.0
jsonschema-path==0.3.4
jsonschema-specifications==2025.9.1
kombu==5.6.2
lazy-object-proxy==1.12.0
legacy-cgi==2.6.4
limits==5.8.0
linehaul==1.0.3
logfury==1.0.1
lxml==6.0.2
Mako==1.3.10
markdown-it-py==4.0.0
MarkupSafe==3.0.3
mdurl==0.1.2
more-itertools==10.8.0
msgpack==1.1.2
natsort==8.4.0
nh3==0.3.3
openapi-core==0.22.0
openapi-schema-validator==0.6.3
openapi-spec-validator==0.7.2
opensearch-protobufs==0.19.0
opensearch-py==3.1.0
opentelemetry-api==1.40.0
orjson==3.11.7
packaging==26.0
packaging-legacy==23.0.post0
paginate==0.5.7
paginate-sqlalchemy==0.3.1
passlib==1.7.4
PasteDeploy==3.1.0
pathable==0.4.4
plaster==1.1.2
plaster-pastedeploy==1.0.1
platformdirs==4.9.2
premailer==3.10.0
prompt_toolkit==3.0.52
proto-plus==1.27.1
protobuf==6.33.5
psycopg==3.3.3
psycopg-binary==3.3.3
pyasn1==0.6.2
pyasn1_modules==0.4.2
pycparser==3.0
pydantic==2.12.5
pydantic_core==2.41.5
Pygments==2.19.2
PyJWT==2.11.0
pymacaroons==0.13.0
PyNaCl==1.6.2
pyOpenSSL==25.3.0
pyparsing==3.3.2
pypi-attestations==0.0.29
pyqrcode-binary==1.2.1
pyramid==2.0.2
pyramid-mailer==0.15.1
pyramid-redirect==0.4
pyramid-retry==2.1.1
pyramid-rpc==0.8
pyramid-services==2.2
pyramid_jinja2==2.10.1
pyramid_openapi3==0.21.0
pyramid_tm==2.6
python-dateutil==2.9.0.post0
python-slugify==8.0.4
pytz==2026.1.post1
PyYAML==6.0.3
readme_renderer==44.0
redis==6.4.0
referencing==0.36.2
repoze.sendmail==4.4.1
requests==2.32.5
requests-aws4auth==1.3.1
requests-file==3.0.1
rfc3161-client==1.0.5
rfc3339-validator==0.1.4
rfc3986==2.0.0
rfc8785==0.1.4
rich==14.3.3
rpds-py==0.30.0
rsa==4.9.1
s3transfer==0.16.0
securesystemslib==1.3.1
sentry-sdk==2.54.0
setuptools==80.10.2
sigstore==4.2.0
sigstore-models==0.0.6
sigstore-rekor-types==0.0.18
six==1.17.0
SQLAlchemy==2.0.48
stdlib-list==0.12.0
stripe==11.6.0
structlog==25.5.0
tenacity==9.1.4
text-unidecode==1.3
tldextract==5.3.1
transaction==5.0
translationstring==1.4
trove-classifiers==2026.1.14.14
tuf==6.0.0
typing-inspection==0.4.2
typing_extensions==4.15.0
tzdata==2025.3
tzlocal==5.3.1
ua-parser==1.0.1
ua-parser-builtins==202603
urllib3==2.6.3
venusian==3.1.1
vine==5.1.0
wcwidth==0.6.0
webauthn==2.7.1
webencodings==0.5.1
WebOb==1.8.9
Werkzeug==3.1.6
whitenoise==6.12.0
wired==0.4
wrapt==2.1.1
WTForms==3.2.1
yara-x==1.13.0
zipp==3.23.0
zope.deprecation==6.0
zope.interface==8.2
zope.sqlalchemy==4.1
zxcvbn==4.5.0

Operating System

Linux pypi-warehouse-worker-5c6cd577fb-ssjgv 6.8.0-1028-aws #30-Ubuntu SMP Tue Apr 15 19:46:32 UTC 2025 aarch64 GNU/Linux