Skip to content

[DNM] OTEL: Instrument ctx.sample() / sampling loop#3892

Draft
strawgate wants to merge 3 commits intoPrefectHQ:mainfrom
strawgate:strawgate/otel-sampling-instrumentation
Draft

[DNM] OTEL: Instrument ctx.sample() / sampling loop#3892
strawgate wants to merge 3 commits intoPrefectHQ:mainfrom
strawgate:strawgate/otel-sampling-instrumentation

Conversation

@strawgate
Copy link
Copy Markdown
Collaborator

Summary

  • Adds OpenTelemetry tracing to the ctx.sample() / sampling loop pipeline (sample_impl, sample_step_impl, execute_tools) with sampling/createMessage, sampling/createMessage step, and sampling.execute_tool spans
  • Records structured attributes (temperature, max_tokens, tool_count, result_type, iteration count, stop reason) and events for validation failures and text response retries
  • Supports opt-in message content capture via OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT env var
  • Adds is_recording() guards to server_span, delegate_span, and client_span to skip attribute serialization on non-recording spans
  • Uses type(e).__qualname__ for error.type attribute for better nested class identification

Closes #3891

Test plan

  • 9 new tests in tests/server/telemetry/test_sampling_tracing.py covering:
    • sampling/createMessage span with correct attributes and iteration count
    • sampling/createMessage step child spans per loop iteration
    • sampling.execute_tool spans for tool calls within sampling
    • Validation failure events
    • Text response retry events
  • All 26 existing telemetry tests continue to pass
  • ruff check src/ passes

🤖 Generated with Claude Code

strawgate and others added 2 commits April 13, 2026 01:28
Wrap attribute-setting blocks in server_span, delegate_span, and
client_span with `if span.is_recording():` to avoid unnecessary work
on non-recording spans. Add error.type attribute using
`type(e).__qualname__` for proper exception class identification.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add OTEL tracing to the sampling pipeline:
- `sampling/createMessage` span wrapping sample_impl() with attributes for
  temperature, max_tokens, tool_count, result_type, and iteration count
- `sampling/createMessage step` child spans per loop iteration with
  iteration index and stop reason
- `sampling.execute_tool <name>` spans for tool calls within sampling,
  including error status on failures
- Validation failure and text response retry events on the parent span
- Opt-in content capture via OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT

Also adds is_recording() guards to server_span, delegate_span, and
client_span to avoid unnecessary attribute serialization when sampling is
disabled, and sets error.type using __qualname__ for better nested class
names.

Closes PrefectHQ#3891

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@marvin-context-protocol marvin-context-protocol bot added too-long Excessively verbose or unedited LLM output. Condense before triage. DON'T MERGE PR is not ready for merging. Used by authors to prevent premature merging. labels Apr 13, 2026
@marvin-context-protocol
Copy link
Copy Markdown
Contributor

Thanks for the report. This issue goes beyond what our contributor guidelines ask for — we just need a short problem description and an MRE. Please see our contributing guidelines and condense this issue. We'll triage it once it's trimmed down.

@marvin-context-protocol
Copy link
Copy Markdown
Contributor

marvin-context-protocol bot commented Apr 13, 2026

Test Failure Analysis

Summary: The static_analysis job failed because ruff format found formatting violations in src/fastmcp/server/sampling/run.py. (ty now passes — that issue from the previous triage has been resolved.)

Root Cause: ruff format reformatted src/fastmcp/server/sampling/run.py in CI, meaning the file wasn't formatted locally before committing. The violations are all in the new OTEL instrumentation code added by this PR.

Suggested Solution:

Run ruff format locally and commit the result:

uv run ruff format src/fastmcp/server/sampling/run.py

Then commit the reformatted file.

Detailed Analysis

CI output:

ruff check...   Passed
ruff format...  Failed
- hook id: ruff-format
- files were modified by this hook
ty check...     Passed

Diff applied by ruff format in CI (all in src/fastmcp/server/sampling/run.py):

  1. _CAPTURE_CONTENT assignment (~line 44) — chained .lower() == "true" needs wrapping in parentheses:
# Before (fails format)
_CAPTURE_CONTENT = os.environ.get(
    "OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT", "false"
).lower() == "true"

# After (passes format)
_CAPTURE_CONTENT = (
    os.environ.get(
        "OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT", "false"
    ).lower()
    == "true"
)
  1. set_attributes({...}) calls (~lines 296, 661) — dict literals passed directly as arguments need to be moved to their own indented block.

  2. set_attribute(...) and dump_python(...) calls (~lines 691, 736, 826) — previously wrapped across multiple lines, ruff collapses them back to single lines.

Related Files
  • src/fastmcp/server/sampling/run.py — new OTEL instrumentation code with ruff format violations

🤖 Triage by Marvin (updated to reflect latest run — ty errors resolved, ruff format still failing)

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DON'T MERGE PR is not ready for merging. Used by authors to prevent premature merging. too-long Excessively verbose or unedited LLM output. Condense before triage.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OTEL: Instrument ctx.sample() / sampling loop with spans and metrics

1 participant