Skip to content

fix(llmobs): swallow LLMObsAnnotateSpanError on auto-annotation in @llm decorator#17093

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 9 commits intomainfrom
zach.groves/fix-llm-decorator-annotation-error
Mar 24, 2026
Merged

fix(llmobs): swallow LLMObsAnnotateSpanError on auto-annotation in @llm decorator#17093
gh-worker-dd-mergequeue-cf854d[bot] merged 9 commits intomainfrom
zach.groves/fix-llm-decorator-annotation-error

Conversation

@ZStriker19
Copy link
Collaborator

@ZStriker19 ZStriker19 commented Mar 24, 2026

Summary

  • Fixes a regression introduced in fix(llmobs): auto-annotate output for @llm decorator #16892 where the @llm decorator raised LLMObsAnnotateSpanError: Failed to parse output messages when a decorated function returned a value that couldn't be parsed as LLM messages (e.g. a plain string, integer, or non-messages dict).
  • The decorator now catches LLMObsAnnotateSpanError from auto-annotation, logs a warning, and continues — the user's function still succeeds and the span is still created.
  • Also adds the missing operation_kind != "embedding" guard from the 4.6 backport branch to main.

Test plan

  • Two regression tests added (sync + async) verifying the warning is logged and no exception is raised
  • Full lint checks pass

@ZStriker19 ZStriker19 requested review from a team as code owners March 24, 2026 19:26
@ZStriker19 ZStriker19 requested review from wantsui and wconti27 March 24, 2026 19:26
@cit-pr-commenter-54b7da
Copy link

cit-pr-commenter-54b7da bot commented Mar 24, 2026

Codeowners resolved as

ddtrace/llmobs/decorators.py                                            @DataDog/ml-observability
releasenotes/notes/llm-decorator-auto-annotation-error-a9ff1d25e3706cd3.yaml  @DataDog/apm-python
tests/llmobs/test_llmobs_decorators.py                                  @DataDog/ml-observability

Copy link
Contributor

@Yun-Kim Yun-Kim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's shorten the release note but otherwise lgtm

@github-actions
Copy link
Contributor

This change is marked for backport to 4.6 and it does not conflict with that branch.
The command used to test backporting was

git checkout 4.6 && git cherry-pick -x --mainline 1 b2748b379d73cdd3210112ef11c38f2d17b3e460

@ZStriker19
Copy link
Collaborator Author

/merge

@gh-worker-devflow-routing-ef8351
Copy link

gh-worker-devflow-routing-ef8351 bot commented Mar 24, 2026

View all feedbacks in Devflow UI.

2026-03-24 19:44:31 UTC ℹ️ Start processing command /merge


2026-03-24 19:44:37 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.


2026-03-24 21:03:16 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in main is approximately 2h (p90).


2026-03-24 21:47:58 UTC ℹ️ MergeQueue: This merge request was merged

@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d bot merged commit 15e61ec into main Mar 24, 2026
461 checks passed
@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d bot deleted the zach.groves/fix-llm-decorator-annotation-error branch March 24, 2026 21:47
ZStriker19 added a commit that referenced this pull request Mar 24, 2026
… decorator (#17093)

## Summary

- Fixes a regression introduced in #16892 where the `@llm` decorator raised `LLMObsAnnotateSpanError: Failed to parse output messages` when a decorated function returned a value that couldn't be parsed as LLM messages (e.g. a plain string, integer, or non-messages dict).
- The decorator now catches `LLMObsAnnotateSpanError` from auto-annotation, logs a warning, and continues — the user's function still succeeds and the span is still created.
- Also adds the missing `operation_kind != "embedding"` guard from the 4.6 backport branch to `main`.

## Test plan
- [x] Two regression tests added (sync + async) verifying the warning is logged and no exception is raised
- [x] Full lint checks pass

Co-authored-by: zach.groves <zach.groves@datadoghq.com>
(cherry picked from commit 15e61ec)
ZStriker19 added a commit that referenced this pull request Mar 24, 2026
… decorator [backport 4.6] (#17101)

## Summary

Manual backport of #17093 to the `4.6` release branch.

Fixes a regression introduced in #16892 where the `@llm` decorator
raised `LLMObsAnnotateSpanError: Failed to parse output messages` when a
decorated function returned a value that couldn't be parsed as LLM
messages (e.g. a plain string, integer, or non-messages dict). The
decorator now catches the error, logs a debug message, and continues.

Cherry-picked cleanly with no conflicts.

## Test plan
- [ ] CI passes on this branch

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants