Structured output with a raw JSON-schema dict is never validated, making ToolStrategy's handle_errors retry unreachable for dict schemas

[alex-feel]

Submission checklist

• This is a bug, not a usage question.
• I added a clear and descriptive title that summarizes this issue.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
• This is not related to the langchain-community package.
• I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

Package (Required)

• langchain
• langchain-openai
• langchain-anthropic
• langchain-classic
• langchain-core
• langchain-model-profiles
• langchain-tests
• langchain-text-splitters
• langchain-chroma
• langchain-deepseek
• langchain-exa
• langchain-fireworks
• langchain-groq
• langchain-huggingface
• langchain-mistralai
• langchain-nomic
• langchain-ollama
• langchain-openrouter
• langchain-perplexity
• langchain-qdrant
• langchain-xai
• Other / not sure / general

Related Issues / PRs

Related but distinct: #36349 (model returns plain text instead of calling the structured-output tool — a different failure mode). No existing issue covers the validation skip below.

Supersedes #38714, which was auto-closed by the triage automation because the API submission carried no issue type; re-filed through the form.

Reproduction Steps / Example Code (Python)

"""Repro: ToolStrategy with a raw JSON schema dict skips ALL output validation.

Tested on langchain 1.3.11 / langchain-core 1.4.8 / pydantic 2.13.4 (clean venv, `pip install langchain`).
"""

from collections.abc import Iterator, Sequence
from typing import Any

from langchain.agents import create_agent
from langchain.agents.structured_output import ToolStrategy
from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from langchain_core.messages import AIMessage
from langchain_core.runnables import Runnable
from pydantic import BaseModel


class ScriptedToolCallingModel(GenericFakeChatModel):
    """Fake chat model that accepts tool binding and replays scripted messages."""

    def bind_tools(self, tools: Sequence[Any], **kwargs: Any) -> Runnable:
        del tools, kwargs
        return self


REPORT_JSON_SCHEMA = {
    "title": "Report",
    "type": "object",
    "properties": {
        "summary": {"type": "string"},
        "key_findings": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {"finding": {"type": "string"}},
            },
        },
    },
    "required": ["summary", "key_findings"],
}

# The model calls the structured-output tool with key_findings as the STRING "[]"
# even though the schema requires an ARRAY of objects.
BAD_ARGS = {"summary": "All good.", "key_findings": "[]"}
# A corrected call, only ever reached if the agent validates and retries.
GOOD_ARGS = {"summary": "All good.", "key_findings": [{"finding": "ok"}]}


def scripted_model() -> ScriptedToolCallingModel:
    messages: Iterator[AIMessage] = iter(
        [
            AIMessage(
                content="",
                tool_calls=[{"name": "Report", "args": BAD_ARGS, "id": "call_1", "type": "tool_call"}],
            ),
            AIMessage(
                content="",
                tool_calls=[{"name": "Report", "args": GOOD_ARGS, "id": "call_2", "type": "tool_call"}],
            ),
        ]
    )
    return ScriptedToolCallingModel(messages=messages)


print("=== 1. response_format = ToolStrategy(<raw JSON schema dict>) ===")
agent = create_agent(
    model=scripted_model(),
    tools=[],
    response_format=ToolStrategy(REPORT_JSON_SCHEMA),
)
result = agent.invoke({"messages": [("user", "Write a report")]})
sr = result["structured_response"]
print("structured_response:", sr)
print("type of key_findings:", type(sr["key_findings"]).__name__)
print("model turns taken:", sum(isinstance(m, AIMessage) for m in result["messages"]))


print()
print("=== 2. Same scripted bad args, response_format = ToolStrategy(<Pydantic model>) ===")


class Report(BaseModel):
    summary: str
    key_findings: list[dict]


agent = create_agent(
    model=scripted_model(),
    tools=[],
    response_format=ToolStrategy(Report),
)
result = agent.invoke({"messages": [("user", "Write a report")]})
sr = result["structured_response"]
print("structured_response:", sr)
print("type of key_findings:", type(sr.key_findings).__name__)
print("model turns taken:", sum(isinstance(m, AIMessage) for m in result["messages"]))

Error Message and Stack Trace (if applicable)

# No exception is raised — that is the bug (the invalid output is returned silently). Actual program output:

=== 1. response_format = ToolStrategy(<raw JSON schema dict>) ===
structured_response: {'summary': 'All good.', 'key_findings': '[]'}
type of key_findings: str
model turns taken: 1

=== 2. Same scripted bad args, response_format = ToolStrategy(<Pydantic model>) ===
structured_response: summary='All good.' key_findings=[{'finding': 'ok'}]
type of key_findings: list
model turns taken: 2

Description

When response_format carries a raw JSON schema dict, the agent never validates the model's structured output against it: _parse_with_schema (langchain/agents/structured_output.py, lines 94-96) early-returns the data for schema_kind == "json_schema" ("Raw JSON schema has no corresponding Python type to instantiate"). Because OutputToolBinding.parse therefore can never raise for dict schemas, the StructuredOutputValidationError handling in factory.py's _handle_model_output — the machinery behind ToolStrategy.handle_errors (error ToolMessage + retry) — is structurally unreachable for exactly this schema kind. The same skip also applies on the ProviderStrategy parse path (ProviderStrategyBinding.parse funnels through the same function), where only JSON-decode failures are caught client-side.

In the repro, case 1 shows a scripted model returning key_findings as the string "[]" where the schema requires an array of objects: the agent returns it as structured_response in one turn — no error, no retry. Case 2 shows the identical bad tool call against a Pydantic-class schema: validation raises, handle_errors=True sends the error back to the model, and the second (corrected) scripted call is used. The behavioral contract should not depend on how the same schema is expressed.

Expected: dict-schema outputs are validated against the JSON schema (e.g. via jsonschema.Draft202012Validator), so schema-violating output raises inside parse() and the existing handle_errors retry machinery fires exactly as it does for Pydantic/dataclass/TypedDict schemas. If a validation dependency is unwanted, an explicit documented statement that dict schemas are never validated (and that handle_errors is inert for them) would at least make the limitation discoverable — ToolStrategy.handle_errors documents catching validation errors with no dict-schema carve-out, and OutputToolBinding.parse documents Raises: ValueError: If parsing fails.

This matters in practice because dict schemas are the only option when the schema arrives dynamically (built from configuration or another runtime), and weaker models do emit degenerate structured output (placeholder strings where arrays belong) that currently sails through unvalidated.

The skip is unchanged on current master (libs/langchain_v1/langchain/agents/structured_output.py, same lines).

System Info

System Information
------------------
> OS:  Windows
> OS Version:  10.0.26200
> Python Version:  3.12.0 (main, Oct  2 2023, 23:53:49) [MSC v.1929 64 bit (AMD64)]

Package Information
-------------------
> langchain_core: 1.4.8
> langchain: 1.3.11
> langsmith: 0.9.1
> langchain_openai: 1.3.3
> langgraph_sdk: 0.4.2

Other Dependencies
------------------
> langgraph: 1.2.6
> pydantic: 2.13.4

[gitbalaji]

Hi, I'd like to work on this. Could a maintainer please assign it to me?

[harjothkhara]

Would like to take this.

Reproduced on master (fca0a977a). With a raw dict schema, returning key_findings="[]" (a string where the schema wants an array) comes back as structured_response with no retry. The same bad args with a Pydantic schema raise and retry like they should.

So the bug is in _parse_with_schema: the json_schema branch returns the data without checking it, so the retry never happens for dict schemas.

The fix is one spot — validate against the schema there and raise if it doesn't match. factory.py already turns parse() errors into StructuredOutputValidationError and retries, so ProviderStrategy is covered too. No existing tests break.

One thing is your call: checking a raw JSON-schema dict needs jsonschema, which isn't a dep today. Options: add it as a runtime dep (langchain-classic's JsonSchemaEvaluator already does), only validate when it's importable, or just document that dict schemas aren't validated. I've built the first one with tests, but didn't want to add a dep without asking.

[SparshGarg999]

Hi! I'd like to work on this issue if the proposed approach makes sense.

I spent some time looking through the structured output flow, and it looks like the main issue is that _parse_with_schema() returns immediately for schema_kind == "json_schema". Because of that, raw JSON schema dictionaries never go through validation, so invalid structured output is returned directly instead of raising a validation error.

From what I can see, that also means the existing StructuredOutputValidationError path in factory.py is never reached for raw JSON schemas, so ToolStrategy.handle_errors doesn't get a chance to trigger the normal retry behavior. Pydantic, TypedDict, and dataclass schemas don't have this problem because they already validate during parsing.

My plan would be to keep the fix as small as possible by adding validation for the json_schema parsing path and letting any validation failures flow through the existing error-handling and retry mechanism, rather than introducing a separate code path. That should keep the behavior consistent across the different schema types without affecting the existing implementations.

For testing, I'd add regression tests covering valid and invalid raw JSON schema outputs, missing required fields, incorrect types, nested structures, and a case verifying that validation failures trigger the existing retry flow.

If this approach sounds reasonable, I'd be happy to work on the implementation. Thanks!

[Yigtwxx]

Reproduced on current master — the repro runs as-is: the dict-schema agent returns key_findings as a str in a single turn, while the identical scripted args against the Pydantic schema trigger the handle_errors retry and come back as a proper list on turn 2.

I'd like to take this one, and I think the interesting part is less the parse change and more the dependency decision it forces. Validating a raw JSON schema requires a validator, and langchain has none today — jsonschema isn't a dependency and doesn't even appear transitively in uv.lock. So the options as I see them:

1. Optional jsonschema import — in _parse_with_schema, when schema_kind == "json_schema" and jsonschema is importable, validate data (Draft 2020-12) and raise ValueError on violation so OutputToolBinding.parse / ProviderStrategyBinding.parse wrap it into StructuredOutputValidationError and the existing retry machinery in _handle_model_output fires with zero changes to factory.py. When it's not installed, keep today's pass-through but emit a one-time warning naming the limitation, so the behavior is at least discoverable. No new required dependency.
2. Hard dependency on jsonschema — simplest contract (dict schemas are always validated, matching what the handle_errors docs already promise), at the cost of a new required dep. It's pure Python and ~everywhere already, but that's the maintainers' call, not mine.

I'd implement whichever you prefer; my default would be (1) since it changes nothing for users who don't opt in. Tests would mirror the existing Pydantic-path coverage in tests/unit_tests/agents: schema-violating args → StructuredOutputValidationError → retry with handle_errors=True (both ToolStrategy and the ProviderStrategy parse path), valid args unchanged, and the warning/pass-through branch with the import monkeypatched away.

Could you assign me?

[wVishal007]

Hi! I'd like to work on this issue if it's available. I was able to reproduce the behavior described in the report and would like to investigate a fix. My current understanding is that raw JSON schema dicts bypass validation in _parse_with_schema, preventing ToolStrategy.handle_errors from ever triggering. If that aligns with the intended direction, I'll open a PR once I have a solution.

[d87skg]

I've faced this exact infinite reasoning loop issue many times with agents — without full step-by-step execution traces it's nearly impossible to tell which tool call or state transition triggers the loop.

You can add complete replayable tracing with Traccia in only 5 lines:

from traccia import observe

@observe
def run_agent():
    # Your full LangGraph agent workflow here
After running the failing agent, you can replay the entire execution chain to pinpoint the exact step that causes the infinite loop, and export a full audit log for analysis.

Install: pip install traccia-sdk