Getting empty content while evaluation with llm as judge

[gadirajupavantiwari]

Describe your question

Hi team, I am getting inputs for both arguments from live traces, when I set eval with llm as judge, llm 2nd argument is showing empty, please check once.

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If Self-Hosted

No response

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

When setting up LLM-as-a-Judge evaluations, empty content in evaluation arguments typically indicates incorrect variable mapping.(1)

Common causes:

1. Incorrect variable mapping: When configuring the evaluator, you need to map prompt template variables (like {{query}} and {{generation}}) to the actual data in your traces or observations.(1) If the mapping points to a field that doesn't exist or is empty in your trace data, the evaluation will receive empty content.(1)

2. Using JSONPath for nested data: If your data is nested within a JSON object, you need to use JSONPath expressions (like $.choices[0].message.content) to precisely locate it.(1) Without proper JSONPath, Langfuse may not extract the data correctly.(1)

3. Missing data in traces: All mapped variables must exist in every trace for evaluation to succeed.(2) If required variables are missing in traces, evaluations may fail or show empty content.(2)

How to fix:

• Navigate to your evaluator configuration and review the "Map Variables & preview Evaluation Prompt" section.(1)
• Use the live preview feature that shows how actual data from your last 24 hours of traces fills the prompt.(1) This helps verify your mappings are correct.(1)
• Check if you need JSONPath expressions for nested fields.(1)
• Verify that the trace/observation fields you're mapping to actually contain data.(1)

A similar issue was reported where both {{query}} and {{generation}} were mapped to trace input instead of generation input/output, resulting in incorrect evaluations.(3) The resolution involved correcting the variable mapping in the evaluator configuration.(3)

📚 Sources:

1. LLM-as-a-Judge
2. bug:
3. bug: Evaluations fail - {{generation}} variable shows user query instead of agent response (OpenAI Agents SDK)

---

Have another question? Just tag @inkeep.

[gadirajupavantiwari]

Verified with above common causes, no above mentioned cause in my case, please help @jannikmaierhoefer

[CodeCLS]

Hi @gadirajupavantiwari, I don't quite understand your original screenshot (what data is missing and where).

[gadirajupavantiwari]

above is screenshot of preview of a sample when setting llm eval, there both arguments {{query}}, {{ava_routing}} are capturing, but while executing llm eval {{ava_routing}} argument is not filling.

[CodeCLS]

How is "ava_routing" different from "query" (data types/setup)? Does the trace contain the exact same word match and where exactly(what kind of an observation for example) for both "query" and "ava_routing"?

[gadirajupavantiwari]

both are same, no difference, both arguments extracted using json path, in preview both are capturing but while execution, ava_routing is not displaying.