Support multi-LLM judging for more robust evaluation

[AstroBoy1]

Problem

Relying on a single LLM judge can introduce model-specific biases and variability in scoring which may affect the reliability and robustness of benchmark results.

Reference

This paper shows using a panel of llms for judging outperforms a single llm as a judge. https://arxiv.org/abs/2404.18796

Solution

Add support for multiple LLM judges such as Deepseek, Qwen, Gemma, Granite, and aggregate their score with a majority vote if the score is binary or an average if the score is continuous. Report agreement metric such as Cohen’s Kappa or Fleiss’ Kappa to give a measure of confidence.

[AyaanShaheer]

I’d like to work on this.

I’m currently exploring the evaluation pipeline and planning a modular multi-judge architecture with:

• provider-agnostic judge interfaces
• async parallel judge execution
• configurable aggregation strategies (majority vote / averaging)
• agreement metrics such as Cohen’s and Fleiss’ Kappa

To keep the initial PR focused, I’m thinking of implementing the core abstraction layer plus support for 1–2 judge providers first, with extensibility for additional providers later.

Open to feedback on whether this aligns with the intended direction.

[AstroBoy1]

That sounds good! I think it could improve transparency to have a reference to support the choice of judges too, which ideally should have the best performance, and maybe in the future (probably beyond the scope of this issue), some specific empirical results relevant to this benchmark of the judge being judged.

[AyaanShaheer]

Thanks for the feedback — that aligns well with the direction I was considering.

I’d be happy to work on this issue.

My plan is to keep the initial PR focused and maintainable by implementing the core multi-judge infrastructure first, including:

• provider-agnostic judge interfaces
• async parallel execution for multiple judges
• configurable aggregation strategies (majority vote / averaging)
• agreement metrics such as Cohen’s Kappa and Fleiss’ Kappa
• transparent reporting of both per-judge and aggregated scores

To avoid over-scoping the first contribution, I’ll likely start with support for 1–2 judge providers initially while designing the abstraction layer to make additional judges easy to integrate later.

I also agree that judge selection should be transparent and grounded in prior work. I’ll reference relevant LLM-as-a-Judge literature (including the paper linked above) and document the rationale behind the included judges in the implementation/docs.

If this approach sounds good, I’ll begin working on the core evaluation abstraction and open a draft PR once the initial design is ready for feedback.