Add "Your LLM-as-Judge Is Lying to You" to §8 (LLM-as-judge & verifiers)#46
Open
loopandretry wants to merge 1 commit into
Open
Add "Your LLM-as-Judge Is Lying to You" to §8 (LLM-as-judge & verifiers)#46loopandretry wants to merge 1 commit into
loopandretry wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds one practitioner essay to §8 · LLM-as-judge & verifiers.
Link: https://loopandretry.github.io/posts/llm-as-judge-is-lying-to-you/
Why it clears the bar (show your work, not a generic 'you need evals' take): the post is a worked mechanism piece. It shows why raw judge-vs-human agreement is misleading — an 84% agreement number collapses to Cohen's kappa 0.50 (only moderate) once you subtract chance agreement, and an always-'pass' judge that reads nothing scores kappa 0.0 — with the arithmetic and code shown inline. It then covers hardening that actually moves kappa: prefer pairwise over absolute scores, and require the judge to quote its evidence before scoring. Same practitioner lane as the existing Eugene Yan / Hamel Husain / Han-Chung Lee entries in this section.
Disclosure: I write this blog; submitting because it's genuinely on-topic for §8, not a marketing page. Link is live (200); single focused diff.