I work on whether we can trust what AI systems tell us about themselves - the scores we benchmark them with, and the internal signals we'd use to catch them misbehaving.

My work has moved along a line: from building with LLMs, to measuring them, to auditing whether those measurements can be trusted. That last question - is a safety signal real, or does it just look real? - is where I spend my time now. The recurring move across everything below is the same: build the measurement, then try honestly to break it, and report exactly what survived, nulls included.

Two questions organize the current work:

• Can we measure a model honestly? A benchmark score can be gamed, contaminated, or confidently wrong when the model senses it's being tested. → RealBench, IsItBenchmark, EvalBracket
• Can we trust the internal signals we'd use to catch it? A probe at AUROC 0.9, a guard classifier, an activation direction, a CoT monitor - decodable is not the same as trustworthy. → SIEVE, RAMP, verbalizer-vs-cot, agent-oversight

SIEVE - Safety Indicator Evidence Validation Engine. Audits whether an activation-based safety signal (a probe direction, an SAE feature, a monitor score, a verbalizer output) is merely decodable or actually causally load-bearing, and emits a scoped, config-hashed evidence card. A validity layer upstream of AI control: vet a signal before it's trusted in a safety case. Five verdicts from not_decodable to causally_sufficient, with four worked example audits in-repo; the discipline is to never claim more than you tested.

RAMP - the robustness companion to SIEVE. Runs a pre-registered survival ladder over a safety signal - leakage-free cross-fitting, blind evaluation, distribution shift - and reports the highest rung it survives, from no_value to distribution_robust. Audits both probes you build and guards others have published (Llama Guard, Qwen3Guard). Where SIEVE asks whether moving the signal moves behavior, RAMP asks whether its measured value survives honest evaluation.

Probing Is Not Enough - eval/deploy directions

• the origin result. Eval/deploy framing is linearly decodable from two open-weight reasoning models (Qwen3-32B, gpt-oss-120b) at high AUROC - yet on Qwen3-32B (L55), steering that decoded direction under a full matched control suite (random / orthogonal / wrong-layer, two judges) moves behavior no more than an equal-magnitude random perturbation. Decodable is not causal - the lesson SIEVE generalizes. (Academic paper drafted; the companion "validity audit for any probe" writeup that proposes SIEVE is published on the EA Forum and LessWrong.)

verbalizer-vs-cot - SIEVE applied to activation verbalizers across five pre-registered runs: when does verbalizing a model's internals beat reading its chain-of-thought? A training-free verbalizer confabulates; a trained decoder recovers the signal (0.615 → 0.911 AUROC) but is span-fragile; the honest verdict is inconclusive. The harness caught a confabulating verbalizer, refused to over-credit a fragile one - and caught a design error in my own pre-registration mid-series.

agent-oversight - does oversight of an agent survive an adversary trying to evade it? A staged attack/defense study (gated experiments, attacks and baselines, its own experiment design and paper outline) - the AI-control face of the same validity question.

EvalBracket - when a model behaves differently because it senses it's being evaluated, a point benchmark score can be confidently wrong. EvalBracket reports a calibrated [floor, ceiling] interval instead - floor ≈ deployed capability, ceiling ≈ elicitable - so the measurement carries its own validity bounds.

IsItBenchmark - is a given prompt "benchmarky," and is a reported score inflated by contamination? A contamination-detection system with a trained specialized detector - the memorization-vs-competence question in code.

RealBench - a real-world benchmark that scores models on tasks people actually use AI for, with multi-dimensional scoring beyond a single number. The earliest thread of "measure models honestly," and where the validity concern started.

csvglow - beautiful interactive HTML dashboards from CSV/Excel, one command. My most-used tool (npm-packaged, Dockerized). MixtureOfRecursions and microgpt-explainer - a paper reimplementation and a visual transformer explainer, from working through the fundamentals.

Pre-registration before results. Matched controls before conclusions. Nulls reported as nulls, and scope stated in the artifact; every claim bound to what was actually tested. The through-line isn't a method; it's a stance: the most useful thing you can do for a safety signal is try honestly to break it, and say what survived.

I come at model internals from a safety-and-deployment angle rather than a research-lab background; and that is the point I double down on in validating safety signals for real world. The questions I chase are the ones that matter when someone has to trust a signal in production, not just publish one.

Interested in AI safety, interpretability validity, and evaluation methodology; you can reach me at ratnaditya@gmail.com