[ICCV 2025] AdsQA: Towards Advertisement Video Understanding Arxiv: https://arxiv.org/abs/2509.08621

Community benchmark database for running LLMs on Apple Silicon Macs

Benchmark LLMs on real professional tasks, not academic puzzles. YAML-driven experiment pipeline + live React dashboard for GDPVal Gold Subset (220 tasks across 11 industries).

Benchmark for evaluating AI epistemic reliability - testing how well LLMs handle uncertainty, avoid hallucinations, and acknowledge what they don't know.

An agent evaluation framework with native multi-turn feedback iteration.

CapBencher toolkit: Give your LLM benchmark a built-in alarm for leakage and gaming

Community-driven behavioral reliability benchmark for LLMs. 231 probes across 19 modules, deterministic scoring, perplexity correlation, layer sensitivity mapping, quant method capture, hardware-stratified community rankings. Every test contributes to the community dataset.

Testing how well LLMs can solve jigsaw puzzles

Open-source multi-agent AI debate arena: pit Claude, GPT, Gemini, Ollama & HuggingFace models against each other with frozen-context fairness, evidence-first judging, 20+ personas, code review, and PDF/Markdown reports. CLI + Web UI.

Self-hosted LLM API benchmark, monitoring & playground. Compare latency, TTFT, throughput across OpenAI, Anthropic, Gemini & any OpenAI-compatible endpoint. Deploy with one command via Docker. | 自托管 LLM API 性能测试、监控与调试平台，一键 Docker 部署，支持多家服务商对比。

Comprehensive benchmark of OpenRouter free-tier LLMs for practical applications. Evaluates models for coding, Thai language, and general use.

🚀 A modern, production-ready refactor of the LoCoMo long-term memory benchmark.

Benchmark for evaluating safety of AI agents in irreversible financial decisions (crypto payment settlement, consensus conflicts, replay attacks, finality races).

🔍 Benchmark jailbreak resilience in LLMs with JailBench for clear insights and improved model defenses against jailbreak attempts.

LiveSecBench（大模型动态安全测评基准）是大模型安全领域的专业、动态、多维度测评基准。我们致力于通过科学、系统、持续演进的测评体系，客观评估与衡量大模型的安全性能，推动大模型技术向更安全、更可靠、更负责任的方向发展，为产业落地和学术研究提供关键的安全标尺。

Benchmark LLM jailbreak resilience across providers with standardized tests, adversarial mode, rich analytics, and a clean Web UI.

Benchmark abierto de 44 modelos LLM con 5,000+ tests reales. Alternativas a Claude, GPT-5 y Gemini para agentes N8N, OpenClaw y emprendedores. Calculadora interactiva + LLM-as-Judge Phi-4.

Исследовательский вопрос: можно ли измерить «офисный интеллект» LLM? Попытка — здесь. 100 сценариев, 10 критериев, русский корпоративный контекст.

Benchmark LLMs Spatial Reasoning with Head-to-Head Bananagrams

RetardBench is an open, no-censorship benchmark that ranks large language models purely on how retarded they are.