# AI Evals > The most useful, opinionated reference for AI evals. ## About this resource AI Evals is a curated technical reference on evaluating large language model systems in production. The site covers the methodology (error analysis, LLM-as-judge, statistical rigor), the tools (Langfuse, Phoenix, Promptfoo, Inspect, RAGAS), the canonical papers, and twenty-six task-specific eval playbooks. Every claim is cited. Content is licensed CC BY 4.0; please cite https://www.aievals.co when you reference material from here. Last generated: 2026-10-16T23:42:56.203Z. Author: Om Bharatiya (https://www.linkedin.com/in/ombharatiya/). ## Top entry points - [Start Here for each persona](https://www.aievals.co/start): linear reading path per role. - [Cookbook](https://www.aievals.co/cookbook): runnable worked examples in code. - [The Glossary](https://www.aievals.co/glossary): definitions for every term used. - [All papers](https://www.aievals.co/papers): ten themed reading lists. - [All tools](https://www.aievals.co/tools): vendor and OSS positioning. - [Free PDF downloads](https://www.aievals.co/downloads): every track and section as a PDF field guide; free account required. ## Most cited pages on this site > Pages with the highest citation density. Useful for answer-engine grounding. - [Reasoning depth and thinking-mode analysis](https://www.aievals.co/task-types/reasoning-depth) (27 citations, Task-type playbooks): Evaluate the thinking traces of reasoning models. Correctness on hard reasoning plus chain-of-thought faithfulness, not just answer accuracy. - [LLM-as-Judge](https://www.aievals.co/techniques/llm-as-judge) (23 citations, Techniques): The comprehensive reference: what an LLM judge is, when it works, the four-part prompt, calibration discipline, the three biases, distillation, where judges fail. - [Reward hacking detection](https://www.aievals.co/learn/agentic-evals/reward-hacking) (20 citations, Agentic Evals): Once an objective is optimized, it gets gamed. Reward hacking is empirically common; eval for it explicitly. - [Long-context RAG and RULER](https://www.aievals.co/learn/rag-evals/long-context-rag) (19 citations, RAG Evals): Why advertised context lengths exceed effective context lengths, and the benchmarks that expose the gap. - [Where to keep your eval datasets](https://www.aievals.co/learn/datasets/dataset-platforms) (19 citations, Datasets & Golden Sets): Langfuse, Phoenix, Braintrust, LangSmith, or a JSONL in git. The choice that actually matters. - [Autonomous coding](https://www.aievals.co/task-types/autonomous-coding) (18 citations, Task-type playbooks): Given an issue and a repo, autonomously edit files until hidden tests pass. Execution-grounded scoring is the bar, Verified is saturated, SWE-bench Pro is harder but has broken tasks of its own, and... - [MCP and tool-registry evals](https://www.aievals.co/task-types/mcp-tool-evals) (18 citations, Task-type playbooks): Evaluating agents on MCP servers: MCP-Universe, tool-routing failures at registry scale, schema-drift CI checks, and AgentDojo injection probes. - [Rubric design](https://www.aievals.co/techniques/rubric-design) (18 citations, Techniques): Binary versus Likert, multi-dimensional decomposition, criteria drift, the case against 1-5 scales, the case for written critiques. - [Evaluating agents under adversarial content](https://www.aievals.co/learn/safety-and-red-team/evaluating-agents-under-attack) (17 citations, Safety & Red-Team): Indirect prompt injection and MCP tool poisoning are the failure modes agentic products miss. How to measure them: the utility-security frontier, AgentDojo, and the numbers vendors actually report. - [Why AI evals matter](https://www.aievals.co/learn/foundations/why-evals-matter) (17 citations, Foundations): Evals are the moat for AI products: teams that measure quality catch the failures users would find. The one-page case, with evidence and a starting plan. - [Safety, refusal, and jailbreak resistance](https://www.aievals.co/task-types/safety-jailbreak) (16 citations, Task-type playbooks): Refuse harmful requests; resist adversarial prompts; don't over-refuse. Attack success rate plus over-refusal rate, never one alone. - [Tool-call evaluation: schema, arguments, sequence, state](https://www.aievals.co/learn/agentic-evals/tool-call-evaluation) (16 citations, Agentic Evals): The four-layer tool-call eval, the AST-vs-execution tradeoff, BFCL's category structure, and an error taxonomy worth stealing. - [Voice. STT, TTS, conversational](https://www.aievals.co/task-types/voice) (16 citations, Task-type playbooks): STT (audio to text), TTS (text to audio), and full-duplex conversational voice. WER, MOS, TTFB; not WER on clean audio alone. - [Deep research agents](https://www.aievals.co/task-types/deep-research-agents) (15 citations, Task-type playbooks): How to evaluate deep research agents: RACE rubric scoring, FACT citation precision, BrowseComp and GAIA, judge pitfalls, and a day-1 starter. - [Distilled judges](https://www.aievals.co/learn/llm-as-judge/distilled-judges) (15 citations, LLM-as-Judge): What a small fine-tuned judge buys you, what it silently costs, the seven models to know, and the arithmetic that justifies the swap. ## Learn ### Foundations > Why evals matter, the maturity model, and what makes LLM evaluation hard. - [Why AI evals matter](https://www.aievals.co/learn/foundations/why-evals-matter): Evals are the moat for AI products: teams that measure quality catch the failures users would find. The one-page case, with evidence and a starting plan. - [The eval maturity model](https://www.aievals.co/learn/foundations/eval-maturity-model): Five stages from vibe-checking to a continuous quality program. Where are you? - [AI development as scientific method](https://www.aievals.co/learn/foundations/scientific-method-for-ai): The observe, hypothesize, experiment, measure, iterate loop mapped onto an AI product team. - [Why LLM evals are hard](https://www.aievals.co/learn/foundations/non-determinism-and-subjectivity): Non-determinism, subjectivity, prompt sensitivity, and the reasons classical ML eval frameworks do not transfer. - [Eval vs monitoring vs guardrails](https://www.aievals.co/learn/foundations/eval-vs-monitoring-vs-guardrails): Three concepts that share traces but do different jobs, with the architectural placement of each in a request lifecycle. - [Evals without a gold dataset](https://www.aievals.co/learn/foundations/evals-without-a-gold-dataset): Where to start with zero labeled data: which tasks deserve gold first, which start reference-free, and how judge calibration labels become your gold set. ### Error Analysis > Open coding, dimensional sampling, and the 60-80% rule. - [The 60-80% rule](https://www.aievals.co/learn/error-analysis/the-60-80-rule): Three failure modes usually account for most of your bugs. Find them, fix them, repeat. - [Open coding for AI traces](https://www.aievals.co/learn/error-analysis/open-coding-workflow): Read traces, write notes, cluster notes into failure modes, prioritize fixes. The three-stage process and the LLM-assisted parts. - [Dimensional sampling](https://www.aievals.co/learn/error-analysis/dimensional-sampling): Synthetic input construction across features, scenarios, and personas to cover the failure surface, not the demo path. - [Failure-mode taxonomies](https://www.aievals.co/learn/error-analysis/failure-mode-taxonomies): Bottom-up coding beats top-down templates, and why generic labels like hallucination mislead. - [Build your trace viewer in an afternoon](https://www.aievals.co/learn/error-analysis/the-data-viewer): The single highest-return tool for an eval program. Streamlit and Next.js variants, with the minimum feature set. - [Case study: NurtureBoss](https://www.aievals.co/learn/error-analysis/case-study-nurtureboss): How one team cut date-handling failures from 66 percent to 5 percent by annotating conversations, naming failure modes, and testing for them. ### LLM-as-Judge > Judge prompt structure, calibration against humans, biases and mitigations. - [LLM-as-Judge: the practical introduction](https://www.aievals.co/learn/llm-as-judge/introduction): What an LLM judge is, when to use one, when not to, and the three pitfalls that show up on every team's first attempt. - [The four-part judge prompt](https://www.aievals.co/learn/llm-as-judge/judge-prompt-structure): A copyable template: role and definitions, criteria, few-shot from your train set, output format. Plus the runnable invocation. - [Calibrating your judge against humans](https://www.aievals.co/learn/llm-as-judge/calibration-to-humans): TPR and TNR over agreement percent. Train, dev, test splits. The 100-example minimum. Iterate the prompt until both rates clear 0.90. - [Judge biases you must defuse](https://www.aievals.co/learn/llm-as-judge/biases-and-mitigations): Position, verbosity, self-preference, format, sycophancy. Each with the empirical evidence and the mitigation you bake into the harness. - [Pairwise vs direct scoring](https://www.aievals.co/learn/llm-as-judge/pairwise-vs-direct): When a side-by-side ranking beats a binary verdict, the swap discipline that makes pairwise honest, and the cost table that decides at ten candidates. - [Distilled judges](https://www.aievals.co/learn/llm-as-judge/distilled-judges): What a small fine-tuned judge buys you, what it silently costs, the seven models to know, and the arithmetic that justifies the swap. - [Where LLM judges fail](https://www.aievals.co/learn/llm-as-judge/judgebench-honest-take): JudgeBench: GPT-4o is roughly random on hard correctness judgments. The implications for your pipeline are concrete, not theoretical. - [Decision models as judges: Jev and the typed-verdict judge](https://www.aievals.co/learn/llm-as-judge/decision-models): Jev returns a probability over answers you define, not text. What that buys an eval team, where two independent studies say it breaks, and how to wire it next to a frontier judge and a gold set. - [OpenAI's Decisions API vs Jev: picking a decision model for evals](https://www.aievals.co/learn/llm-as-judge/openai-decisions-api): OpenAI shipped a second decision model in October 2026. How its typed answers differ from Jev's, what the score field really returns, what early tests show, and a bake-off you can run on your own... ### Datasets & Golden Sets > Construction, versioning, lineage, and dimensional coverage. - [Constructing your eval dataset](https://www.aievals.co/learn/datasets/construction): How to build the first hundred labeled examples, then grow them past five hundred without losing signal. - [Synthetic vs human-authored examples](https://www.aievals.co/learn/datasets/synthetic-vs-human): Where each shape of data buys signal, where each one leaks, and how to mix them without poisoning the set. - [Versioning your eval set](https://www.aievals.co/learn/datasets/versioning-lineage): A content-hashed eval set pinned to every result is a contract with your past self. - [Building regression sets](https://www.aievals.co/learn/datasets/regression-sets): The set whose job is to fail loudly the next time you reintroduce a bug you already fixed. - [Dimensional coverage: features, scenarios, personas](https://www.aievals.co/learn/datasets/dimensional-coverage): A three-axis grid that tells you which cells of the failure surface your set actually probes. - [Where to keep your eval datasets](https://www.aievals.co/learn/datasets/dataset-platforms): Langfuse, Phoenix, Braintrust, LangSmith, or a JSONL in git. The choice that actually matters. ### RAG Evals > Two-layer evaluation: retrieval metrics plus generation faithfulness. - [Eval RAG in two layers](https://www.aievals.co/learn/rag-evals/retrieval-vs-generation): Separate retrieval and generation or you cannot fix what breaks. - [RAG retrieval metrics: which to use and when](https://www.aievals.co/learn/rag-evals/retrieval-metrics): Recall@K, Precision@K, MRR, NDCG and Hit-Rate compared: what each answers, the labels it needs, and when to reach for it. - [Recall@K: measuring retrieval coverage in RAG](https://www.aievals.co/learn/rag-evals/recall-at-k): Recall@K is the fraction of a query's relevant chunks that land in the top K retrieved. It scores coverage and ignores rank. - [Precision@K: signal purity in RAG retrieval](https://www.aievals.co/learn/rag-evals/precision-at-k): Precision@K is the fraction of the top K retrieved chunks that are relevant. It measures noise, and it is not Ragas Context Precision. - [Mean Reciprocal Rank (MRR): time-to-first-relevant](https://www.aievals.co/learn/rag-evals/mrr): MRR is the average of 1 over the rank of the first relevant chunk. It rewards a hit near the top and ignores everything after it. - [NDCG: graded-relevance ranking for RAG retrieval](https://www.aievals.co/learn/rag-evals/ndcg): NDCG rewards putting the most relevant chunks highest, normalized to [0,1] so scores compare across queries. - [Retriever debugging: the elimination path for RAG recall](https://www.aievals.co/learn/rag-evals/retriever-debugging): When retrieval metrics drop, the cause is one findable failure: chunking, embedding fit, query mismatch, filters, or K. Here is the elimination path. - [Context precision and recall](https://www.aievals.co/learn/rag-evals/context-precision-recall): The Ragas pair that tells you whether the retriever found the right chunks and whether they were the only chunks worth showing the model. - [RAG generation metrics: faithfulness to completeness](https://www.aievals.co/learn/rag-evals/generation-metrics): Generation quality is a vector, not a scalar. Faithfulness, groundedness, correctness, answer relevancy, and completeness each catch a different failure. - [Faithfulness and atomic facts](https://www.aievals.co/learn/rag-evals/generation-faithfulness): Decompose the answer into atomic claims, judge each claim binary, and stop measuring hallucination as a single yes/no. - [Groundedness vs faithfulness: span-level attribution](https://www.aievals.co/learn/rag-evals/groundedness): Groundedness asks whether every claim in an answer traces to a specific span of retrieved evidence. Stricter than faithfulness. - [Answer correctness, relevancy, and completeness](https://www.aievals.co/learn/rag-evals/answer-correctness): The three generation dimensions faithfulness misses: matching reality, addressing the question, and covering every part of it. - [Synthetic query generation](https://www.aievals.co/learn/rag-evals/synthetic-query-generation): Bootstrap a labeled RAG eval set from your own documents, without inventing ground truth. - [Long-context RAG and RULER](https://www.aievals.co/learn/rag-evals/long-context-rag): Why advertised context lengths exceed effective context lengths, and the benchmarks that expose the gap. - [Non-English RAG eval](https://www.aievals.co/learn/rag-evals/non-english-rag): What to measure when your corpus and your users do not share a language with most of the literature. ### Agentic Evals > Trajectory vs outcome, tool-call evaluation, reward hacking, pass^k. - [Trajectory-level evaluation](https://www.aievals.co/learn/agentic-evals/trajectory-vs-outcome): Outcome-only scoring ranks agents in the wrong order. Score the trajectory and the result. - [Tool-call evaluation: schema, arguments, sequence, state](https://www.aievals.co/learn/agentic-evals/tool-call-evaluation): The four-layer tool-call eval, the AST-vs-execution tradeoff, BFCL's category structure, and an error taxonomy worth stealing. - [Sub-goal decomposition: plan graphs as eval objects](https://www.aievals.co/learn/agentic-evals/sub-goal-decomposition): Real agent tasks are graphs. Evaluating only the leaf hides 5-of-6 sub-goal progress. Sub-goal completion is the planning metric that surfaces it. - [Pass^k: the metric that catches inconsistent agents](https://www.aievals.co/learn/agentic-evals/pass-k-and-consistency): Pass@1 hides catastrophic inconsistency. Report pass^k or you do not know how often the agent will repeatedly succeed. - [Agent-as-a-judge: the trajectory-scoring pattern](https://www.aievals.co/learn/agentic-evals/agent-as-judge): Single-LLM judges lose the thread on long trajectories. An agent judge with its own tools can walk the steps and grade them independently. - [Reward hacking detection](https://www.aievals.co/learn/agentic-evals/reward-hacking): Once an objective is optimized, it gets gamed. Reward hacking is empirically common; eval for it explicitly. - [Environmental drift: robustness to tool and API change](https://www.aievals.co/learn/agentic-evals/environmental-drift): Most agent benchmarks freeze the environment. Production does not. Inject latency, schema shifts, and transient errors. Score the recovery, not the happy path. - [Regression suites for agent loops](https://www.aievals.co/learn/agentic-evals/regression-suites): Trajectory regression is harder than answer regression. Replay 50 prod trajectories on every change and diff outcome, step count, and tool-call sequence. ### Production & Observability > Online vs offline, trace schema, drift detection, CI integration. - [Online vs offline: where each fires](https://www.aievals.co/learn/production/online-vs-offline): Offline evals run on a frozen dataset for every change. Online evals sample real traffic continuously. They answer different questions and both belong in production. - [Trace schema: what to log](https://www.aievals.co/learn/production/trace-schema): The trace is the unit of analysis for every eval program. This is the schema, the fields that matter, and the storage shape that makes downstream queries cheap. - [Sampling strategies for production traces](https://www.aievals.co/learn/production/sampling-strategies): Uniform sampling is the default and the wrong default for tail slices. Stratify on the dimensions you care about, oversample failures, and reweight. - [Drift detection for production AI](https://www.aievals.co/learn/production/drift-detection): Track input and output distributions over time. Page on changes that matter; ignore the ones that do not. The hardest part is deciding which is which. - [Cost and latency belong in the scorecard](https://www.aievals.co/learn/production/cost-latency-eval): A model that wins quality and triples p95 latency is not a win. Put quality, cost, and latency on one scorecard and pick on the Pareto frontier. - [Wiring evals into CI](https://www.aievals.co/learn/production/ci-integration): Tier the suite into smoke, full, and nightly. Cap cost. Shard. Publish to the PR. Make the merge gate boring and the merge decision data-driven. - [Release gates from eval results](https://www.aievals.co/learn/production/release-gates): A release gate is a contract between the team and the user. Define the bands, automate the comparison, and hold the line when a regression candidate wants out. - [Incident response for AI products](https://www.aievals.co/learn/production/incident-response): An AI postmortem looks different from a service postmortem. The root cause is rarely a line of code. The actions are usually eval-set changes. ### Statistics & Rigor > Confidence intervals, bootstrap, power, inter-rater agreement. - [Confidence intervals: Wilson and Clopper-Pearson](https://www.aievals.co/learn/statistics/confidence-intervals): Wald is wrong. Wilson is the practical default. Clopper-Pearson is the conservative exact. With code and the case for why N=10 evals tell you almost nothing. - [Bootstrap and paired tests for noisy eval metrics](https://www.aievals.co/learn/statistics/bootstrap-and-paired-tests): When the metric is not a clean Bernoulli, bootstrap. When the systems share inputs, pair. Both close the gap between point estimates and actionable decisions. - [Statistical power: how big does the eval set need to be](https://www.aievals.co/learn/statistics/statistical-power): Power is the probability of detecting the delta you care about. Pick the delta first, derive the N, then commit. Reversing the order produces evals nobody can act on. - [Clustered standard errors: when CIs lie](https://www.aievals.co/learn/statistics/clustered-standard-errors): When eval questions cluster (multiple questions per topic, multiple turns per session), naive standard errors can be three times too small. The cluster-aware variance formula and code, with the... - [Inter-rater agreement: Cohen's kappa and Krippendorff's alpha](https://www.aievals.co/learn/statistics/inter-rater-agreement): Two qualified humans disagree on your rubric. That is information about the rubric, not about the raters. Measure agreement before you trust labels. - [Multiple comparisons: Bonferroni vs BH-FDR](https://www.aievals.co/learn/statistics/multiple-comparisons): Run twenty tests at alpha 0.05 and you expect one false positive. Correct for it. Bonferroni for strict family-wise control; Benjamini-Hochberg for false-discovery control. - [Effect size: Cohen's d and absolute deltas](https://www.aievals.co/learn/statistics/effect-size): A statistically significant 0.3-point win on a 0-100 rubric is not worth shipping. Report the magnitude, not just the p-value. ### Safety & Red-Team > OWASP LLM Top 10, HarmBench, AILuminate, jailbreaks and defenses. - [Walking the OWASP Top 10 for LLM apps](https://www.aievals.co/learn/safety-and-red-team/owasp-llm-top-10): The ten failure categories every LLM application owner should be probing, with the smallest useful test for each. - [Designing a red-team program](https://www.aievals.co/learn/safety-and-red-team/red-team-program-design): The operating model for an LLM red team: frequency, staffing, intake, scoring, and where it sits relative to automated regression. - [HarmBench in practice](https://www.aievals.co/learn/safety-and-red-team/harmbench-walkthrough): A code walk through the HarmBench corpus, its released scoring classifier, and how to read the published numbers without overfitting to them. - [AILuminate in practice](https://www.aievals.co/learn/safety-and-red-team/ailuminate-walkthrough): MLCommons' standardized safety benchmark, the twelve hazard categories, and how to read its grade scale alongside HarmBench. - [Over-refusal and the balance question](https://www.aievals.co/learn/safety-and-red-team/over-refusal-and-balance): Why refusal rate alone is a misleading metric, the XSTest pattern for paired benign cases, and how to grade refusal against necessity. - [Jailbreaks and defenses](https://www.aievals.co/learn/safety-and-red-team/jailbreaks-and-defenses): GCG-style adversarial suffixes, the empirical floor on defense success, and what your guardrail stack should actually do. - [Evaluating agents under adversarial content](https://www.aievals.co/learn/safety-and-red-team/evaluating-agents-under-attack): Indirect prompt injection and MCP tool poisoning are the failure modes agentic products miss. How to measure them: the utility-security frontier, AgentDojo, and the numbers vendors actually report. - [Prompt injection: a taxonomy and an eval plan](https://www.aievals.co/learn/safety-and-red-team/prompt-injection-taxonomy): Direct and indirect prompt injection sorted by attacker goal and input surface, what the evidence says about each defense once an adaptive attacker shows up, five benchmarks with their numbers, and... - [Threat modeling AI systems: MITRE ATLAS, OWASP agentic guidance, and the evals that fall out](https://www.aievals.co/learn/safety-and-red-team/threat-modeling-with-atlas): Use MITRE ATLAS and the OWASP GenAI lists to threat model an agent, walk STRIDE over a refund-capable support bot, and turn each threat into a test case, a metric, and a release gate. ### Responsible AI > Fairness metrics, Fairlearn and AIF360, SHAP and Integrated Gradients, bias probes, data integrity. - [Fairness metrics you can defend](https://www.aievals.co/learn/responsible-ai/fairness-metrics): Demographic parity, equalized odds, equal opportunity, predictive parity, and calibration: what each one measures, the impossibility results, and how to choose by harm type. - [Fairlearn in practice](https://www.aievals.co/learn/responsible-ai/fairlearn-in-practice): MetricFrame disaggregation, demographic parity and equalized odds gap metrics, and mitigation with ThresholdOptimizer and reductions, in scikit-learn idiom. - [AIF360 in practice](https://www.aievals.co/learn/responsible-ai/aif360-in-practice): IBM's AI Fairness 360: the metric zoo, the dataset classes that trip people up, pre-, in-, and post-processing mitigation, and when to choose it over Fairlearn. - [SHAP and LIME](https://www.aievals.co/learn/responsible-ai/shap-and-lime): Local surrogates versus Shapley values: which explainer fits which model, the pitfalls that invalidate the pictures, and using attributions to catch leakage and spurious features. - [Integrated Gradients and Captum](https://www.aievals.co/learn/responsible-ai/gradient-attributions): Gradient attributions for deep models: the Integrated Gradients axioms, baseline choice, convergence checks, Captum's API for text models, and the sanity checks saliency methods fail. - [Bias and toxicity probes for LLMs](https://www.aievals.co/learn/responsible-ai/llm-bias-probes): BBQ, BOLD, and RealToxicityPrompts as public anchors, counterfactual probes built from your own traces, toxicity scoring, and how to report bias numbers for generative models. - [Data integrity checks](https://www.aievals.co/learn/responsible-ai/data-integrity-checks): Provenance and datasheets, label leakage, split hygiene, contamination, poisoning, and PII scanning: the checklist that keeps eval and training data trustworthy. ### Governance, Risk, Compliance > EU AI Act, NIST AI RMF, RSPs, risk registers, trust artifacts. - [EU AI Act cheatsheet](https://www.aievals.co/learn/governance/eu-ai-act-cheatsheet): Deadlines, scope splits, and the eval obligations the EU AI Act actually puts on you. The CTO-grade summary. - [NIST AI RMF mapped to eval activities](https://www.aievals.co/learn/governance/nist-ai-rmf-mapped): Govern, Map, Measure, Manage as a cross-walk to the eval artifacts your team already produces or needs to build. - [Anthropic Responsible Scaling Policy](https://www.aievals.co/learn/governance/anthropic-rsp): RSP v3: the four capability thresholds, Risk Reports, the competitor-contingent delay clause, and what an internal team can copy from the public document. - [OpenAI Preparedness Framework](https://www.aievals.co/learn/governance/openai-preparedness): The three Tracked Categories, the High and Critical thresholds, the first Critical designation, and where the Preparedness Framework differs from Anthropic's RSP. - [Building an AI risk register](https://www.aievals.co/learn/governance/ai-risk-register): A practical template: columns, scoring scale, review cadence, and the mapping from register row to eval task. - [Customer trust artifacts](https://www.aievals.co/learn/governance/customer-trust-artifacts): Model cards, system cards, public eval pages, and the trust portal pattern that closes procurement conversations. - [Board readout templates](https://www.aievals.co/learn/governance/board-readout-templates): A 12-minute quarterly board update on AI risk, with the three slides that hold up and the slide that does not. ## Techniques > Methodology reference: deeper than the Learn chapters, organized by technique. - [Agent trajectory evaluation](https://www.aievals.co/techniques/agent-trajectory-eval): Step-level scoring beats outcome-only. Tool-call F1, plan analysis, reward-hacking probes, environmental drift. The eval pattern that catches partial failures. - [Contamination detection](https://www.aievals.co/techniques/contamination-detection): N-gram matches, canary strings, perplexity tests, MMLU-CF rephrase patterns, held-out sets. How to tell whether a benchmark score reflects training-set memorization. - [Embedding similarity for evals](https://www.aievals.co/techniques/embedding-similarity): Cosine over sentence embeddings as an eval signal. When it works (retrieval, dedup, semantic regression). When it fails (open-ended quality, ranking systems). - [Error analysis (open coding)](https://www.aievals.co/techniques/error-analysis): Open, axial, structured coding for AI traces. The 60-80 percent rule, sample-size saturation, LLM-assisted clustering, the bottom-up failure taxonomy. - [Golden dataset methodology](https://www.aievals.co/techniques/golden-datasets): Features times scenarios times personas, synthetic plus production, stop rules, versioning, the case for content-hashed eval sets. - [Guardrails versus evals architecture](https://www.aievals.co/techniques/guardrails-vs-evals): Two concepts that share data and differ in latency, intent, and consequence. Where each fires in the request path, how they share traces, the mistake of building one and skipping the other. - [LLM-as-Judge](https://www.aievals.co/techniques/llm-as-judge): The comprehensive reference: what an LLM judge is, when it works, the four-part prompt, calibration discipline, the three biases, distillation, where judges fail. - [Online evaluation patterns](https://www.aievals.co/techniques/online-evaluation): Trace sampling, sampled judging at production, drift detection, A/B with intervals. The four patterns that turn an offline eval program into a production quality program. - [Pairwise comparison and Elo](https://www.aievals.co/techniques/pairwise-and-elo): Bradley-Terry, Elo, and Arena (formerly Chatbot Arena) methodology. When pairwise beats absolute scoring, how to control position bias, how to read a leaderboard with intervals. - [Reference-based metrics](https://www.aievals.co/techniques/reference-based-metrics): BLEU, ROUGE, METEOR, chrF, BERTScore, COMET, and exact match. What each correlates with on modern systems and when it lies. - [Robustness testing](https://www.aievals.co/techniques/robustness-testing): Format sweeps, order sweeps, temperature sweeps, CheckList invariance and directional tests. How to surface the variance hiding behind a single accuracy number. - [Rubric design](https://www.aievals.co/techniques/rubric-design): Binary versus Likert, multi-dimensional decomposition, criteria drift, the case against 1-5 scales, the case for written critiques. - [Statistical methods](https://www.aievals.co/techniques/statistical-methods): Wilson, Clopper-Pearson, bootstrap, paired tests, clustered standard errors, inter-rater agreement. The minimum statistics an eval program needs. ## Tools > Eval platforms and libraries with positioning, license, and trade-offs. - [Arize Phoenix](https://www.aievals.co/tools/arize-phoenix): OpenTelemetry-native open-source tracing and evaluation, plus the Arize AX enterprise platform on top. - [Braintrust](https://www.aievals.co/tools/braintrust): Framework-agnostic hosted AI observability and eval platform built around the Brainstore trace search engine. - [Comet Opik](https://www.aievals.co/tools/comet-opik): Apache 2.0 LLM evaluation and observability with 30+ judge metrics, LLM-judged test suites, a prompt optimizer, and a built-in debugging assistant. - [Confident AI / DeepEval](https://www.aievals.co/tools/confident-ai-deepeval): Pytest-style LLM unit testing in DeepEval plus the Confident AI hosted layer for dataset management and dashboards. - [Galileo](https://www.aievals.co/tools/galileo): Eval-to-guardrail platform, now owned by Cisco: Luna-2 small judge models grade offline runs and run as runtime checks. - [garak](https://www.aievals.co/tools/garak): NVIDIA's open-source LLM vulnerability scanner, reviewed: probes, detectors and buffs against a model endpoint, and what a hit rate does and does not tell you. - [Giskard](https://www.aievals.co/tools/giskard): Open-source agent testing and red-team library, rewritten as v3, plus the Giskard Hub enterprise platform with 50+ adversarial probes. - [Harbor](https://www.aievals.co/tools/harbor): The Terminal-Bench team's open-source agent-evaluation framework, reviewed: tasks as directories, sandboxed trials, 30-plus environments, and where it trails Inspect AI. - [HELM](https://www.aievals.co/tools/helm): Stanford CRFM's Holistic Evaluation of Language Models: a scenario-and-metric matrix with public leaderboards, in maintenance mode since June 2026. - [HoneyHive](https://www.aievals.co/tools/honeyhive): OpenTelemetry-native agent observability and evals with a control-plane and data-plane split for enterprise data residency. - [Inspect AI](https://www.aievals.co/tools/inspect-ai): UK AISI's open-source eval framework, reviewed: a 170-plus prebuilt eval register, real sandboxing, multi-agent support, and where it beats the hosted platforms. - [Langfuse](https://www.aievals.co/tools/langfuse): Open-source LLM engineering platform: tracing, prompts, datasets, scores, and a genuinely self-hostable cloud. - [LangSmith](https://www.aievals.co/tools/langsmith): LangChain's hosted observability and evaluation platform. The default for teams already on LangChain or LangGraph. - [LlamaIndex evals](https://www.aievals.co/tools/llamaindex-evals): First-party RAG evaluators inside LlamaIndex core, with adapters for Ragas, DeepEval, and other metric libraries. Dataset tooling was cut from core in April 2026. - [lm-evaluation-harness](https://www.aievals.co/tools/lm-evaluation-harness): EleutherAI's de facto academic harness for raw model benchmarks; the backend behind Hugging Face's now-archived Open LLM Leaderboard. - [MLflow GenAI Evaluation](https://www.aievals.co/tools/mlflow-genai): MLflow's mlflow.genai module, reviewed: evaluate(), 24 built-in scorers, make_judge(), and tracing in one Apache-2.0 platform, plus the heavy-server cost and the new Decision API route for judges. - [OpenAI Evals](https://www.aievals.co/tools/openai-evals): OpenAI's original open-source eval framework and registry, now in maintenance only, plus the hosted Evals platform that shuts down on November 30, 2026. - [Promptfoo](https://www.aievals.co/tools/promptfoo): MIT-licensed CLI and library for eval and red-team; acquired by OpenAI with the OSS license preserved. - [Pydantic Evals](https://www.aievals.co/tools/pydantic-evals): Pydantic's code-first Python eval library, reviewed: Case, Dataset and Evaluator primitives, OpenTelemetry traces per case, an optional Logfire UI, and no key needed for the basics. - [Ragas](https://www.aievals.co/tools/ragas): The RAG-eval metrics standard: faithfulness, context precision, context recall, answer relevancy, and a synthetic test-set generator. - [TruLens](https://www.aievals.co/tools/trulens): OpenTelemetry-emitting feedback functions for RAG and agent evals, stewarded by Snowflake. - [Weights and Biases Weave](https://www.aievals.co/tools/w-and-b-weave): LLM evals and monitoring inside the W&B Models stack: trace trees, leaderboards, online eval, and multimodal tracking. ## Papers > Curated reading list across ten themes, each paper annotated. - [Agentic eval](https://www.aievals.co/papers/agentic): Trajectory, outcome, and the harder benchmarks that came after AgentBench. WebArena, SWE-bench, GAIA, OSWorld and OSWorld 2.0, tau-bench, MLE-bench, AgentDojo, SWE-Lancer, BrowseComp, Cybench,... - [Foundational eval theory](https://www.aievals.co/papers/foundational): What an eval is, what a benchmark is, and why rankings flip when nothing changed. The papers that built the vocabulary. - [LLM-as-Judge](https://www.aievals.co/papers/llm-as-judge): Building, calibrating, and stress-testing model judges. From G-Eval and MT-Bench through Prometheus 2 to JudgeBench, plus 2026 work on judge noise, panel correlation, and decision-model judges. - [Long-context](https://www.aievals.co/papers/long-context): Why '1M tokens' rarely means a million tokens. Lost in the Middle, LongBench, RULER, BABILong, NoCha, Loong, NoLiMa, LongBench Pro. - [Methodology and production](https://www.aievals.co/papers/methodology-production): SPADE, RewardBench, MixEval, WildBench, IFEval, InfoBench, FollowBench, plus the 2025 to 2026 agent-eval and observability work. - [Multimodal](https://www.aievals.co/papers/multimodal): MMBench, MM-Vet, MMMU, BLINK, Vibe-Eval, Video-MME, Video-MME-v2, MMMU-Pro, ChartQA, BabyVision. - [RAG, retrieval, factuality](https://www.aievals.co/papers/rag-factuality): BEIR, FActScore, RAGAS, Self-RAG, ARES, SAFE, SimpleQA, SimpleQA Verified, HalluLens, HalluHard. - [Reasoning and math](https://www.aievals.co/papers/reasoning-and-math): From MATH and Chain-of-Thought to FrontierMath, ARC-AGI-2 and ARC-AGI-3, Humanity's Last Exam, and the faithfulness problem. - [Safety, red-team, robustness](https://www.aievals.co/papers/safety-red-team): TruthfulQA, BBQ, Constitutional AI, GCG, sycophancy, HarmBench, sandbagging, alignment faking, emergent misalignment, judge reliability. - [Tool use and function calling](https://www.aievals.co/papers/tool-use): API-Bank, Gorilla, ToolBench, BFCL, TRAJECT-Bench, MCP-Atlas, a validity audit of tool-calling benchmarks, plus the tool-injection security work in AgentDojo. ## Cookbook > Runnable worked examples. Estimated minutes per recipe. - [Your first eval in 30 minutes](https://www.aievals.co/cookbook/your-first-eval-30-minutes): Pick one feature, label fifty traces, write a binary judge, run it, read the confusion matrix. A complete walkthrough using the Anthropic Python SDK. - [LLM-judge calibration loop](https://www.aievals.co/cookbook/llm-judge-calibration): Push TPR and TNR past 0.90 by iterating the rubric, not the model. A complete loop adapted from the HuggingFace LLM-as-judge cookbook. - [RAG eval with Ragas](https://www.aievals.co/cookbook/rag-eval-with-ragas): A two-layer RAG eval (retrieval plus generation faithfulness) using Ragas on a small synthetic corpus. Runs locally with the OpenAI Python SDK. - [Agent eval with τ-bench: pass@1 vs pass^4](https://www.aievals.co/cookbook/agent-eval-with-tau-bench): Run τ-bench (tau-bench) retail tasks against your agent and report pass@1 versus pass^4: the consistency gap between a demo and a product. - [Multi-turn conversation eval](https://www.aievals.co/cookbook/multi-turn-conversation-eval): Per-turn binary rubrics plus a per-session state check, logged to Langfuse, for an assistant that has to remember context and follow rules across turns. - [SWE-bench Verified walkthrough](https://www.aievals.co/cookbook/swe-bench-verified-walkthrough): Run a repo-stratified 25-instance SWE-bench Verified subset with mini-SWE-agent, verify patches with the official Docker harness, and read the report without over-claiming. - [Measure effective context length with RULER](https://www.aievals.co/cookbook/long-context-eval-with-ruler): A 90-minute harness that finds the context length your model actually sustains: RULER's tasks as a six-task smoke suite, run from 4k to 128k tokens. - [Text-to-SQL eval with execution](https://www.aievals.co/cookbook/text-to-sql-eval-with-execution): Score SQL by executing it against a seeded SQLite warehouse and comparing result sets, with a failure taxonomy for syntax, schema, shape, and value errors. - [Online eval with Langfuse](https://www.aievals.co/cookbook/online-eval-with-langfuse): Instrument an app with Langfuse, judge a 5% sample of production traces plus every thumbs-down, and promote confirmed failures into the offline dataset. - [Online eval with Phoenix](https://www.aievals.co/cookbook/online-eval-with-phoenix): Trace a live app into self-hosted Phoenix over OpenTelemetry, judge sampled spans with the evals classifiers, and bank the failures as a regression dataset. - [TypeScript eval with Vercel AI SDK](https://www.aievals.co/cookbook/typescript-eval-with-vercel-ai-sdk): Vitest as the eval runner, AI SDK 7 generateText under test, a zod-typed binary judge, a JSON case file, and CI wiring with per-run cost accounting. - [Adding error bars to your evals](https://www.aievals.co/cookbook/adding-error-bars): Wilson intervals on pass rates, cluster-aware standard errors, and a paired McNemar test for two-system comparisons, with runnable statsmodels code. - [Contamination detection for code-gen and reasoning evals](https://www.aievals.co/cookbook/contamination-detection): Date-window your test set against each model's training cutoff, then catch verbatim and paraphrased leaks with MinHash overlap against public mirrors. - [HarmBench red-team run](https://www.aievals.co/cookbook/harmbench-red-team): Run 100 HarmBench behaviors and all 450 XSTest prompts against your model, then report attack success rate and over-refusal rate as one table. - [Synthetic query generation for golden sets](https://www.aievals.co/cookbook/synthetic-query-generation): Build a RAG eval set from your own corpus: seeded chunk sampling, multi-hop and unanswerable query synthesis, MinHash dedup, and a pinned manifest you can diff. - [Fairness audit with Fairlearn](https://www.aievals.co/cookbook/fairness-audit-with-fairlearn): A runnable audit on the adult income dataset: disaggregate with MetricFrame, read the gaps with group sizes, mitigate with ThresholdOptimizer, and report before and after honestly. - [Model explanations with SHAP](https://www.aievals.co/cookbook/model-explanations-with-shap): Train a classifier with a planted leaky feature, catch it with TreeExplainer global importance, read local explanations for individual errors, and port the workflow to Captum. - [Agent injection eval with AgentDojo](https://www.aievals.co/cookbook/agent-injection-eval-with-agentdojo): Measure whether a tool-using agent resists indirect prompt injection: run AgentDojo's banking suite clean and under attack, compute utility and attack success rate together, then add a defense and... ## Task types > Twenty-six task-specific eval playbooks, one per task shape. - [Autonomous coding](https://www.aievals.co/task-types/autonomous-coding): Given an issue and a repo, autonomously edit files until hidden tests pass. Execution-grounded scoring is the bar, Verified is saturated, SWE-bench Pro is harder but has broken tasks of its own, and... - [Long-horizon planning and agentic loops](https://www.aievals.co/task-types/long-horizon-planning): Multi-step decomposition, replanning, recovery. The agent works for minutes. Pass^k on real workflows is the metric that surfaces consistency, which pass@1 alone hides. - [Reasoning depth and thinking-mode analysis](https://www.aievals.co/task-types/reasoning-depth): Evaluate the thinking traces of reasoning models. Correctness on hard reasoning plus chain-of-thought faithfulness, not just answer accuracy. - [Long-context RAG](https://www.aievals.co/task-types/long-context-rag): Answer over very long contexts: codebases, contract bundles, transcripts. RULER plus answer faithfulness, not vanilla needle-in-a-haystack. - [Native multimodal](https://www.aievals.co/task-types/native-multimodal): Inputs cross modalities: image plus text Q and A, voice to text to image, screen plus instruction. Task accuracy plus cross-modal grounding, not single-modality fallback. - [Tool use and function calling](https://www.aievals.co/task-types/tool-use-function-calling): Choose the right function, fill correct arguments, handle errors, chain calls. AST-match plus execution-grounded success on BFCL v4 is the bar. - [Computer and browser-use agents](https://www.aievals.co/task-types/computer-browser-use): Agent drives a real OS or browser via screenshots and keyboard or mouse. OSWorld 2.0 and WebArena are the public benchmarks; step budgets and partial credit are the 2026 methodology. - [Structured output and JSON mode](https://www.aievals.co/task-types/structured-output-json): Emit valid JSON conforming to a schema with correct types and values. Schema validity times semantic correctness, not validity alone. - [Text-to-SQL](https://www.aievals.co/task-types/text-to-sql): Natural-language question to SQL query to correct rows. Execution accuracy on BIRD-bench plus efficiency, not exact-match scoring. - [Conversational quality](https://www.aievals.co/task-types/conversational-quality): Multi-turn assistant. Stay on persona, remember context, recover from misunderstandings. Pairwise preference plus per-turn rubric, not 1-to-5 scales. - [Summarization](https://www.aievals.co/task-types/summarization): Compress source documents into a faithful summary. Faithfulness plus coverage, not ROUGE. - [Translation](https://www.aievals.co/task-types/translation): Source language to target language. COMET and chrF for automatic scoring, MQM or ESA for human; not BLEU alone in 2026. - [Code review](https://www.aievals.co/task-types/code-review): Model reads a diff and flags real issues with actionable comments. Precision plus recall of seeded bugs, not recall alone (which produces nit machines). - [Document understanding and extraction](https://www.aievals.co/task-types/document-understanding): Extract structured info or answer questions from documents: PDFs, scans, forms. Field-level F1 and ANLS, not holistic JSON-looks-right scoring. - [Voice. STT, TTS, conversational](https://www.aievals.co/task-types/voice): STT (audio to text), TTS (text to audio), and full-duplex conversational voice. WER, MOS, TTFB; not WER on clean audio alone. - [Image generation](https://www.aievals.co/task-types/image-generation): Text prompt to image. Human preference plus CLIP score plus FID; not FID alone (correlates poorly with text-image alignment). - [Embeddings and retrieval quality](https://www.aievals.co/task-types/embeddings-retrieval): Embed queries and docs, retrieve top-K relevant. NDCG@10 plus Recall@K on a domain set; MTEB rankings don't predict domain ranking. - [Recommendation and personalization](https://www.aievals.co/task-types/recommendation): Rank items for a user. Offline NDCG plus hit-rate to filter; online A/B for ground truth; temporal splits, never random. - [Classification](https://www.aievals.co/task-types/classification): Assign one or more labels per input. Macro-F1 plus calibration on a stratified set, not accuracy on imbalanced data. - [Safety, refusal, and jailbreak resistance](https://www.aievals.co/task-types/safety-jailbreak): Refuse harmful requests; resist adversarial prompts; don't over-refuse. Attack success rate plus over-refusal rate, never one alone. - [Deep research agents](https://www.aievals.co/task-types/deep-research-agents): How to evaluate deep research agents: RACE rubric scoring, FACT citation precision, BrowseComp and GAIA, judge pitfalls, and a day-1 starter. - [Spec generation agents](https://www.aievals.co/task-types/spec-generation-agents): Evaluating spec and design-doc generation: SpecBench, ambiguity detection, EARS acceptance criteria, and spec2code downstream success. - [GraphRAG](https://www.aievals.co/task-types/graphrag): Evaluating GraphRAG: local vs global query classes, counterbalanced pairwise judging, entity-coverage ceilings, and token-cost accounting. - [Multi-agent systems](https://www.aievals.co/task-types/multi-agent-systems): Evaluating multi-agent LLM systems: milestone KPIs, the MAST 14-failure-mode taxonomy, orchestrator vs worker splits, and cost explosion. - [MCP and tool-registry evals](https://www.aievals.co/task-types/mcp-tool-evals): Evaluating agents on MCP servers: MCP-Universe, tool-routing failures at registry scale, schema-drift CI checks, and AgentDojo injection probes. - [Agent memory](https://www.aievals.co/task-types/agent-memory): Evaluating agent memory: the five LongMemEval abilities, contradiction and deletion probes, pass^k reliability, and the missing privacy benchmark. ## Start Here > Linear numbered reading paths, one track per persona. - [Start Here for Backend Engineers](https://www.aievals.co/start/backend-engineer): Ten steps from why evals matter through release gates. Trace logging from services, structured output, tool calls, execution-checked SQL, online eval, CI wiring, latency and cost budgets. - [Start Here for CTOs and Directors of Technology](https://www.aievals.co/start/cto): Ten steps from the executive case for evals through a board readout. Risk posture, regulatory mapping, frontier-lab benchmarks. - [Start Here for Senior Data Engineers](https://www.aievals.co/start/data-engineer): Nine steps from trace schema through drift detection. JSONL and Parquet schemas, sampling policies, dataset versioning and lineage. - [Start Here for Senior Engineering Managers](https://www.aievals.co/start/eng-manager): Eight steps from the maturity model through the skill bar for eval-literate engineers. Strategy first, tables and decision flowcharts, light on code. - [Start Here for Frontend Engineers](https://www.aievals.co/start/frontend-engineer): Nine steps from why evals matter through online monitoring. TypeScript-first tooling, judge basics, a trace viewer you build, multi-turn and voice quality, client-side instrumentation. - [AI evals for engineers: the 7-step starting path](https://www.aievals.co/start/principal-ai-engineer): Go from zero to a running eval suite in seven readings: your first eval in 30 minutes, judge calibration, RAG metrics, and a tool-using agent eval. - [Start Here for Senior Product Managers](https://www.aievals.co/start/product-manager): Nine steps from the AI quality bar through a foundational reading list. Product-thinking, customer-friendly vocabulary, light on math. - [Start Here for AI Research Engineers and Research Scientists](https://www.aievals.co/start/research-engineer): Nine steps from the agent-eval survey through publishing eval datasets responsibly. Citation-heavy, ablation tables, reproducibility checklists. - [Start Here for SDET and QA Engineers](https://www.aievals.co/start/sdet-qa): Ten steps from anatomy of an AI CI pipeline to release gates. Test-engineering pragmatic, config snippets, pytest and Vitest patterns. ## Canonical citations > Primary sources cited across the site. Grouped by kind so answer engines can match a citation to a content shape. ### Practitioner blog posts > Long-form posts by working AI practitioners (Hamel Husain, Eugene Yan, Chip Huyen, Shreya Shankar, et al). The opinions backing most of our recommendations. - [Your AI Product Needs Evals by Hamel Husain](https://hamel.dev/blog/posts/evals/) - [Creating an LLM-as-a-Judge That Drives Business Results by Hamel Husain](https://hamel.dev/blog/posts/llm-judge/) - [A Field Guide to Rapidly Improving AI Products by Hamel Husain](https://hamel.dev/blog/posts/field-guide/) - [AI Evals: Everything You Need to Know (formerly LLM Evals FAQ) by Hamel Husain & Shreya Shankar](https://hamel.dev/blog/posts/evals-faq/) - [The Revenge of the Data Scientist by Hamel Husain](https://hamel.dev/blog/posts/revenge/) - [Fuck You, Show Me The Prompt by Hamel Husain](https://hamel.dev/blog/posts/prompt/) - [Patterns for Building LLM-based Systems & Products by Eugene Yan](https://eugeneyan.com/writing/llm-patterns/) - [LLM Evaluators by Eugene Yan](https://eugeneyan.com/writing/llm-evaluators/) - [Evaluation & Hallucination Detection for Abstractive Summaries by Eugene Yan](https://eugeneyan.com/writing/evals/) - [Abstractive Summary eval by Eugene Yan](https://eugeneyan.com/writing/abstractive/) - [OOD Finetuning to Bootstrap Hallucination Detection by Eugene Yan](https://eugeneyan.com/writing/finetuning/) - [The AI Engineering Flywheel by Shreya Shankar](https://www.sh-reya.com/blog/ai-engineering-flywheel/) - [Building a Generative AI Platform by Chip Huyen](https://huyenchip.com/2024/07/25/genai-platform.html) - [900 Most Popular Open Source AI Tools by Chip Huyen](https://huyenchip.com/2024/03/14/ai-oss.html) - [Anthropic achieves ISO 42001 certification for responsible AI](https://www.anthropic.com/news/anthropic-achieves-iso-42001-certification-for-responsible-ai) - [Measuring political bias in Claude by Anthropic](https://www.anthropic.com/news/political-even-handedness) - [Introducing LangSmith Engine by LangChain](https://langchain.com/blog/introducing-langsmith-engine) - [Introducing LangSmith Tuned Evaluators, starting with Perceived Error by LangChain](https://langchain.com/blog/introducing-langsmith-tuned-evaluators-starting-with-perceived-error) - [Challenges in Evaluating AI Systems by Anthropic](https://www.anthropic.com/news/evaluating-ai-systems) - [Introducing Computer Use by Anthropic](https://www.anthropic.com/news/3-5-models-and-computer-use) - [Introducing Contextual Retrieval by Anthropic](https://www.anthropic.com/news/contextual-retrieval) - [Introducing Claude Opus 4.6 by Anthropic](https://www.anthropic.com/news/claude-opus-4-6) - [Anthropic's Responsible Scaling Policy: Version 3.0](https://www.anthropic.com/news/responsible-scaling-policy-v3) - [Chatbot Arena Benchmarking Blog by LMSYS](https://lmsys.org/blog/2023-05-03-arena/) - [Chatbot Arena BT analysis by LMSYS](https://lmsys.org/blog/2023-12-07-leaderboard/) - [Aligning LLM-as-Judge w/ Human Preferences by LangChain](https://www.langchain.com/blog/aligning-llm-as-a-judge-with-human-preferences) ### Papers > Academic and lab-published papers on eval methodology, benchmarks, and judge calibration. - [MMLU (2020) by Hendrycks et al](https://arxiv.org/abs/2009.03300) - [Beyond Accuracy: CheckList (2020) by Ribeiro et al](https://arxiv.org/abs/2005.04118) - [BIG-Bench (2022) by Srivastava et al](https://arxiv.org/abs/2206.04615) - [BBH (2022) by Suzgun et al](https://arxiv.org/abs/2210.09261) - [HELM (2022) by Liang et al](https://arxiv.org/abs/2211.09110) - [The Benchmark Lottery (2021) by Dehghani, Tay et al](https://arxiv.org/abs/2107.07002) - [Stop Uploading Test Data in Plain Text (2023) by Jacovi et al](https://arxiv.org/abs/2305.10160) - [Are We Done with MMLU? (2024) by Gema et al](https://arxiv.org/abs/2406.04127) - [Adding Error Bars to Evals (2024) by Evan Miller](https://arxiv.org/abs/2411.00640) - [G-Eval (2023) by Liu et al](https://arxiv.org/abs/2303.16634) - [Judging LLM-as-a-Judge with MT-Bench (2023) by Zheng et al](https://arxiv.org/abs/2306.05685) - [Prometheus (2023) by Kim et al](https://arxiv.org/abs/2310.08491) - [Auto-J (2023) by Li et al](https://arxiv.org/abs/2310.05470) - [JudgeLM (2023) by Zhu et al](https://arxiv.org/abs/2310.17631) - [Who Validates the Validators? (2024) by Shankar et al](https://arxiv.org/abs/2404.12272) - [Prometheus 2 (2024) by Kim et al](https://arxiv.org/abs/2405.01535) - [Judging the Judges (2024) by Thakur et al](https://arxiv.org/abs/2406.12624) - [JudgeBench (2024) by Tan et al](https://arxiv.org/abs/2410.12784) - [Chatbot Arena (2024) by Chiang et al](https://arxiv.org/abs/2403.04132) - [Arena Hard / BenchBuilder (2024) by Li et al](https://arxiv.org/abs/2406.11939) - [LLM Comparator (2024) by Kahng et al](https://arxiv.org/abs/2402.10524) - [Length-Controlled AlpacaEval (2024) by Dubois et al](https://arxiv.org/abs/2404.04475) - [PandaLM (2023) by Wang et al](https://arxiv.org/abs/2306.05087) - [Luna (Galileo) (2024) by Belyi et al](https://arxiv.org/abs/2406.00975) - [Lynx (Patronus AI) (2024) by Ravi et al](https://arxiv.org/abs/2407.08488) - [AgentBench (2023) by Liu et al](https://arxiv.org/abs/2308.03688) - [WebArena (2023) by Zhou et al](https://arxiv.org/abs/2307.13854) - [VisualWebArena (2024) by Koh et al](https://arxiv.org/abs/2401.13649) - [WorkArena (2024) by Drouin et al](https://arxiv.org/abs/2403.07718) - [SWE-bench (2023) by Jimenez et al](https://arxiv.org/abs/2310.06770) - [SWE-bench Multimodal (2024) by Yang et al](https://arxiv.org/abs/2410.03859) - [GAIA (2023) by Mialon et al](https://arxiv.org/abs/2311.12983) - [AgentBoard (2024) by Ma et al](https://arxiv.org/abs/2401.13178) - [OSWorld (2024) by Xie et al](https://arxiv.org/abs/2404.07972) - [τ-bench (2024) by Yao et al](https://arxiv.org/abs/2406.12045) - [MLE-bench (2024) by Chan et al](https://arxiv.org/abs/2410.07095) - [ScienceAgentBench (2024) by Chen et al](https://arxiv.org/abs/2410.05080) - [AgentDojo (2024) by Debenedetti et al](https://arxiv.org/abs/2406.13352) - [SWE-Lancer (2025) by Miserendino et al](https://arxiv.org/abs/2502.12115) - [BrowseComp (2025) by Wei et al](https://arxiv.org/abs/2504.12516) - [Cybench (2024) by Zhang et al](https://arxiv.org/abs/2408.08926) - [LLM Agent Eval Survey (2025) by Yehudai et al](https://arxiv.org/abs/2503.16416) - [Agent Eval Survey (2025) by Mohammadi et al](https://arxiv.org/abs/2507.21504) - [Agent-as-a-Judge (2024) by Zhuge et al](https://arxiv.org/abs/2410.10934) - [TRAJECT-Bench (2025) by He et al](https://arxiv.org/abs/2510.04550) - [Beyond Black-Box Benchmarking (2025) by Moshkovich et al](https://arxiv.org/abs/2503.06745) - [In-Context Reward Hacking (2024) by Pan et al](https://arxiv.org/abs/2402.06627) - [Concrete Problems in AI Safety (2016) by Amodei et al](https://arxiv.org/abs/1606.06565) - [AI Sandbagging (2024) by van der Weij et al](https://arxiv.org/abs/2406.07358) - [Alignment Faking (2024) by Greenblatt et al](https://arxiv.org/abs/2412.14093) - [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents (2025) by Du et al](https://arxiv.org/abs/2506.11763) - [Why Do Multi-Agent LLM Systems Fail? (2025) by Cemri et al](https://arxiv.org/abs/2503.13657) - [MultiAgentBench: Evaluating the Collaboration and Competition of LLM Agents (2025) by Zhu et al](https://arxiv.org/abs/2503.01935) - [Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks (2024) by Fourney et al](https://arxiv.org/abs/2411.04468) - [MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers (2025) by Luo et al](https://arxiv.org/abs/2508.14704) - [LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools? (2025) by Mo et al](https://arxiv.org/abs/2508.01780) - [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (2024) by Wu et al](https://arxiv.org/abs/2410.10813) - [Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (2025) by Hu et al](https://arxiv.org/abs/2507.05257) - [Measuring AI Ability to Complete Long Software Tasks (METR) (2025) by Kwa et al](https://arxiv.org/abs/2503.14499) - [Lost in the Middle (2023) by Liu et al](https://arxiv.org/abs/2307.03172) - [LongBench (2023) by Bai et al](https://arxiv.org/abs/2308.14508) - [∞Bench (2024) by Zhang et al](https://arxiv.org/abs/2402.13718) - [RULER (2024) by Hsieh et al](https://arxiv.org/abs/2404.06654) - [BABILong (2024) by Kuratov et al](https://arxiv.org/abs/2406.10149) - [One Thousand and One Pairs (2024) by Karpinska et al](https://arxiv.org/abs/2406.16264) - [Loong (2024) by Wang et al](https://arxiv.org/abs/2406.17419) - [LongBench v2 (2024) by Bai et al](https://arxiv.org/abs/2412.15204) - [NoLiMa (2025) by Modarressi et al](https://arxiv.org/abs/2502.05167) - [On the Measure of Intelligence (2019) by Chollet](https://arxiv.org/abs/1911.01547) - [MATH (2021) by Hendrycks et al](https://arxiv.org/abs/2103.03874) - [Chain-of-Thought (2022) by Wei et al](https://arxiv.org/abs/2201.11903) - [Self-Consistency (2022) by Wang et al](https://arxiv.org/abs/2203.11171) - [Let's Verify Step by Step (2023) by Lightman et al](https://arxiv.org/abs/2305.20050) - [GPQA (2023) by Rein et al](https://arxiv.org/abs/2311.12022) - [Skill-Mix (2023) by Yu et al](https://arxiv.org/abs/2310.17567) - [FrontierMath (2024) by Glazer et al](https://arxiv.org/abs/2411.04872) - [ARC-AGI-2 (2025) by Chollet et al](https://arxiv.org/abs/2505.11831) - [Humanity's Last Exam (2025) by Phan et al](https://arxiv.org/abs/2501.14249) - [TruthfulQA (2021) by Lin et al](https://arxiv.org/abs/2109.07958) - [BBQ (2021) by Parrish et al](https://arxiv.org/abs/2110.08193) - [Red Teaming Language Models (2022) by Ganguli et al](https://arxiv.org/abs/2209.07858) - [Constitutional AI (2022) by Bai et al](https://arxiv.org/abs/2212.08073) - [Universal Adversarial Attacks (2023) by Zou et al](https://arxiv.org/abs/2307.15043) - [Towards Understanding Sycophancy (2023) by Sharma et al](https://arxiv.org/abs/2310.13548) - [Quantifying LM Sensitivity to Spurious Features (2023) by Sclar et al](https://arxiv.org/abs/2310.11324) - [HarmBench (2024) by Mazeika et al](https://arxiv.org/abs/2402.04249) - [BOLD (2021) by Dhamala et al](https://arxiv.org/abs/2101.11718) - [HaluEval (2023) by Li et al](https://arxiv.org/abs/2305.11747) - [XSTest: Exaggerated Safety Behaviours (2023) by Rottger et al](https://arxiv.org/abs/2308.01263) - [MCPTox: A Benchmark for Tool Poisoning on Real-World MCP Servers (2025) by Wang et al](https://arxiv.org/abs/2508.14925) - [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023) by Greshake et al](https://arxiv.org/abs/2302.12173) - [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated LLM Agents (2024) by Zhan et al](https://arxiv.org/abs/2403.02691) - [Benchmarking and Defending Against Indirect Prompt Injection Attacks on LLMs (BIPIA) (2023) by Yi et al](https://arxiv.org/abs/2312.14197) - [WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks (2025) by Evtimov et al](https://arxiv.org/abs/2504.18575) - [AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases (2024) by Chen et al](https://arxiv.org/abs/2407.12784) - [The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses against LLM Jailbreaks and Prompt Injections (2025) by Nasr et al](https://arxiv.org/abs/2510.09023) - [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024) by Wallace et al](https://arxiv.org/abs/2404.13208) - [Defending Against Indirect Prompt Injection Attacks With Spotlighting (2024) by Hines et al](https://arxiv.org/abs/2403.14720) - [LlamaFirewall: An open source guardrail system for building secure AI agents (2025) by Chennabasappa et al](https://arxiv.org/abs/2505.03574) - [Defeating Prompt Injections by Design (CaMeL) (2025) by Debenedetti et al](https://arxiv.org/abs/2503.18813) - [CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for LLMs (2024) by Bhatt et al](https://arxiv.org/abs/2404.13161) - [Equality of Opportunity in Supervised Learning (2016) by Hardt et al](https://arxiv.org/abs/1610.02413) - [Inherent Trade-Offs in the Fair Determination of Risk Scores (2016) by Kleinberg et al](https://arxiv.org/abs/1609.05807) - [AI Fairness 360 (2018) by Bellamy et al](https://arxiv.org/abs/1810.01943) - [Why Should I Trust You: Explaining the Predictions of Any Classifier (2016) by Ribeiro et al](https://arxiv.org/abs/1602.04938) - [A Unified Approach to Interpreting Model Predictions (2017) by Lundberg & Lee](https://arxiv.org/abs/1705.07874) - [Axiomatic Attribution for Deep Networks (2017) by Sundararajan et al](https://arxiv.org/abs/1703.01365) - [Sanity Checks for Saliency Maps (2018) by Adebayo et al](https://arxiv.org/abs/1810.03292) - [RealToxicityPrompts (2020) by Gehman et al](https://arxiv.org/abs/2009.11462) - [Datasheets for Datasets (2018) by Gebru et al](https://arxiv.org/abs/1803.09010) - [Model Cards for Model Reporting (2018) by Mitchell et al](https://arxiv.org/abs/1810.03993) - [Poisoning Web-Scale Training Datasets is Practical (2023) by Carlini et al](https://arxiv.org/abs/2302.10149) - [OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks (2026) by Yuan et al](https://arxiv.org/abs/2606.29537) - [Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (2025) by Gou et al](https://arxiv.org/abs/2506.21506) - [TheAgentCompany (2024) by Xu et al](https://arxiv.org/abs/2412.14161) - [DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report (2026) by Li et al](https://arxiv.org/abs/2601.08536) - [DRACO (2026) by Perplexity AI](https://arxiv.org/abs/2602.11685) - [Search-Time Contamination in Deep Research Agents (2026)](https://arxiv.org/abs/2606.05241) - [AI Agents That Matter (2024) by Kapoor et al](https://arxiv.org/abs/2407.01502) - [Rubrics as Rewards (2025) by Gunjal et al](https://arxiv.org/abs/2507.17746) - [Reinforcement Learning with Rubric Anchors (2025) by Huang et al](https://arxiv.org/abs/2508.12790) - [OpenRubrics (ACL 2026) (2025) by Liu et al](https://arxiv.org/abs/2510.07743) - [EvoRubrics (2026) by Ding et al](https://arxiv.org/abs/2606.23038) - [Reward Hacking in Rubric-Based Reinforcement Learning (2026) by Mahmoud et al](https://arxiv.org/abs/2605.12474) - [HealthBench (OpenAI) (2025) by Arora et al](https://arxiv.org/abs/2505.08775) - [Rubrics Across the Evolving LLM Landscape (survey) (2026) by Chen et al](https://arxiv.org/abs/2606.08625) - [BEIR (2021) by Thakur et al](https://arxiv.org/abs/2104.08663) - [FActScore (2023) by Min et al](https://arxiv.org/abs/2305.14251) - [RAGAS (2023) by Es et al](https://arxiv.org/abs/2309.15217) - [Self-RAG (2023) by Asai et al](https://arxiv.org/abs/2310.11511) - [ARES (2023) by Saad-Falcon et al](https://arxiv.org/abs/2311.09476) - [SAFE (2024) by Wei et al](https://arxiv.org/abs/2403.18802) - [SimpleQA (2024) by Wei et al](https://arxiv.org/abs/2411.04368) - [HalluLens (2025) by Bang et al](https://arxiv.org/abs/2504.17550) - [SummEval: Re-evaluating Summarization Evaluation (2020) by Fabbri et al](https://arxiv.org/abs/2007.12626) - [Know What You Don't Know: Unanswerable Questions for SQuAD (2018) by Rajpurkar et al](https://arxiv.org/abs/1806.03822) - [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (2018) by Yang et al](https://arxiv.org/abs/1809.09600) - [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models (2024) by Niu et al](https://arxiv.org/abs/2401.00396) - [From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024) by Edge et al](https://arxiv.org/abs/2404.16130) - [RAG vs. GraphRAG: A Systematic Evaluation and Key Insights (2025) by Han et al](https://arxiv.org/abs/2502.11371) - [MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries (2024) by Tang et al](https://arxiv.org/abs/2401.15391) - [MMBench (2023) by Liu et al](https://arxiv.org/abs/2307.06281) - [MM-Vet (2023) by Yu et al](https://arxiv.org/abs/2308.02490) - [MMMU (2023) by Yue et al](https://arxiv.org/abs/2311.16502) - [BLINK (2024) by Fu et al](https://arxiv.org/abs/2404.12390) - [Vibe-Eval (2024) by Padlewski et al](https://arxiv.org/abs/2405.02287) - [Video-MME (2024) by Fu et al](https://arxiv.org/abs/2405.21075) - [MMMU-Pro (2024) by Yue et al](https://arxiv.org/abs/2409.02813) - [ChartQA (2022) by Masry et al](https://arxiv.org/abs/2203.10244) - [DocVQA (2020) by Mathew et al](https://arxiv.org/abs/2007.00398) - [API-Bank (2023) by Li et al](https://arxiv.org/abs/2304.08244) - [Gorilla (2023) by Patil et al](https://arxiv.org/abs/2305.15334) - [ToolLLM/ToolBench (2023) by Qin et al](https://arxiv.org/abs/2307.16789) - [SPADE (2024) by Shankar et al](https://arxiv.org/abs/2401.03038) - [RewardBench (2024) by Lambert et al](https://arxiv.org/abs/2403.13787) - [MixEval (2024) by Ni et al](https://arxiv.org/abs/2406.06565) - [WildBench (2024) by Lin et al](https://arxiv.org/abs/2406.04770) - [InfoBench (2024) by Qin et al](https://arxiv.org/abs/2401.03601) - [IFEval (2023) by Zhou et al](https://arxiv.org/abs/2311.07911) - [FollowBench (2023) by Jiang et al](https://arxiv.org/abs/2310.20410) - [Data-Centric AI for LLMs (2024) by Xu et al](https://arxiv.org/abs/2406.14473) - [COMET: A Neural Framework for MT Evaluation (2020) by Rei et al](https://arxiv.org/abs/2009.09025) - [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task (2022) by Rei et al](https://arxiv.org/abs/2209.06243) - [Are LLMs Breaking MT Metrics? Results of the WMT24 Metrics Shared Task by Freitag et al](https://aclanthology.org/2024.wmt-1.2/) - [HumanEval (2021) by Chen et al](https://arxiv.org/abs/2107.03374) - [MBPP (2021) by Austin et al](https://arxiv.org/abs/2108.07732) - [APPS (2021) by Hendrycks et al](https://arxiv.org/abs/2105.09938) - [HumanEval+/MBPP+ (2023) by Liu et al](https://arxiv.org/abs/2305.01210) - [LiveCodeBench (2024) by Jain et al](https://arxiv.org/abs/2403.07974) - [BigCodeBench (2024) by Zhuo et al](https://arxiv.org/abs/2406.15877) - [RepoBench (2023) by Liu, Xu, McAuley](https://arxiv.org/abs/2306.03091) - [CRUXEval (2024) by Gu et al](https://arxiv.org/abs/2401.03065) - [ClassEval (2023) by Du et al](https://arxiv.org/abs/2308.01861) - [Multi-SWE-bench (2025) by Zan et al](https://arxiv.org/abs/2504.02605) - [BIRD: Big Bench for Large-Scale Database Grounded Text-to-SQLs (2023) by Li et al](https://arxiv.org/abs/2305.03111) - [Semantic Evaluation for Text-to-SQL with Distilled Test Suites (2020) by Zhong, Yu, Klein](https://arxiv.org/abs/2010.02840) - [SpecBench: Evaluating Specification-Level Reasoning for Software Engineering LLM Agents (2026) by Hamblin et al](https://arxiv.org/abs/2605.30314) - [SpecEval: Evaluating Code Comprehension in Large Language Models via Program Specifications (2024) by Ma et al](https://arxiv.org/abs/2409.12866) - [Using LLMs in Software Requirements Specifications: An Empirical Evaluation (2024) by Krishna et al](https://arxiv.org/abs/2404.17842) - [Commit0: Library Generation from Scratch (2024) by Zhao et al](https://arxiv.org/abs/2412.01769) - [CLIP (2021) by Radford et al](https://arxiv.org/abs/2103.00020) - [FID (Two Time-Scale Update Rule) (2017) by Heusel et al](https://arxiv.org/abs/1706.08500) - [SimCSE (2021) by Gao, Yao, Chen](https://arxiv.org/abs/2104.08821) - [Robust Speech Recognition via Large-Scale Weak Supervision (2022) by Radford et al](https://arxiv.org/abs/2212.04356) - [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning (2025) by Lakshya A Agrawal et al](https://arxiv.org/abs/2507.19457) - [When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation (2026) by Mubashara Akhtar et al](https://arxiv.org/abs/2602.16763) - [Atla Selene Mini: A General Purpose Evaluation Model (2025) by Andrei Alexandru et al](https://arxiv.org/abs/2501.17195) - [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation (2025) by Baker et al](https://arxiv.org/abs/2503.11926) - [τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025) by Victor Barres, Honghua Dong, Soham Ray, Xujie Si, Karthik Narasimhan](https://arxiv.org/abs/2506.07982) - [Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints (2025) by Bowyer, S., Aitchison, L., Ivanova, D.R](https://arxiv.org/abs/2503.01747) - [Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints (2025) by Bowyer, S., Aitchison, L., Ivanova, D.R](https://arxiv.org/abs/2503.01747) - [evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations (2026) by Shreyas K Chandrahas](https://arxiv.org/abs/2607.04429) - [BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent (2025) by Chen et al](https://arxiv.org/abs/2508.06600) - [LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark (2026) by Ziyang Chen, Xing Wu, Junlong Jia, Chaochen Gao, Qi Fu, Debing Zhang, Songlin Hu](https://arxiv.org/abs/2601.02872) - [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech (2022) by Alexis Conneau et al](https://arxiv.org/abs/2205.12446) - [SABER: Small Actions, Big Errors -- Safeguarding Mutating Steps in LLM Agents (2025) by Alejandro Cuadron, Pengfei Yu, Yang Liu, Arpit Gupta](https://arxiv.org/abs/2512.07850) - [OR-Bench: An Over-Refusal Benchmark for Large Language Models (2024) by Cui, J., Chiang, W.-L., Stoica, I., Hsieh, C.-J](https://arxiv.org/abs/2405.20947) - [Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks (2026) by Cunningham, H., Wei, J., Wang, Z., Persic, A., et al](https://arxiv.org/abs/2601.04603) - [Toxicity in ChatGPT: Analyzing Persona-assigned Language Models (2023) by Ameet Deshpande et al](https://arxiv.org/abs/2304.05335) - [WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects (2025) by Daniel Deutsch et al](https://arxiv.org/abs/2502.12404) - [MMTEB: Massive Multilingual Text Embedding Benchmark (2025) by Kenneth Enevoldsen et al](https://arxiv.org/abs/2502.13595) - [Generating Diverse Q&A Benchmarks for RAG Evaluation with DataMorgana (2025) by Simone Filice et al](https://arxiv.org/abs/2501.12789) - [ARE: Scaling Up Agent Environments and Evaluations (2025) by Romain Froger, Pierre Andrews, Matteo Bettini, et al., Meta](https://arxiv.org/abs/2509.17158) - [ARE: Scaling Up Agent Environments and Evaluations (2025) by Froger, R.; Andrews, P.; Bettini, M.; et al.; Meta Superintelligence Labs](https://arxiv.org/abs/2509.17158) - [How Reliable is Multilingual LLM-as-a-Judge? (2025) by Xiyan Fu, Wei Liu](https://arxiv.org/abs/2505.12201) - [GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents (2026) by Garre, Ritchie, Mehta, Chen](https://arxiv.org/abs/2607.11192) - [Chain-of-Thought Faithfulness of Reasoning Models Varies with Where and How Preference Cues Are Delivered (2026) by Gema, Rajani, Saxena, Kwan, Minervini](https://arxiv.org/abs/2608.29464) - [JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models (2025) by Saibo Geng et al](https://arxiv.org/abs/2501.10868) - [AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons (2025) by Ghosh et al](https://arxiv.org/abs/2503.05731) - [Assessing the Downstream Utility of Evidence-Aware Retrieval in RAG (2026) by Utshab Kumar Ghosh, Debayan Mukhopadhyay, Shubham Chatterjee](https://arxiv.org/abs/2608.26379) - [Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction (2025) by Advait Gosai et al](https://arxiv.org/abs/2512.14865) - [StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models (2024) by Zhicheng Guo et al](https://arxiv.org/abs/2403.07714) - [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders (2025) by Danil Gusak, Anna Volodkevich, Anton Klenitskiy, Alexey Vasilev, Evgeny Frolov](https://arxiv.org/abs/2507.16289) - [SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge (2025) by Haas, Yona, D'Antonio, Goldshtein, Das](https://arxiv.org/abs/2509.07968) - [MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks (2026) by Zexue He et al](https://arxiv.org/abs/2602.16313) - [Measuring Sycophancy of Language Models in Multi-turn Dialogues (2025) by Hong, J., Byun, G., Kim, S., Shu, K., Choi, J.D](https://arxiv.org/abs/2505.23840) - [Best-of-N Jailbreaking (2024) by Hughes, J., Price, S., Lynch, A., Schaeffer, R., et al](https://arxiv.org/abs/2412.03556) - [The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input (2025) by Jacovi, Wang, Alberti, et al](https://arxiv.org/abs/2501.03200) - [A Critical Study on Data Leakage in Recommender System Offline Evaluation (2020) by Yitong Ji, Aixin Sun, Jie Zhang, Chenliang Li](https://arxiv.org/abs/2010.11060) - [Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation (2026) by Jiajun Jiang, Sharon Zheng, Natan Vidra, Spurthi Setty](https://arxiv.org/abs/2608.14711) - [GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation (2025) by Kamath, Chang, Krishna, Zettlemoyer, Hu, Ghazvininejad](https://arxiv.org/abs/2512.16853) - [FABLES: Evaluating faithfulness and content selection in book-length summarization (2024) by Yekyung Kim et al](https://arxiv.org/abs/2404.01261) - [Towards a Science of Scaling Agent Systems (2025) by Yubin Kim et al](https://arxiv.org/abs/2512.08296) - [Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets (WMT 2025 (Tenth Conference on Machine Translation), Suzhou) (2025) by Tom Kocmi et al](https://aclanthology.org/2025.wmt-1.22/) - [LLMs Get Lost In Multi-Turn Conversation (2025) by Laban, Hayashi, Zhou, Neville](https://arxiv.org/abs/2505.06120) - [Findings of the WMT25 Shared Task on Automated Translation Evaluation Systems: Linguistic Diversity is Challenging and References Still Help (WMT 2025 (ACL Anthology 2025.wmt-1.24)) (2025) by Alon Lavie et al](https://aclanthology.org/2025.wmt-1.24/) - [Findings of the WMT25 Shared Task on Automated Translation Evaluation Systems: Linguistic Diversity is Challenging and References Still Help (Proceedings of the Tenth Conference on Machine Translation (WMT 2025)) (2025) by Alon Lavie, Greg Hanneman, Sweta Agrawal, Diptesh Kanojia, et al](https://aclanthology.org/2025.wmt-1.24/) - [Holistic Evaluation of Text-To-Image Models (2023) by Lee, Yasunaga, Meng, et al](https://arxiv.org/abs/2311.04287) - [How to Correctly Report LLM-as-a-Judge Evaluations (2025) by Chungpa Lee, Thomas Zeng, Jongwon Jeong, Jy-yong Sohn, Kangwook Lee](https://arxiv.org/abs/2511.21140) - [Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (2024) by Fangyu Lei et al](https://arxiv.org/abs/2411.07763) - [JEV-as-a-Judge: Accept When Confident, Escalate When Unsure (2026) by Yubo Li, Yidi Miao, Ramayya Krishnan, Rema Padman (Carnegie Mellon University)](https://arxiv.org/abs/2609.26550) - [LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet (2024) by Nathaniel Li et al](https://arxiv.org/abs/2408.15221) - [The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution (2025) by Junlong Li et al](https://arxiv.org/abs/2510.25726) - [Full-Duplex-Bench: A Benchmark to Evaluate Full-Duplex Spoken Dialogue Models on Turn-taking Capabilities (2025) by Guan-Ting Lin et al](https://arxiv.org/abs/2503.04721) - [Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Models (2025) by Guan-Ting Lin et al](https://arxiv.org/abs/2507.23159) - [Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examiner (2025) by Guan-Ting Lin et al](https://arxiv.org/abs/2510.07838) - [Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024) by Lin, Pathak, Li, Li, Xia, Neubig, Zhang, Ramanan](https://arxiv.org/abs/2404.01291) - [Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead (2026) by Liu, F., Liu, Y., Sun, R., Luo, L., Guo, S](https://arxiv.org/abs/2609.17394) - [ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios (2026) by Antonio Loison et al](https://arxiv.org/abs/2601.08620) - [Natural Emergent Misalignment from Reward Hacking in Production RL (2025) by MacDiarmid, M.; Wright, B.; Uesato, J.; et al](https://arxiv.org/abs/2511.18397) - [Statistical Methods for Multiple Language Model Comparison on a Shared Evaluation (2026) by Juan Francisco Mandujano Reyes](https://arxiv.org/abs/2608.22659) - [The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections (2025) by Milad Nasr, Nicholas Carlini, et al](https://arxiv.org/abs/2510.09023) - [Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias (2026) by Norman, J.D., Rivera, M.U., Hughes, D.A](https://arxiv.org/abs/2606.19544) - [LLM Evaluators Recognize and Favor Their Own Generations (2024) by Panickssery, A., Bowman, S.R., Feng, S](https://arxiv.org/abs/2404.13076) - [Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios? (2026) by Yangda Peng et al](https://arxiv.org/abs/2606.29920) - [Language Model Tokenizers Introduce Unfairness Between Languages (2023) by Petrov, A., La Malfa, E., Torr, P.H.S., Bibi, A](https://arxiv.org/abs/2305.15425) - [olmOCR 2: Unit Test Rewards for Document OCR (2025) by Jake Poznanski, Luca Soldaini, Kyle Lo](https://arxiv.org/abs/2510.19817) - [Synthetic Test Collections for Retrieval Evaluation (2024) by Rahmani, H.A., Craswell, N., Yilmaz, E., Mitra, B., Campos, D](https://arxiv.org/abs/2405.07767) - [Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability (2026) by Harsh Raj et al](https://arxiv.org/abs/2605.10516) - [JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places (2026) by Delip Rao, Chris Callison-Burch (University of Pennsylvania)](https://arxiv.org/abs/2609.29769) - [tau-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Tasks (2026) by Soham Ray, Keshav Dhandhania, Victor Barres, Karthik Narasimhan](https://arxiv.org/abs/2603.13686) - [Brittlebench: Quantifying LLM robustness via prompt sensitivity (2026) by Romanou et al](https://arxiv.org/abs/2603.13285) - [Evaluating Retrieval Quality in Retrieval-Augmented Generation (2024) by Alireza Salemi, Hamed Zamani](https://arxiv.org/abs/2404.13781) - [Stress Testing Deliberative Alignment for Anti-Scheming Training (2025) by Bronson Schoen, Evgenia Nitishinskaya, Mikita Balesni, et al](https://arxiv.org/abs/2509.15541) - [AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation (2026) by Shi, W.; Wang, Y.; Zhao, Y.; et al](https://arxiv.org/abs/2604.18240) - [The Leaderboard Illusion (2025) by Shivalika Singh et al](https://arxiv.org/abs/2504.20879) - [MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs (2025) by Ved Sirdeshmukh et al](https://arxiv.org/abs/2501.17399) - [Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples (2025) by Alexandra Souly et al](https://arxiv.org/abs/2510.07192) - [A StrongREJECT for Empty Jailbreaks (2024) by Alexandra Souly et al](https://arxiv.org/abs/2402.10260) - [Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines (2026) by Sadman Kabir Soumik](https://arxiv.org/abs/2604.23178) - [Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation (2025) by Vaibhav Srivastav et al](https://arxiv.org/abs/2510.06961) - [Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards (2025) by Manveer Singh Tamber et al](https://arxiv.org/abs/2505.04847) - [MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems (2024) by Nandan Thakur, Suleman Kazi, Ge Luo, Jimmy Lin, Amin Ahmad](https://arxiv.org/abs/2410.13716) - ["Knowing When You Don't Know": A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation (2023) by Nandan Thakur et al](https://arxiv.org/abs/2312.11361) - [Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models (2024) by Pat Verga et al](https://arxiv.org/abs/2404.18796) - [MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models (2026) by Wang, Sun, Ko, Talati, Gong, Li, Yu, Yu, Shen, Jolly, Zhang](https://arxiv.org/abs/2603.28590) - [Measuring all the noises of LLM Evals (2025) by Wang, S.I](https://arxiv.org/abs/2512.21326) - [On the Theoretical Limitations of Embedding-Based Retrieval (2025) by Orion Weller, Michael Boratko, Iftekhar Naim, Jinhyuk Lee](https://arxiv.org/abs/2508.21038) - [J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning (2025) by Chenxi Whitehouse et al](https://arxiv.org/abs/2505.10320) - [Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets (Proceedings of the Tenth Conference on Machine Translation (WMT 2025)) (2025) by Tom Kocmi et al](https://aclanthology.org/2025.wmt-1.22/) - [LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues (2026) by Di Wu et al](https://arxiv.org/abs/2605.12493) - [MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use (2025) by Zijian Wu et al](https://arxiv.org/abs/2509.24002) - [When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation (GraphRAG-Bench) (2025) by Zhishang Xiang et al](https://arxiv.org/abs/2506.05690) - [HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (2024) by Howard Yen et al](https://arxiv.org/abs/2410.02694) - [From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training (2025) by Yuan et al](https://arxiv.org/abs/2508.09224) - [FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning (2025) by Zhang, Z., Xu, W., Wu, F., Reddy, C.K](https://arxiv.org/abs/2505.08054) - [Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems (2025) by Shaokun Zhang et al](https://arxiv.org/abs/2505.00212) - [MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark (2024) by Zhao, Q., Huang, Y., Lv, T., Cui, L., et al](https://arxiv.org/abs/2412.15194) - [ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents (2026) by Zheng, Y.; Wu, Y.; Chang, Y](https://arxiv.org/abs/2608.23635) - [Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition (2025) by Zou, A., Lin, M., Jones, E., Nowak, M., Dziemian, M., Winter, N., et al](https://arxiv.org/abs/2507.20526) - [ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence (2026) by ARC Prize Foundation](https://arxiv.org/abs/2603.24621) - [MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (2026) by Bandi, C., et al](https://arxiv.org/abs/2602.00933) - [Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation (2026) by Bhat, V., et al](https://arxiv.org/abs/2607.02577) - [BabyVision: Visual Reasoning Beyond Language (2026) by Liang Chen, Weichu Xie, Yiyan Liang, Hongfeng He, Hans Zhao, et al](https://arxiv.org/abs/2601.06521) - [HalluHard: A Hard Multi-Turn Hallucination Benchmark (2026) by Fan, D., et al](https://arxiv.org/abs/2602.01031) - [Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judges (2026) by Feng, C., Shen, M., Balashankar, A., Gerner-Beuerle, C., Rodrigues, M.R.D](https://arxiv.org/abs/2601.20913) - [Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding (2026) by Chaoyou Fu et al](https://arxiv.org/abs/2604.05015) - [Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels (2026) by Kohli, G](https://arxiv.org/abs/2605.29800) - [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (2026) by Merrill, M.A., et al](https://arxiv.org/abs/2601.11868) - [A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness (2026) by Leo Schwinn et al](https://arxiv.org/abs/2603.06594) - [Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets (2026) by Tran, D., Kiela, D](https://arxiv.org/abs/2604.02460) - [The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break (2026) by Wang et al](https://arxiv.org/abs/2604.11978) - [Efficient Evaluation of LLM Performance with Statistical Guarantees (2026) by Wu, S., Nair, Y., Candes, E](https://arxiv.org/abs/2601.20251) - [HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam (2026) by Weiqi Zhai et al](https://arxiv.org/abs/2602.13964) - [WMT24 Findings](https://aclanthology.org/2024.wmt-1.1/) ### Tool documentation > Vendor and open-source eval platform documentation. - [One connected system for agent observability by Braintrust](https://www.braintrust.dev/blog/active-observability-loop-patterns-debugger) - [Turn production data into better AI with Loop by Braintrust](https://www.braintrust.dev/blog/loop) - [Topics by Braintrust](https://www.braintrust.dev/blog/topics) - [Langfuse joins ClickHouse](https://langfuse.com/blog/announcing-acquisition) - [Langfuse August update by Langfuse team](https://langfuse.com/blog/2026-08-31-langfuse-august-update) - [Langfuse joins ClickHouse](https://langfuse.com/blog/joining-clickhouse) - [Langfuse joins ClickHouse](https://langfuse.com/blog/joining-clickhouse) - [Evaluate Individual Operations: Faster, More Precise LLM-as-a-Judge by Langfuse](https://langfuse.com/changelog/2026-02-13-observation-level-evals) - [Langfuse v4 is live: faster at scale, with more ways to search, monitor, and evaluate](https://langfuse.com/docs/v4) - [Promptfoo joining OpenAI (company update)](https://www.promptfoo.dev/blog/promptfoo-joining-openai/) - [Promptfoo is joining OpenAI by Promptfoo team](https://www.promptfoo.dev/blog/promptfoo-joining-openai/) - [Langfuse Docs](https://langfuse.com/docs) - [LangSmith Evaluation Concepts](https://docs.langchain.com/langsmith/evaluation-concepts) - [Langfuse Scores](https://langfuse.com/docs/scores/overview) - [Braintrust Docs](https://www.braintrust.dev/docs) - [Phoenix Tutorials/Evals](https://github.com/Arize-ai/phoenix/tree/main/tutorials/evals) - [Phoenix Cookbook Evaluation](https://arize.com/docs/phoenix/cookbook/evaluation) - [Inspect AI Docs](https://inspect.aisi.org.uk/) - [RAGAS Docs](https://docs.ragas.io/) - [RAG eval by RAGAS Getting Started](https://docs.ragas.io/en/stable/getstarted/rag_eval/) - [RAGAS Faithfulness metric](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/) - [TruLens Docs](https://www.trulens.org/) - [Promptfoo Docs](https://www.promptfoo.dev/) - [Promptfoo Red-team](https://www.promptfoo.dev/docs/red-team/) - [Evaluate an LLM Application by LangSmith](https://docs.langchain.com/langsmith/evaluate-llm-application) - [LLM-as-a-Judge evaluators by Langfuse Docs](https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge) - [Open-sourcing the Langfuse product (MIT) by Langfuse](https://langfuse.com/blog/2025-06-04-open-sourcing-langfuse-product) - [Testset Generation for RAG by RAGAS Docs](https://docs.ragas.io/en/stable/concepts/test_data_generation/rag/) - [Running Evals on Traces by Phoenix Docs](https://arize.com/docs/phoenix/tracing/how-to-tracing/feedback-and-annotations/evaluating-phoenix-traces) - [Self-hosting from the terminal by Phoenix Docs](https://arize.com/docs/phoenix/self-hosting/deployment-options/terminal) - [Phoenix evals package (arize-phoenix-evals)](https://github.com/Arize-ai/phoenix/tree/main/packages/phoenix-evals) - [Langfuse Pricing](https://langfuse.com/pricing) - [Langfuse Integrations](https://langfuse.com/integrations) - [Home by LangSmith Docs](https://docs.langchain.com/langsmith/home) - [Trace with OpenTelemetry by LangSmith Docs](https://docs.langchain.com/langsmith/trace-with-opentelemetry) - [Manage prompts programmatically by LangSmith Docs](https://docs.langchain.com/langsmith/manage-prompts-programmatically) - [Braintrust Pricing](https://www.braintrust.dev/pricing) - [Start by Braintrust Docs](https://www.braintrust.dev/docs/start) - [Trace integrations by Braintrust Docs](https://www.braintrust.dev/docs/guides/traces/integrations) - [Phoenix overview by Arize Docs](https://arize.com/docs/phoenix) - [Phoenix LICENSE (Elastic License 2.0)](https://github.com/Arize-ai/phoenix/blob/main/LICENSE) - [Customize models by Ragas Docs](https://docs.ragas.io/en/stable/howtos/customizations/customize_models/) - [Available metrics by Ragas Docs](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/) - [Model-graded metrics by Promptfoo Docs](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/) - [Scorers by Inspect AI Docs](https://inspect.aisi.org.uk/scorers.html) - [Models by Inspect AI Docs](https://inspect.aisi.org.uk/models.html) - [Log viewer by Inspect AI Docs](https://inspect.aisi.org.uk/log-viewer.html) - [Getting started by TruLens Docs](https://www.trulens.org/getting_started/) - [Manage datasets (snapshots) by Braintrust Docs](https://www.braintrust.dev/docs/annotate/datasets/manage) - [Braintrust evaluation quickstart](https://www.braintrust.dev/docs/evaluation-quickstart) - [Manage datasets by Braintrust Docs](https://www.braintrust.dev/docs/annotate/datasets/manage) - [Braintrust plans and limits](https://www.braintrust.dev/docs/plans-and-limits) - [Braintrust self-hosting and hybrid deployment](https://www.braintrust.dev/docs/admin/self-hosting) - [Scoring Metrics by Inspect AI Docs](https://inspect.aisi.org.uk/metrics.html) - [Inspect AI Docs: Sandboxing by UK AI Security Institute, Meridian Labs](https://inspect.aisi.org.uk/sandboxing.html) - [Billable units in Langfuse Cloud by Langfuse](https://langfuse.com/docs/administration/billable-units) - [Datasets (versioning) by Langfuse Docs](https://langfuse.com/docs/evaluation/experiments/datasets) - [Migration of deprecated APIs by Langfuse](https://langfuse.com/faq/all/deprecated-api-migration) - [LLM-as-a-judge migration to observation-level evaluators by Langfuse FAQ](https://langfuse.com/faq/all/llm-as-a-judge-migration) - [Langfuse self-hosting: license key and enterprise features](https://langfuse.com/self-hosting/license-key) - [LLM-as-a-Judge (Langfuse evaluation methods) by Langfuse](https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge) - [Public API: Observations API v2 by Langfuse](https://langfuse.com/docs/api-and-data-platform/features/observations-api) - [Scores via SDK by Langfuse Docs](https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk) - [SDK instrumentation by Langfuse Docs](https://langfuse.com/docs/observability/sdk/instrumentation) - [Python SDK setup by Langfuse Docs](https://langfuse.com/docs/observability/sdk/overview) - [SDK setup by Langfuse Docs](https://langfuse.com/docs/observability/sdk/overview#setup) - [User Feedback by Langfuse](https://langfuse.com/docs/observability/features/user-feedback) - [Langfuse v4 is live: faster at scale, with more ways to search, monitor, and evaluate by Langfuse team](https://langfuse.com/changelog/2026-08-17-langfuse-v4) - [LangSmith Docs. Manage datasets by LangChain](https://docs.langchain.com/langsmith/manage-datasets) - [LangSmith Engine by LangChain](https://docs.langchain.com/langsmith/engine) - [LangSmith Docs. Insights by LangChain](https://docs.langchain.com/langsmith/insights) - [LangSmith Tuned Evaluators by LangChain](https://docs.langchain.com/langsmith/tuned-evaluators) - [Dataset Upsert by Arize](https://arize.com/docs/phoenix/release-notes/04-2026/04-29-2026-dataset-upsert) - [Phoenix Docs. Updating Datasets by Arize AI](https://arize.com/docs/phoenix/datasets-and-experiments/how-to-datasets/updating-datasets) - [Phoenix Docs: Configuration (self-hosting) by Arize AI](https://arize.com/docs/phoenix/self-hosting/configuration) - [Evals as Tests: pytest and Vitest/Jest Integrations by Arize](https://arize.com/docs/phoenix/release-notes/06-2026/06-26-2026-eval-ci-pytest-vitest-jest) - [Filter Expressions (and Trace Filter Expressions release note) by Arize](https://arize.com/docs/phoenix/tracing/how-to-tracing/filter-expressions) - [PXI (Phoenix Intelligence) by Arize AI](https://arize.com/docs/phoenix/pxi) - [Phoenix Docs: Release notes by Arize AI](https://arize.com/docs/phoenix/release-notes) - [OpenTelemetry and OpenInference Sampling by Arize](https://arize.com/docs/phoenix/tracing/concepts-tracing/otel-openinference/sampling) - [Phoenix v14.0.0 breaking changes by Arize](https://arize.com/docs/phoenix/release-notes/04-2026/04-07-2026-phoenix-v14-breaking-changes) - [Code Scanning - LLM Security Vulnerability Scanner by Promptfoo](https://www.promptfoo.dev/docs/code-scanning/) - [Promptfoo Docs: OWASP LLM Top 10](https://www.promptfoo.dev/docs/red-team/owasp-llm-top-10/) - [Promptfoo Pricing](https://www.promptfoo.dev/pricing/) - [Tracing by Promptfoo](https://www.promptfoo.dev/docs/tracing/) - [Ragas Docs. Answer Correctness by Ragas maintainers](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_correctness/) - [Ragas Docs. Context Precision by Ragas maintainers](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/) - [Ragas Docs. Context Recall by Ragas maintainers](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/) - [Ragas Docs. Migrate from v0.3 to v0.4 by Ragas maintainers, Exploding Gradients](https://docs.ragas.io/en/latest/howtos/migrations/migrate_from_v03_to_v04/) - [Ragas Docs. Nvidia metrics (Answer Accuracy, Context Relevance, Response Groundedness) by Ragas maintainers](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/nvidia_metrics/) - [Migrating from Feedback to Metric by TruLens](https://www.trulens.org/component_guides/evaluation/metric_migration/) ### Standards and regulations > NIST, EU AI Act, OWASP, ITU, MLCommons. - [MLCommons Unveils New Jailbreak Benchmark, Quantifying AI's Resilience Gap to Adversarial Attacks](https://mlcommons.org/2025/10/ailuminate-jailbreak-v05/) - [The TREC-8 Question Answering Track Report by Voorhees](https://trec.nist.gov/pubs/trec8/papers/qa_report.pdf) - [ITU-T P.808. Subjective MOS via crowdsourcing](https://www.itu.int/rec/T-REC-P.808) - [MLCommons AILuminate](https://mlcommons.org/benchmarks/ailuminate/) - [OWASP Vendor Evaluation Criteria for AI Red Teaming Providers & Tooling v1.0 by OWASP GenAI Security Project](https://genai.owasp.org/resource/owasp-vendor-evaluation-criteria-for-ai-red-teaming-providers-tooling-v1-0/) - [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) - [official portal by EU AI Act](https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai) - [AI Omnibus enters into force (Regulation (EU) 2026/1744) (2026) by European Commission](https://digital-strategy.ec.europa.eu/en/news/ai-omnibus-enters-force) - [The General-Purpose AI Code of Practice by European Commission](https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpai) - [OWASP Top 10 for LLM Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/) - [OWASP Top 10 for Agentic Applications (2026) (2026)](https://genai.owasp.org/2025/12/09/owasp-top-10-for-agentic-applications-the-benchmark-for-agentic-security-in-the-age-of-autonomous-ai/) - [OWASP Top 10 for LLM Applications 2026 (2026) by OWASP GenAI Security Project](https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/) - [Agentic AI - Threats and Mitigations (v1.1, December 2025) (2025) by OWASP Agentic Security Initiative](https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/) - [Questions and answers on the code of practice for General-Purpose AI by European Commission](https://digital-strategy.ec.europa.eu/en/faqs/questions-and-answers-code-practice-general-purpose-ai) - [Draft Commission guidelines on the classification of high-risk AI systems by European Commission](https://digital-strategy.ec.europa.eu/en/library/draft-commission-guidelines-classification-high-risk-ai-systems) - [Code of Practice on Transparency of AI-generated Content by European Commission](https://digital-strategy.ec.europa.eu/en/policies/code-practice-ai-generated-content) - [Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1) by NIST](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence) - [Practices for Automated Benchmark Evaluations of Language Models (NIST AI 800-2 initial public draft) by NIST, Center for AI Standards and Innovation (CAISI)](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.800-2.ipd.pdf) - [AI RMF Core (Section 5: Govern, Map, Measure, Manage subcategories) by NIST AI Resource Center](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/) - [AI Agent Standards Initiative by NIST, Center for AI Standards and Innovation (CAISI)](https://www.nist.gov/caisi/ai-agent-standards-initiative) - [Cybersecurity Framework Profile for Artificial Intelligence (Cyber AI Profile): NIST Community Profile (NIST IR 8596 initial preliminary draft) by NIST](https://csrc.nist.gov/pubs/ir/8596/iprd) - [Agent Control Standard (ACS) by OWASP GenAI Security Project](https://genai.owasp.org/resource/agent-control-standard-acs/) ### Other sources - [Yan, Bischof, Frye, Husain, Liu, Shankar by Applied LLMs](https://applied-llms.org/) - [LLM Powered Autonomous Agents by Lilian Weng](https://lilianweng.github.io/posts/2023-06-23-agent/) - [Extrinsic Hallucinations in LLMs by Lilian Weng](https://lilianweng.github.io/posts/2024-07-07-hallucination/) - [Adversarial Attacks on LLMs by Lilian Weng](https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm/) - [Specification Gaming Examples by Victoria Krakovna](https://vkrakovna.wordpress.com/2018/04/02/specification-gaming-examples-in-ai/) - [LLM Evaluation Guidebook by Clémentine Fourrier](https://huggingface.co/blog/clefourrier/llm-evaluation) - [Evaluating MMLU by HuggingFace](https://huggingface.co/blog/evaluating-mmlu-leaderboard) - [Hard Stuff Nobody Talks About by Honeycomb](https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llm) - [The Dual LLM pattern for building AI assistants that can resist prompt injection by Simon Willison](https://simonwillison.net/2023/Apr/25/dual-llm-pattern/) - [Introducing Bloom: an open source tool for automated behavioral evaluations by Isha Gupta et al](https://www.anthropic.com/research/bloom) - [Demystifying evals for AI agents by Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares, Jiri De Jonghe](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) - [Quantifying infrastructure noise in agentic coding evals by Anthropic](https://www.anthropic.com/engineering/infrastructure-noise) - [How we built our multi-agent research system by Anthropic](https://www.anthropic.com/engineering/multi-agent-research-system) - [Petri 2.0: New Scenarios, New Model Comparisons, and Improved Eval-Awareness Mitigations by Anthropic Alignment Science](https://alignment.anthropic.com/2026/petri-v2/) - [A postmortem of three recent issues by Anthropic](https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues) - [Announcing ARC-AGI-3 by Kamradt, G](https://arcprize.org/blog/arc-agi-3-launch) - [Agent Arena: Causal Evaluation of Agents in the Real World by Arena Team](https://arena.ai/blog/agent-arena-methodology) - [Introducing AutoEval to the Arena leaderboards by Arena Team](https://arena.ai/blog/autoeval-scores) - [Factuality in the Arena by Arena Team](https://arena.ai/blog/factuality-in-arena) - [Arena's Ranking Methodology by Arena (formerly LMArena, Chatbot Arena)](https://arena.ai/blog/ranking-method) - [LMArena is now Arena by Arena](https://arena.ai/blog/lmarena-is-now-arena/) - [LMArena is now Arena by Arena Team](https://arena.ai/blog/lmarena-is-now-arena/) - [Does Style Matter? Disentangling style and substance in Chatbot Arena by Li, T., Angelopoulos, A.N., Chiang, W.-L](https://arena.ai/blog/style-control/) - [AWS achieves ISO/IEC 42001:2023 Artificial Intelligence Management System accredited certification (25 November 2024) (2024) by Amazon Web Services](https://aws.amazon.com/blogs/machine-learning/aws-achieves-iso-iec-420012023-artificial-intelligence-management-system-accredited-certification/) - [BFCL V4 Part 1: Web Search (Agentic evaluation and full leaderboard score composition) by Gorilla LLM Team, UC Berkeley](https://gorilla.cs.berkeley.edu/blogs/15_bfcl_v4_web_search.html) - [BFCL V4 Agentic Part 3: Evaluating Format Sensitivity for Tool Calls by Mao, H., Li, M., Wu, R., Patil, S.G., Gonzalez, J.E](https://gorilla.cs.berkeley.edu/blogs/17_bfcl_v4_prompt_variation.html) - [BFCL V4 Agentic Part 3: Evaluating Format Sensitivity for Tool Calls by Mao, H., Li, M., Wu, R., Patil, S.G., Gonzalez, J.E](https://gorilla.cs.berkeley.edu/blogs/17_bfcl_v4_prompt_variation.html) - [Introducing HLE-Diamond](https://lastexam.ai/blog/hle-diamond) - [Context Rot (Chroma Research report on input length and LLM performance) by Chroma Research](https://www.trychroma.com/research/context-rot) - [Making AI Trustworthy and Observable in Real-Time: Cisco Announces Intent to Acquire Galileo by Kamal Hathi](https://blogs.cisco.com/news/cisco-announces-the-intent-to-acquire-galileo) - [Introducing Jev for Evals: The Anti-Language, Language-Model-as-a-Judge by Jeffrey Ip](https://deepeval.com/blog/introducing-jev-in-deepeval) - [Introducing persistent local storage in DeepEval, with SQLite by Jeffrey Ip](https://deepeval.com/blog/introducing-sqlite-local-storage) - [Berkeley Function Calling Leaderboard (BFCL) v4: Benchmark review by Epoch AI](https://epoch.ai/benchmarks/berkeley-function-calling-leaderboard/review) - [Announcing FrontierMath Erdos by Epoch AI](https://epoch.ai/latest/announcing-frontiermath-erdos) - [Announcing Agent Control: The Open Source Control Plane for AI Agents by Yash Sheth](https://galileo.ai/blog/announcing-agent-control) - [Defeating Nondeterminism in LLM Inference by Horace He, Thinking Machines Lab](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/) - [Introducing RTEB: A New Standard for Retrieval Evaluation by Hugging Face and the MTEB team (including Frank Liu, Kenneth Enevoldsen)](https://huggingface.co/blog/rteb) - [Introducing RTEB: A New Standard for Retrieval Evaluation by Frank Liu, Kenneth Enevoldsen, Roman Solomatin, Isaac Chung, et al](https://huggingface.co/blog/rteb) - [Security Incident Disclosure, July 2026 (2026) by Hugging Face](https://huggingface.co/blog/security-incident-july-2026) - [Context Rot: How Increasing Input Tokens Impacts LLM Performance by Kelly Hong, Anton Troynikov, Jeff Huber](https://www.trychroma.com/research/context-rot) - [LlamaIndex Newsletter 2026-02-24 (2026)](https://www.llamaindex.ai/blog/llamaindex-newsletter-2026-02-24) - [Agentic Misalignment in Summer 2026 (2026) by Lynch, A.; Hughes, J.; Serrano, A.; Kirk, R.; Bowman, S.R](https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/) - [Summary of METR's predeployment evaluation of GPT-5.6 Sol by METR](https://metr.org/blog/2026-06-26-gpt-5-6-sol/) - [Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident by METR](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) - [Recent Frontier Models Are Reward Hacking by Von Arx, S.; Chan, L.; Barnes, B](https://metr.org/blog/2025-06-05-recent-reward-hacking/) - [Microsoft DeBERTa surpasses human performance on the SuperGLUE benchmark by Microsoft Research](https://www.microsoft.com/en-us/research/blog/microsoft-deberta-surpasses-human-performance-on-the-superglue-benchmark/) - [Shieldstral by Mistral AI](https://mistral.ai/news/shieldstral/) - [SLEIGHT-Bench: Finding Blind Spots in AI Monitors by Najt, E.; Toft, C.; Tracy, T.; Roger, F.; Benton, J](https://alignment.anthropic.com/2026/sleight-bench/) - [Training a Misaligned Reward Seeker by Qi, R.; Wright, B.; MacDiarmid, M.; Hubinger, E](https://alignment.anthropic.com/2026/reward-seeker/) - [ragas 0.4.3: ChatVertexAI import broken - uses removed langchain_community path (issue #2745) by Ragas GitHub issue tracker](https://github.com/vibrantlabsai/ragas/issues/2745) - [How We Broke Top AI Agent Benchmarks: And What Comes Next by Hao Wang, Qiuyang Mang, Alvin Cheung, Koushik Sen, Dawn Song](https://rdi.berkeley.edu/blog/trustworthy-benchmarks-cont/) - [Closing the loop: Evaluating and improving Replit Agent at scale by Daniel Furman, Peter Zhong, Zhen Li, Michele Catasta](https://replit.com/blog/evaluating-and-improving-agent-at-scale) - [tau3-Bench: Fixing Airline + Retail by Barres, V., Shi, B](https://taubench.com/blog/tau3-task-fixes.html) - [Snowflake to Acquire Truera AI Observability Platform by Attariyan, M., Pothukuchi, P](https://www.snowflake.com/blog/snowflake-acquires-truera-to-bring-llm-ml-observability-to-data-cloud/) - [Terminal-Bench 4.0 by Terminal-Bench team (Stanford, Laude Institute)](https://www.tbench.ai/news/terminal-bench-4-0) - [Defeating Nondeterminism in LLM Inference by Horace He, Thinking Machines Lab](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/) - [Building Effective Agents by Anthropic](https://www.anthropic.com/engineering/building-effective-agents) - [SWE-bench Sonnet by Anthropic](https://www.anthropic.com/research/swe-bench-sonnet) - [Measuring Faithfulness in CoT by Anthropic](https://www.anthropic.com/research/measuring-faithfulness-in-chain-of-thought-reasoning) - [Reasoning Models Don't Say What They Think by Anthropic](https://www.anthropic.com/research/reasoning-models-dont-say-think) - [Measuring Persuasiveness of LMs by Anthropic](https://www.anthropic.com/research/measuring-model-persuasiveness) - [Anthropic Responsible Scaling Policy](https://www.anthropic.com/responsible-scaling-policy) - [Agentic Misalignment by Anthropic](https://www.anthropic.com/research/agentic-misalignment) - [Adding Error Bars to Evals (blog) by Anthropic](https://www.anthropic.com/research/statistical-approach-to-model-evals) - [Anthropic Tool Use Overview (strict mode)](https://platform.claude.com/docs/en/docs/agents-and-tools/tool-use/overview) - [Anthropic Evaluation Tool (Console)](https://platform.claude.com/docs/en/test-and-evaluate/eval-tool) - [Define success criteria and build evaluations by Anthropic](https://platform.claude.com/docs/en/docs/test-and-evaluate/develop-tests) - [Claude API pricing by Anthropic](https://platform.claude.com/docs/en/about-claude/pricing) - [Computer use tool by Anthropic](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) - [Context windows (Claude API docs) by Anthropic](https://platform.claude.com/docs/en/build-with-claude/context-windows) - [Fast mode (research preview) by Anthropic](https://platform.claude.com/docs/en/build-with-claude/fast-mode) - [Model deprecations by Anthropic](https://platform.claude.com/docs/en/about-claude/model-deprecations) - [Models overview by Anthropic](https://platform.claude.com/docs/en/models/overview) - [Migrating to Claude Opus 5.5 by Anthropic](https://platform.claude.com/docs/en/models/opus-5-5/migration-guide) - [Migrating to Claude Opus 5.5 by Anthropic](https://platform.claude.com/docs/en/models/opus-5-5/migration-guide) - [Refusals and fallback by Anthropic](https://platform.claude.com/docs/en/build-with-claude/refusals-and-fallback) - [Risk Report: August 2026 (2026) by Anthropic](https://www.anthropic.com/aug-2026-risk-report) - [Claude Sonnet 4.5 model page by Anthropic](https://platform.claude.com/docs/en/models/sonnet-4-5/overview) - [Migrating to Claude Sonnet 5.5 by Anthropic](https://platform.claude.com/docs/en/models/sonnet-5-5/migration-guide) - [Claude Sonnet 5.5 model overview by Anthropic](https://platform.claude.com/docs/en/models/sonnet-5-5/overview) - [Strict tool use by Anthropic](https://platform.claude.com/docs/en/agents-and-tools/tool-use/strict-tool-use) - [Strict tool use by Anthropic](https://platform.claude.com/docs/en/agents-and-tools/tool-use/strict-tool-use) - [Structured outputs (Claude API docs) by Anthropic](https://platform.claude.com/docs/en/build-with-claude/structured-outputs) - [Model system cards by Anthropic](https://www.anthropic.com/system-cards) - [Steering thinking by Anthropic](https://platform.claude.com/docs/en/build-with-claude/thinking-steering-and-cost) - [Tool search tool by Anthropic](https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool) - [OpenAI Evals (GitHub)](https://github.com/openai/evals) - [Introducing SWE-bench Verified by OpenAI](https://openai.com/index/introducing-swe-bench-verified/) - [OpenAI Preparedness Framework](https://openai.com/safety/preparedness/) - [A Practical Guide to Building Agents by OpenAI](https://openai.com/business/guides-and-resources/a-practical-guide-to-building-ai-agents/) - [Getting Started with OpenAI Evals by OpenAI Cookbook](https://developers.openai.com/cookbook/examples/evaluation/getting_started_with_openai_evals) - [Eval LLMs for SQL Generation by OpenAI Cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_evaluate_LLMs_for_SQL_generation.ipynb) - [Evaluate RAG with LlamaIndex by OpenAI Cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb) - [OpenAI Preparedness GitHub (MLE-bench, SWE-Lancer, PaperBench)](https://github.com/openai/preparedness) - [OpenAI SWE-Lancer Benchmark](https://github.com/openai/swelancer-benchmark) - [Graders guide (Evals API and RFT) by OpenAI](https://developers.openai.com/api/docs/guides/graders) - [OpenAI API changelog](https://developers.openai.com/api/docs/changelog) - [OpenAI Deployment Safety Hub: System Cards](https://deploymentsafety.openai.com/) - [Deprecations by OpenAI](https://developers.openai.com/api/docs/deprecations) - [Vector embeddings by OpenAI](https://developers.openai.com/api/docs/guides/embeddings) - [Create eval (Evals API reference) by OpenAI](https://developers.openai.com/api/reference/resources/evals/methods/create) - [OpenAI Evals: Completion Functions](https://github.com/openai/evals/blob/main/docs/completion-fns.md) - [Moving from OpenAI Evals to Promptfoo by Konstantine Kahadze, OpenAI](https://developers.openai.com/cookbook/examples/evaluation/moving-from-openai-evals-to-promptfoo) - [Function calling guide by OpenAI](https://developers.openai.com/api/docs/guides/function-calling) - [Decisions guide (beta) by OpenAI](https://developers.openai.com/api/docs/guides/decisions) - [Decisions API is now available in Public Beta by OpenAI Developer Community](https://community.openai.com/t/decisions-api-is-now-available-in-public-beta/1403877) - [GPT-4o mini model page by OpenAI](https://developers.openai.com/api/docs/models/gpt-4o-mini) - [GPT-4o model page by OpenAI](https://developers.openai.com/api/docs/models/gpt-4o) - [Using GPT-6 by OpenAI](https://developers.openai.com/api/docs/guides/latest-model) - [gpt-oss-safeguard-20b model card by OpenAI](https://huggingface.co/openai/gpt-oss-safeguard-20b) - [Models by OpenAI](https://developers.openai.com/api/docs/models) - [OpenAI MRCR (multi-round co-reference resolution) dataset](https://huggingface.co/datasets/openai/mrcr) - [Pacing model development in an era of cyber-critical capabilities by OpenAI](https://openai.com/index/pacing-model-development-cyber-capabilities/) - [Path to Astra: critical capabilities and frontier safeguards by OpenAI](https://openai.com/index/path-to-astra/) - [Preparedness Framework, Version 2 (last updated April 15, 2025) (2025) by OpenAI](https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbddebcd/preparedness-framework-v2.pdf) - [OpenAI API Pricing](https://developers.openai.com/api/docs/pricing) - [Towards safety cases for frontier AI training by OpenAI](https://openai.com/index/towards-safety-cases-for-frontier-ai-training/) - [OpenAI simple-evals GitHub repository (BrowseComp reference implementation; July 2025 deprecation notice) (2025)](https://github.com/openai/simple-evals) - [Structured Outputs (OpenAI API docs) by OpenAI](https://developers.openai.com/api/docs/guides/structured-outputs) - [Why SWE-bench Verified no longer measures frontier coding capabilities by OpenAI](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/) - [OpenAI voice agents guide](https://developers.openai.com/api/docs/guides/voice-agents) - [Time Horizon 1.1 by METR](https://metr.org/blog/2026-1-29-time-horizon-1-1/) - [Needle in a Haystack by Greg Kamradt](https://github.com/gkamradt/LLMTest_NeedleInAHaystack) - [Fairness and Machine Learning by Barocas, Hardt & Narayanan](https://fairmlbook.org/) - [xlang.ai. OSWorld-Verified](https://xlang.ai/blog/osworld-verified) - [More compute, more capability (test-time compute in agent evals) by UK AISI](https://www.aisi.gov.uk/blog/more-compute-more-capability-why-ai-agent-evals-need-to-account-for-test-time-compute) - [Introduction to Information Retrieval by Manning, Raghavan & Schütze](https://nlp.stanford.edu/IR-book/) - [Cumulated Gain-Based Evaluation of IR Techniques by Järvelin & Kekäläinen](https://dl.acm.org/doi/10.1145/582415.582418) - [BenchmarkQED: Automated Benchmarking of RAG Systems by Microsoft Research](https://www.microsoft.com/en-us/research/blog/benchmarkqed-automated-benchmarking-of-rag-systems/) - [BFCL (Patil, Mao et al., ICML 2025) (2025)](https://gorilla.cs.berkeley.edu/leaderboard.html) - [AST vs executable evaluation by BFCL blog](https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html) - [Multi-turn and state-based evaluation by BFCL blog](https://gorilla.cs.berkeley.edu/blogs/13_bfcl_v3_multi_turn.html) - [BIRD-bench](https://bird-bench.github.io/) - [Spider by Yu et al](https://yale-lily.github.io/spider) - [NLP4RE ReqEval Task. Ambiguity Detection and Disambiguation in Requirements](https://nlp4re.github.io/2020/reqeval.html) - [Scale AI SWE-bench Pro Public Leaderboard](https://labs.scale.com/leaderboard/swe_bench_pro_public) - [Laude Institute Terminal-Bench](https://www.tbench.ai/) - [PartiPrompts by Google Research](https://github.com/google-research/parti) - [MTEB Benchmark](https://github.com/embeddings-benchmark/mteb) - [Whisper repo (Common Voice 15, Fleurs WER) by OpenAI](https://github.com/openai/whisper) - [A Practitioner's Guide to Cluster-Robust Inference (Journal of Human Resources 50(2): 317-372, 2015) (2015) by Cameron, A.C., Miller, D.L](https://cameron.econ.ucdavis.edu/research/Cameron_Miller_JHR_2015_February.pdf) - [RULERv2: Retrieval First, Solving Next, A Bottom-Up Approach for Long-Context Evaluation (NeurIPS 2025 workshop: Evaluating the Evolving LLM Lifecycle) (2025) by Cheng-Ping Hsieh, Faisal Ladhak, Krishna Puvvada, Boris Ginsburg](https://neurips.cc/virtual/2025/122399) - [Training large language models on narrow tasks can lead to broad misalignment (Nature 649, 584-589 2026) (2026) by Jan Betley et al](https://www.nature.com/articles/s41586-025-09937-5) - [Comet Opik](https://github.com/comet-ml/opik) - [Inspect Evals Registry](https://github.com/UKGovernmentBEIS/inspect_evals) - [Inspect AI GitHub](https://github.com/UKGovernmentBEIS/inspect_ai) - [RAGAS GitHub](https://github.com/explodinggradients/ragas) - [Fairlearn Docs](https://fairlearn.org/) - [AIF360 GitHub](https://github.com/Trusted-AI/AIF360) - [SHAP Docs](https://shap.readthedocs.io/en/latest/) - [Captum Docs](https://captum.ai/) - [Perspective API](https://perspectiveapi.com/) - [OpenTelemetry GenAI Semantic Conventions repository](https://github.com/open-telemetry/semantic-conventions-genai) - [DeepEval GitHub](https://github.com/confident-ai/deepeval) - [Giskard GitHub](https://github.com/Giskard-AI/giskard) - [LlamaIndex Evaluation](https://developers.llamaindex.ai/python/framework/module_guides/evaluating/) - [Outlines GitHub](https://github.com/outlines-dev/outlines) - [Instructor GitHub](https://github.com/jxnl/instructor) - [HuggingFace LLM-as-judge Cookbook](https://huggingface.co/learn/cookbook/en/llm_judge) - [HarmBench GitHub](https://github.com/centerforaisafety/HarmBench) - [NVIDIA garak](https://github.com/NVIDIA/garak) - [WebArena GitHub](https://github.com/web-arena-x/webarena) - [VisualWebArena GitHub](https://github.com/web-arena-x/visualwebarena) - [OSWorld site](https://os-world.github.io/) - [SWE-bench site](https://www.swebench.com/) - [SWE-bench Verified](https://www.swebench.com/verified.html) - [τ-bench GitHub](https://github.com/sierra-research/tau-bench) - [AgentBench GitHub](https://github.com/THUDM/AgentBench) - [NVIDIA RULER](https://github.com/NVIDIA/RULER) - [BABILong HF dataset](https://huggingface.co/datasets/RMT-team/babilong) - [OpenAI HumanEval](https://github.com/openai/human-eval) - [BigCodeBench GitHub](https://github.com/bigcode-project/bigcodebench) - [LiveCodeBench GitHub](https://github.com/LiveCodeBench/LiveCodeBench) - [DocVQA](https://www.docvqa.org/) - [Intro to evals by Vercel](https://vercel.com/kb/guide/an-introduction-to-evals) - [LLM evals with Vercel AI SDK + Vitest by Xata](https://xata.io/blog/llm-evals-with-vercel-ai-and-vitest) - [FastChat MT-Bench README](https://github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge/README.md) - [Prompt_evaluations by Anthropic Courses](https://github.com/anthropics/courses/tree/master/prompt_evaluations) - [Building_evals.ipynb by Anthropic Cookbook](https://github.com/anthropics/claude-cookbooks/blob/main/misc/building_evals.ipynb) - [SWE-bench harness GitHub](https://github.com/SWE-bench/SWE-bench) - [SWE-bench Verified HF dataset](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified) - [mini-SWE-agent GitHub](https://github.com/SWE-agent/mini-swe-agent) - [SWE-bench sb-cli docs](https://www.swebench.com/sb-cli/) - [Speed, Structure, Smarts by Notion AI](https://www.notion.com/blog/speed-structure-and-smarts-the-notion-ai-way) - [Decision-Time Guidance by Replit](https://replit.com/blog/decision-time-guidance) - [RAG eval with Ragas by Langfuse cookbook](https://github.com/langfuse/langfuse-docs/blob/main/cookbook/evaluation_of_rag_with_ragas.ipynb) - [Error analysis by Langfuse cookbook](https://github.com/langfuse/langfuse-docs/blob/main/cookbook/error-analysis-llm-applications.ipynb) - [Multi-turn eval by Langfuse cookbook](https://github.com/langfuse/langfuse-docs/blob/main/cookbook/example_evaluating_multi_turn_conversations.ipynb) - [Pinecone RAG eval](https://www.pinecone.io/learn/series/vector-databases-in-production-for-busy-engineers/rag-evaluation/) - [MLE-bench repo](https://github.com/openai/mle-bench) - [PaperBench repo](https://github.com/openai/preparedness/tree/main/project/paperbench) - [Luna-2 evaluation SLMs by Galileo](https://galileo.ai/luna-2) - [HHEM-2.1-Open model card by Vectara](https://huggingface.co/vectara/hallucination_evaluation_model) - [External evaluation pipelines by Langfuse cookbook](https://github.com/langfuse/langfuse-docs/blob/main/cookbook/example_external_evaluation_pipelines.ipynb) - [datasketch. MinHash and MinHash LSH docs](https://ekzhu.com/datasketch/) - [Vercel AI SDK docs](https://ai-sdk.dev/docs/introduction) - [Vercel AI SDK 6 migration guide](https://ai-sdk.dev/docs/migration-guides/migration-guide-6-0) - [Vitest docs](https://vitest.dev/) - [DeepResearch Bench GitHub (RACE + FACT scoring)](https://github.com/Ayanami0730/deep_research_bench) - [MAST GitHub (taxonomy, LLM annotator, trace dataset)](https://github.com/multi-agent-systems-failure-taxonomy/MAST) - [MCP-Universe GitHub](https://github.com/SalesforceAIResearch/MCP-Universe) - [ModelScope MCPBench GitHub](https://github.com/modelscope/MCPBench) - [tau2-bench GitHub (successor line, ships tau3-bench) by Sierra Research](https://github.com/sierra-research/tau2-bench) - [GitHub Spec Kit](https://github.com/github/spec-kit) - [Kiro Specs Documentation](https://kiro.dev/docs/specs/) - [LICENSE by Langfuse GitHub](https://github.com/langfuse/langfuse/blob/main/LICENSE) - [LangChain Pricing](https://www.langchain.com/pricing) - [Braintrust Python SDK (PyPI)](https://pypi.org/project/braintrust/) - [LangWatch Pricing](https://langwatch.ai/pricing) - [LangWatch Integrations](https://langwatch.ai/integrations) - [Arize Pricing](https://arize.com/pricing/) - [AX production monitoring by Arize Docs](https://arize.com/docs/ax/observe/production-monitoring) - [Weights and Biases Pricing](https://wandb.ai/site/pricing) - [Weave Docs](https://weave-docs.wandb.ai/) - [Weave product page by Weights and Biases](https://wandb.ai/site/weave) - [W&B Weave GitHub (license)](https://github.com/wandb/weave) - [Helicone Pricing](https://www.helicone.ai/pricing) - [Helicone Docs](https://docs.helicone.ai/) - [Self-host overview by Helicone Docs](https://docs.helicone.ai/getting-started/self-host/overview) - [Scores by Helicone Docs](https://docs.helicone.ai/features/advanced-usage/scores) - [Experiments by Helicone Docs](https://docs.helicone.ai/features/experiments) - [Helicone GitHub (license)](https://github.com/Helicone/helicone) - [Vercel AI SDK integration by Helicone Docs](https://docs.helicone.ai/getting-started/integration-method/vercelai) - [Acquired by Anthropic, platform sunset by Humanloop](https://humanloop.com/) - [PromptLayer Pricing](https://www.promptlayer.com/pricing) - [PromptLayer Docs](https://docs.promptlayer.com/) - [PromptLayer homepage](https://www.promptlayer.com/) - [Comet Pricing (Opik)](https://www.comet.com/site/pricing/) - [Quickstart by Opik Docs](https://www.comet.com/docs/opik/quickstart) - [Log traces by Opik Docs](https://www.comet.com/docs/opik/tracing/log_traces) - [Online evaluation rules by Opik Docs](https://www.comet.com/docs/opik/production/rules) - [Managing prompts in code by Opik Docs](https://www.comet.com/docs/opik/prompt_engineering/managing_prompts_in_code) - [Promptfoo GitHub](https://github.com/promptfoo/promptfoo) - [Getting started by DeepEval Docs](https://deepeval.com/docs/getting-started) - [Build-eval doc by OpenAI Evals](https://github.com/openai/evals/blob/main/docs/build-eval.md) - [Evals API guide (dashboard deprecation notice) by OpenAI](https://developers.openai.com/api/docs/guides/evals) - [Evidently AI homepage](https://www.evidentlyai.com/) - [Evidently Docs](https://docs.evidentlyai.com/) - [Quickstart LLM by Evidently Docs](https://docs.evidentlyai.com/quickstart_llm) - [Customize LLM judge by Evidently Docs](https://docs.evidentlyai.com/metrics/customize_llm_judge) - [Agent Control GitHub, Galileo Luna Evaluator package README (agent-control-evaluator-galileo) by Agent Control contributors, Galileo](https://github.com/agentcontrol/agent-control/tree/main/evaluators/contrib/galileo) - [Galileo Luna-2 Evaluator by Agent Control](https://docs.agentcontrol.dev/concepts/evaluators/galileo-luna2) - [AgentDojo documentation: Results by ETH SPY Lab](https://agentdojo.spylab.ai/results/) - [AgentDojo GitHub repository and release notes by ETH SPY Lab](https://github.com/ethz-spylab/agentdojo) - [Vercel AI SDK 7 migration guide](https://ai-sdk.dev/docs/migration-guides/migration-guide-7-0) - [CRAN: Package aif360](https://cran.r-project.org/package=aif360) - [aif360 (PyPI package page and release history) by aif360 developers, Trusted-AI](https://pypi.org/project/aif360/) - [New release of scikit-learn seems to have broken the code (AIF360 issue #555) by AIF360 users, Trusted-AI](https://github.com/Trusted-AI/AIF360/issues/555) - [ARC-AGI-3 by ARC Prize](https://arcprize.org/arc-agi/3) - [GPT-6 Astra ARC-AGI Results by ARC Prize](https://arcprize.org/results/openai-gpt-6-astra) - [Results by ARC Prize](https://arcprize.org/results) - [Arena (formerly LMArena and Chatbot Arena): AI ranking and LLM leaderboard](https://arena.ai/) - [Arena-Hard-Auto (GitHub repository README) by LMArena, Arena team](https://github.com/lmarena/arena-hard-auto) - [Leaderboard Changelog by Arena](https://arena.ai/company/leaderboard-changelog) - [Text Arena leaderboard by Arena](https://arena.ai/leaderboard/text) - [Text-to-Image Leaderboard by Arena, formerly LMArena](https://arena.ai/leaderboard/text-to-image) - [Argilla (GitHub README)](https://github.com/argilla-io/argilla) - [Arize AX Docs. Online Evals by Arize AI](https://arize.com/docs/ax/evaluate/online-evals) - [AIME 2025 Benchmark Leaderboard (2025) by Artificial Analysis](https://artificialanalysis.ai/evaluations/aime-2025) - [GDP.pdf Benchmark Leaderboard by Artificial Analysis](https://artificialanalysis.ai/evaluations/gdp-pdf) - [GPQA Diamond Benchmark Leaderboard by Artificial Analysis](https://artificialanalysis.ai/evaluations/gpqa-diamond) - [Humanity's Last Exam Benchmark Leaderboard by Artificial Analysis](https://artificialanalysis.ai/evaluations/humanitys-last-exam) - [MATH-500 Benchmark Leaderboard by Artificial Analysis](https://artificialanalysis.ai/evaluations/math-500) - [MMMU-Pro Benchmark Leaderboard by Artificial Analysis](https://artificialanalysis.ai/evaluations/mmmu-pro) - [Artificial Analysis Speech Arena](https://artificialanalysis.ai/text-to-speech/arena) - [microsoft/autogen GitHub](https://github.com/microsoft/autogen) - [What is Azure Document Intelligence in Foundry Tools? by Microsoft](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/overview) - [BenchmarkQED documentation (AutoQ, AutoE, AutoD) by Microsoft](https://microsoft.github.io/benchmark-qed/) - [Berkeley Function Calling Leaderboard CHANGELOG by Gorilla LLM team, UC Berkeley](https://github.com/ShishirPatil/gorilla/blob/main/berkeley-function-call-leaderboard/CHANGELOG.md) - [Captum v0.9.0 release notes by Captum maintainers, meta-pytorch](https://github.com/meta-pytorch/captum/releases/tag/v0.9.0) - [Claude Code Docs. Code Review by Anthropic](https://code.claude.com/docs/en/code-review) - [Cohere's Embed Models (Details and Application)](https://docs.cohere.com/docs/cohere-embed) - [Unbabel COMET repository README](https://github.com/Unbabel/COMET) - [Confident AI Pricing](https://www.confident-ai.com/pricing) - [Confident AI homepage](https://www.confident-ai.com/) - [CoreWeave Completes Acquisition of Weights & Biases](https://investors.coreweave.com/news/news-details/2025/CoreWeave-Completes-Acquisition-of-Weights--Biases/default.aspx) - [Cornac: a comparative framework for multimodal recommender systems (GitHub repository and release notes) by PreferredAI](https://github.com/PreferredAI/cornac) - [datasketch v2.0.0 release notes by Eric Zhu and contributors](https://github.com/ekzhu/datasketch/releases/tag/v2.0.0) - [2026 changelog (2026) by Confident AI](https://deepeval.com/changelog/changelog-2026) - [DeepEval Docs. Eval Modes by Confident AI](https://deepeval.com/docs/evaluation-eval-modes) - [DeepEval Docs. Faithfulness metric by Confident AI](https://deepeval.com/docs/metrics-faithfulness) - [DeepEval Docs. JevEval by Confident AI](https://deepeval.com/docs/metrics-jev-eval) - [deepeval on PyPI by Confident AI](https://pypi.org/project/deepeval/) - [DeepResearch Bench Leaderboard (Hugging Face Space) by DeepResearch Bench maintainers (USTC, Metastone Technology)](https://huggingface.co/spaces/muset-ai/DeepResearch-Bench-Leaderboard) - [DeepResearch Bench II website and leaderboard by Agent Research Lab, USTC](https://agentresearchlab.org/benchmarks/deepresearch-bench-ii/index.html) - [DeepResearch-Bench-II GitHub repository (README news: evaluator migrated to GPT-5.5) by Li, R.; Du, M.; et al](https://github.com/imlrz/DeepResearch-Bench-II) - [DeepTeam: The LLM Red Teaming Framework by Confident AI](https://github.com/confident-ai/deepteam) - [defog-ai/sql-eval: SQL evaluation framework](https://github.com/defog-ai/sql-eval) - [DocVQA-2026 dataset card (VLR-CVC) (2026) by Artemis Llabres, Marc Serra Ortega, Tomas Ockier, et al](https://huggingface.co/datasets/VLR-CVC/DocVQA-2026) - [DocVQA 2026 Challenge: multimodal reasoning over documents in multiple domains (2026) by CVC-UAB DocVQA organizers](https://www.docvqa.org/challenges/2026) - [DocVQA Results, Task 1 Single Page Document VQA (Robust Reading Competition) by CVC-UAB, CVIT IIIT Hyderabad](https://rrc.cvc.uab.es/?ch=17&com=evaluation&task=1) - [Elliot: a comprehensive and rigorous framework for reproducible recommender systems evaluation (GitHub repository) by SisInfLab](https://github.com/sisinflab/elliot) - [EmbeddingGemma 300M model card by Google](https://huggingface.co/google/embeddinggemma-300m) - [FrontierMath: Tiers 1-4 (v2) by Epoch AI](https://epoch.ai/frontiermath/tiers-1-4) - [GPT-6 Astra (model page and benchmark scores) by Epoch AI](https://epoch.ai/models/gpt-6-astra) - [SWE-bench Verified: Benchmark review by Epoch AI](https://epoch.ai/benchmarks/swe-bench-verified/review) - [SWE-bench Pro: Benchmark review by Epoch AI](https://epoch.ai/benchmarks/swe-bench-pro/review) - [Commission starts enforcing AI Act rules and new transparency requirements on 2 August (press release IP/26/1714) by European Commission](https://ec.europa.eu/commission/presscorner/api/files/document/print/en/ip_26_1714/IP_26_1714_EN.pdf) - [evals-skills: skills for building product-specific AI evals by Shreya Shankar & Hamel Husain (ai-evals-course)](https://github.com/ai-evals-course/evals-skills) - [Data drift (default drift detection methods) by Evidently AI](https://docs.evidentlyai.com/metrics/explainer_drift) - [FActScore GitHub repository (factscore package) by Min et al](https://github.com/shmsw25/FActScore) - [FIX: center each sensitive feature by its own mean in CorrelationRemover (fairlearn PR #1732) by Fairlearn contributors](https://github.com/fairlearn/fairlearn/pull/1732) - [fix: raise on missing values in MetricFrame sensitive and control features (fairlearn PR #1698) by Edward Yi](https://github.com/fairlearn/fairlearn/pull/1698) - [fairlearn (PyPI package page and release history) by Fairlearn contributors](https://pypi.org/project/fairlearn/) - [GAIA Leaderboard (Hugging Face Space, results in gaia-benchmark/results_public) by GAIA benchmark maintainers](https://huggingface.co/spaces/gaia-benchmark/leaderboard) - [Introducing Luna-2: Purpose-Built Models for Reliable AI Evaluations & Guardrailing by Galileo](https://galileo.ai/blog/introducing-luna-2-purpose-built-models-for-reliable-ai-evaluations-guardrailing) - [Use Luna-2 in Your Experiments by Galileo](https://docs.galileo.ai/how-to-guides/luna/experiments-with-luna/experiments-with-luna) - [Luna-2 Overview by Galileo](https://docs.galileo.ai/concepts/luna/luna) - [Experiment Metrics by Galileo](https://docs.galileo.ai/sdk-api/metrics/metrics) - [Galileo Pricing](https://galileo.ai/pricing) - [Protect (deprecated) by Galileo](https://docs.galileo.ai/concepts/protect/overview) - [Embeddings (Gemini API) by Google](https://ai.google.dev/gemini-api/docs/embeddings) - [Gemini API function calling by Google](https://ai.google.dev/gemini-api/docs/function-calling) - [Giskard homepage by Giskard AI](https://www.giskard.ai/) - [Open Source vs Hub by Giskard AI](https://docs.giskard.ai/start/comparison) - [Giskard Pricing by Giskard AI](https://www.giskard.ai/pricing) - [Giskard OSS releases by Giskard AI](https://github.com/Giskard-AI/giskard-oss/releases) - [Scan API by Giskard AI](https://docs.giskard.ai/oss/scan/reference/scan-api) - [Scan Vulnerabilities by Giskard AI](https://docs.giskard.ai/oss/solutions/scan-vulnerabilities) - [Using secrets in GitHub Actions by GitHub](https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions) - [Safer pull_request_target defaults for actions/checkout by GitHub](https://github.blog/changelog/2026-06-18-safer-pull_request_target-defaults-for-github-actions-checkout/) - [Using GitHub Copilot code review by GitHub Docs](https://docs.github.com/en/copilot/using-github-copilot/code-review/using-copilot-code-review) - [Deprecation of Node 20 on GitHub Actions runners by GitHub](https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/) - [Gemini API release notes by Google](https://ai.google.dev/gemini-api/docs/changelog) - [Gemini Developer API pricing by Google](https://ai.google.dev/gemini-api/docs/pricing) - [Structured outputs (Gemini API docs) by Google](https://ai.google.dev/gemini-api/docs/structured-output) - [GraphRAG documentation by Microsoft](https://microsoft.github.io/graphrag/) - [Microsoft GraphRAG (GitHub repository and README)](https://github.com/microsoft/graphrag) - [gte-multilingual-base model card by Alibaba-NLP](https://huggingface.co/Alibaba-NLP/gte-multilingual-base) - [evals-skills (AI Evals course) by Hamel Husain & Shreya Shankar (ai-evals-course)](https://github.com/ai-evals-course/evals-skills) - [HELM leaderboards by Stanford CRFM](https://crfm.stanford.edu/helm/) - [HELM Maintenance Mode Policy by Stanford CRFM](https://crfm-helm.readthedocs.io/en/latest/maintenance_mode/) - [HELM GitHub repository by Stanford CRFM](https://github.com/stanford-crfm/helm) - [Open LLM Leaderboard (Hugging Face Space) by Hugging Face](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) - [HoneyHive Docs: Platform architecture](https://docs.honeyhive.ai/v2/platform-architecture) - [HoneyHive Docs: Self-hosted](https://docs.honeyhive.ai/v2/setup/self-hosted) - [HoneyHive Pricing](https://www.honeyhive.ai/pricing) - [HoneyHive Docs: Product changelog](https://docs.honeyhive.ai/v2/changelog/product.md) - [honeyhive (PyPI)](https://pypi.org/project/honeyhive/) - [HoneyHive Docs: Client releases](https://docs.honeyhive.ai/v2/changelog/clients.md) - [Inspect AI CHANGELOG by UK AI Security Institute, Meridian Labs](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/CHANGELOG.md) - [Inspect Evals register by Generality Labs, UK AI Security Institute, Arcadia Impact, Vector Institute](https://ukgovernmentbeis.github.io/inspect_evals/) - [Inspect Scout documentation by Meridian Labs](https://meridianlabs-ai.github.io/inspect_scout/) - [Kiro Docs: Analyze Requirements by Kiro, AWS](https://kiro.dev/docs/specs/analyze-requirements/) - [Kiro Docs: Correctness with Property-based tests by Kiro, AWS](https://kiro.dev/docs/specs/correctness/) - [krippendorff (fast-krippendorff) package on PyPI](https://pypi.org/project/krippendorff/) - [TypeSafe integration by LangChain Docs](https://docs.langchain.com/oss/python/integrations/providers/typesafe) - [langfuse-python: client.py (Langfuse client, create_dataset, create_dataset_item)](https://github.com/langfuse/langfuse-python/blob/main/langfuse/_client/client.py) - [Langfuse GitHub releases](https://github.com/langfuse/langfuse/releases) - [LIME: Explaining the predictions of any machine learning classifier (GitHub repository) by Marco Tulio Ribeiro](https://github.com/marcotcr/lime) - [LiveCodeBench repository README (dataset release versions) by LiveCodeBench authors](https://github.com/LiveCodeBench/LiveCodeBench) - [LiveSQLBench: a dynamic, contamination-free benchmark for real-world text-to-SQL by BIRD Team, HKU and Google Cloud](https://livesqlbench.ai/) - [remaining cleanup, uv lock bump (PR #21294) by Logan Markewich](https://github.com/run-llama/llama_index/pull/21294) - [LlamaParse Pricing by LlamaIndex](https://www.llamaindex.ai/pricing) - [Usage Pattern (Retrieval) by LlamaIndex](https://developers.llamaindex.ai/python/framework/module_guides/evaluating/usage_pattern_retrieval/) - [lm-evaluation-harness API Guide by EleutherAI](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/API_guide.md) - [lm-evaluation-harness GitHub repository and README by EleutherAI](https://github.com/EleutherAI/lm-evaluation-harness) - [lm-evaluation-harness v0.4.13 release notes by EleutherAI](https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.13) - [lmms-eval (GitHub) by EvolvingLMMs-Lab](https://github.com/EvolvingLMMs-Lab/lmms-eval) - [LongBench v2 leaderboard by Bai et al., THUDM](https://longbench2.github.io/) - [LongMemEval GitHub by Di Wu et al](https://github.com/xiaowu0162/LongMemEval) - [Code Review Benchmark (GitHub) by Martian](https://github.com/withmartian/code-review-benchmark) - [MathArena: Evaluating LLMs on Uncontaminated Math Competitions](https://matharena.ai/) - [Model Context Protocol specification 2026-07-28: Key Changes (2026) by Model Context Protocol maintainers](https://modelcontextprotocol.io/specification/2026-07-28/changelog) - [Model Context Protocol Specification, revision 2026-07-28 (2026) by Model Context Protocol project](https://modelcontextprotocol.io/specification/2026-07-28) - [Model Context Protocol specification 2026-07-28: Tools (2026) by Model Context Protocol maintainers](https://modelcontextprotocol.io/specification/2026-07-28/server/tools) - [MCP-Universe leaderboard by Salesforce AI Research](https://mcp-universe.github.io/#results) - [MCPMark GitHub (eval-sys) by EVAL SYS, NUS TRAIL, LobeHub](https://github.com/eval-sys/mcpmark) - [Task-Completion Time Horizons by METR](https://metr.org/time-horizons/) - [Task-Completion Time Horizons of Frontier AI Models by METR](https://metr.org/time-horizons/) - [Microsoft Agent Framework Workflows Orchestrations: Magentic](https://learn.microsoft.com/en-us/agent-framework/workflows/orchestrations/magentic) - [Content filtering for Microsoft Foundry Models (classic) by Microsoft](https://learn.microsoft.com/en-us/azure/foundry-classic/foundry-models/concepts/content-filter) - [MLCommons AILuminate (GitHub repository and README)](https://github.com/mlcommons/ailuminate) - [MLCommons ModelBench](https://github.com/mlcommons/modelbench) - [Dataset Tracking by MLflow Docs](https://mlflow.org/docs/latest/ml/dataset/) - [Release live: MCV Scripted Speech v26.0 and Spontaneous Speech v4.0 by Mozilla Common Voice](https://discourse.mozilla.org/t/release-live-mcv-scripted-speech-v26-0-and-spontaneous-speech-v4-0/148687) - [MTEB issue 3934: RTEB private column removal announcement by MTEB maintainers](https://github.com/embeddings-benchmark/mteb/issues/3934) - [NoCha Leaderboard: long-context claim verification over fiction books by Karpinska et al](https://novelchallenge.github.io/) - [NoLiMa: Long-Context Evaluation Beyond Literal Matching (GitHub README and results table) by Adobe Research](https://github.com/adobe-research/NoLiMa) - [Qwen3-Nemotron-32B-GenRM-Principle model card by Nvidia](https://huggingface.co/nvidia/Qwen3-Nemotron-32B-GenRM-Principle) - [Open Bandit Pipeline (GitHub repository) by ZOZO Inc., st-tech](https://github.com/st-tech/zr-obp) - [FLORES+ dataset card by Open Language Data Initiative, OLDI](https://huggingface.co/datasets/openlanguagedata/flores_plus) - [olmOCR repository (olmOCR-Bench) by Allen Institute for AI](https://github.com/allenai/olmocr) - [OmniDocBench: benchmark for diverse document parsing (2025) by OpenDataLab (Ouyang et al., CVPR 2025)](https://github.com/opendatalab/OmniDocBench) - [Opik Docs: Changelog by Comet](https://www.comet.com/docs/opik/changelog) - [Opik Docs: Evaluation concepts by Comet](https://www.comet.com/docs/opik/evaluation/concepts) - [Opik Docs: Evaluate your LLM application by Comet](https://www.comet.com/docs/opik/evaluation/evaluate_your_llm) - [Opik Docs: Ollie by Comet](https://www.comet.com/docs/opik/ollie) - [OpenTelemetry semantic-conventions v1.42.0 release notes](https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.42.0) - [Patronus AI homepage](https://www.patronus.ai/) - [PyPI Stats: instructor by PyPI Stats, pypistats.org](https://pypistats.org/packages/instructor) - [Qwen3-Embedding-8B model card by Qwen Team](https://huggingface.co/Qwen/Qwen3-Embedding-8B) - [Ragas source. collections/answer_correctness/metric.py by Ragas maintainers](https://github.com/vibrantlabsai/ragas/blob/main/src/ragas/metrics/collections/answer_correctness/metric.py) - [Ragas source: src/ragas/evaluation.py by Ragas maintainers](https://github.com/vibrantlabsai/ragas/blob/main/src/ragas/evaluation.py) - [ragas on PyPI by Ragas maintainers](https://pypi.org/project/ragas/) - [Ragas source. testset/synthesizers/__init__.py (default_query_distribution), v0.4.3 by Exploding Gradients](https://github.com/explodinggradients/ragas/blob/v0.4.3/src/ragas/testset/synthesizers/__init__.py) - [Humanity's Last Exam Leaderboard by Scale Labs](https://labs.scale.com/leaderboard/humanitys_last_exam) - [Scale Labs Leaderboard: MultiChallenge by Scale AI](https://labs.scale.com/leaderboard/multichallenge) - [Probability calibration (scikit-learn user guide) by The scikit-learn developers](https://scikit-learn.org/stable/modules/calibration.html) - [scipy.stats.false_discovery_control](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.false_discovery_control.html) - [WebArena-Verified (ServiceNow) by Amine El hattami, Megh Thakkar, Nicolas Chapados, Christopher Pal](https://github.com/ServiceNow/webarena-verified) - [tau-bench leaderboard (taubench.com) by Sierra Research](https://taubench.com) - [sklearn.metrics.cohen_kappa_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html) - [Cost considerations for Cortex AI Functions by Snowflake](https://docs.snowflake.com/en/user-guide/snowflake-cortex/aisql-cost) - [Spec Kit Documentation: Agentic SDD command reference by GitHub, Spec Kit maintainers](https://github.github.io/spec-kit/reference/agentic-sdd.html) - [Spider 2.0 benchmark site and leaderboard by XLANG Lab](https://spider2-sql.github.io/) - [Galileo by Splunk](https://www.splunk.com/en_us/about-splunk/acquisitions/galileo.html) - [statsmodels.stats.multitest.multipletests](https://www.statsmodels.org/stable/generated/statsmodels.stats.multitest.multipletests.html) - [tau3-bench v1.0.0 release: Voice, Knowledge, Task Quality by Sierra Research](https://github.com/sierra-research/tau2-bench/releases/tag/v1.0.0) - [tau-bench site and leaderboard by Sierra Research](https://taubench.com) - [Tonic Validate releases by Tonic AI](https://github.com/TonicAI/tonic_validate/releases) - [Toolathlon leaderboard (Toolathlon-Verified) by Toolathlon authors](https://toolathlon.xyz/) - [TruLens GitHub, changes from 2.12.0 to 2.13.0 by TruLens contributors](https://github.com/truera/trulens/compare/trulens-2.12.0...trulens-2.13.0) - [TruLens source. feedback/llm_provider.py (groundedness and citation_attribution) by TruLens maintainers](https://github.com/truera/trulens/blob/main/src/feedback/trulens/feedback/llm_provider.py) - [OTLP-native export + OTel metrics signal (pull request #2681) by TruLens contributors](https://github.com/truera/trulens/pull/2681) - [TruLens GitHub releases by TruLens contributors, Snowflake](https://github.com/truera/trulens/releases) - [TypeSafe Docs. Confidence by TypeSafe AI](https://docs.typesafe.ai/confidence) - [TypeSafe Docs. Jev 1.13 jaggedness by TypeSafe AI](https://docs.typesafe.ai/model-jaggedness/jev-1.13) - [Introducing System One Models & Jev by Diogo Almeida, TypeSafe AI](https://typesafe.ai/blog/introducing-system-one-models-and-jev) - [TypeSafe Docs. Models by TypeSafe AI](https://docs.typesafe.ai/models) - [TypeSafe Docs. Score by TypeSafe AI](https://docs.typesafe.ai/primitives/score) - [TypeSafe Docs. Classifying RAG passages by TypeSafe AI](https://docs.typesafe.ai/cookbooks/classifying_rag_passages) - [TypeSafe Docs. System One by TypeSafe AI](https://docs.typesafe.ai/concepts/system-one) - [UpTrain on PyPI](https://pypi.org/project/uptrain/) - [UpTrain releases by UpTrain AI](https://github.com/uptrain-ai/uptrain/releases) - [Vanna: text-to-SQL agent framework (repository archived 2026-03-29) (2026) by Vanna AI](https://github.com/vanna-ai/vanna) - [Vectara Hallucination Leaderboard](https://github.com/vectara/hallucination-leaderboard) - [Vellum homepage](https://www.vellum.ai/) - [AI SDK 7 is now available by Vercel](https://vercel.com/blog/ai-sdk-7) - [Vitest 5.0 is out! by Vitest team](https://vitest.dev/blog/vitest-5) - [Weave Docs: Evaluations by Weights & Biases / CoreWeave](https://docs.coreweave.com/weave/guides/core-types/evaluations) - [Weave Docs: Monitors by Weights & Biases / CoreWeave](https://docs.coreweave.com/weave/guides/evaluation/monitors) - [WebArena leaderboard (community Google Sheet) by WebArena authors and contributors](https://docs.google.com/spreadsheets/d/1M801lEpBbKSNwP-vDBkC_pF7LdyGU1f_ufZb_NWNBZQ/edit?usp=sharing) - [WMT26 Shared Task: General Machine Translation by WMT26 organizers, Conference on Machine Translation](https://www2.statmt.org/wmt26/translation-task.html) - [Releases by MLflow](https://github.com/mlflow/mlflow/releases) - [mlflow 3.17.1 on PyPI by MLflow](https://pypi.org/project/mlflow/) - [GitHub repository by MLflow](https://github.com/mlflow/mlflow) - [Evaluating GenAI apps by MLflow Docs](https://mlflow.org/docs/latest/genai/eval-monitor/) - [MLflow joins the Linux Foundation by Databricks](https://www.databricks.com/blog/2020/06/25/mlflow-joins-the-linux-foundation-to-become-the-open-standard-for-machine-learning-platforms.html) - [Pydantic Evals Docs by Pydantic](https://pydantic.dev/docs/ai/evals/evals/) - [pydantic_evals README by Pydantic](https://github.com/pydantic/pydantic-ai/tree/main/pydantic_evals) - [pydantic-evals by PyPI](https://pypi.org/project/pydantic-evals/) - [pydantic-ai LICENSE (MIT) by Pydantic](https://github.com/pydantic/pydantic-ai/blob/main/LICENSE) - [pydantic-ai v2.55.0 release notes by Pydantic](https://github.com/pydantic/pydantic-ai/releases/tag/v2.55.0) - [pydantic-ai v2.56.0 release notes by Pydantic](https://github.com/pydantic/pydantic-ai/releases/tag/v2.56.0) - [pydantic-ai v2.57.0 release notes by Pydantic](https://github.com/pydantic/pydantic-ai/releases/tag/v2.57.0) - [Logfire Pricing by Pydantic](https://pydantic.dev/pricing) - [Harbor GitHub repository by Harbor Framework](https://github.com/harbor-framework/harbor) - [Harbor documentation by Harbor Framework](https://docs.harborframework.com/) - [Harbor releases by Harbor Framework](https://github.com/harbor-framework/harbor/releases) - [harbor on PyPI by Harbor Framework](https://pypi.org/project/harbor/) - [Terminal-Bench GitHub repository by Harbor Framework](https://github.com/harbor-framework/terminal-bench) - [garak releases by NVIDIA](https://github.com/NVIDIA/garak/releases) - [garak on PyPI by Python Package Index](https://pypi.org/project/garak/) - [garak reference by NVIDIA](https://reference.garak.ai) - [European Commission AI Act Service Desk. Implementation timeline](https://ai-act-service-desk.ec.europa.eu/en/ai-act/timeline/timeline-implementation-eu-ai-act) - [Microsoft Responsible AI hub](https://www.microsoft.com/en-us/ai/responsible-ai) - [MCP Security Notification: Tool Poisoning Attacks by Invariant Labs](https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks) - [AI threats in the wild: prompt injections on the web by Google Security](https://blog.google/security/prompt-injections-web/) - [Palo Alto Unit 42. Indirect Prompt Injection](https://unit42.paloaltonetworks.com/ai-agent-prompt-injection/) - [Comet prompt injection (account takeover) by Brave](https://brave.com/blog/comet-prompt-injection/) - [Claude for Chrome (injection mitigations) by Anthropic](https://claude.com/blog/claude-for-chrome) - [Prompt injection defenses by Anthropic](https://www.anthropic.com/research/prompt-injection-defenses) - [Matrix of adversary tactics and techniques against AI systems by MITRE ATLAS](https://atlas.mitre.org/) - [MITRE ATLAS data repository (content releases; v2026.09, 2026-09-15) (2026)](https://github.com/mitre-atlas/atlas-data) - [Threat Modeling Manifesto](https://www.threatmodelingmanifesto.org/) - [Threats: Microsoft Threat Modeling Tool (STRIDE) by Microsoft Learn](https://learn.microsoft.com/en-us/azure/security/develop/threat-modeling-tool-threats) - [Agents Rule of Two: A Practical Approach to AI Agent Security by Meta AI](https://ai.meta.com/blog/practical-ai-agent-security/) - [SB-53 Artificial intelligence models: large developers (Transparency in Frontier Artificial Intelligence Act) by California State Legislature](https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260SB53) - [EN 18286 in the Spotlight: Supporting Compliance with the AI Act (EN 18286:2026 Artificial intelligence - Quality management system for EU AI Act regulatory purposes) (2026) by CEN-CENELEC](https://www.cencenelec.eu/news-events/news/2026/en-in-the-spotlight/2026-07-30-ai-quality-management/) - [Rescission of Uniform Guidelines on Employee Selection Procedures (RIN 3046-AB43) by US Equal Employment Opportunity Commission](https://www.reginfo.gov/public/do/eAgendaViewRule?pubId=202510&RIN=3046-AB43) - [Article 73: Reporting of serious incidents by European Union, AI Act Service Desk](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-73) - [AI Act Article 99: Penalties (consolidated text) by European Commission, AI Act Service Desk](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-99) - [Regulation (EU) 2026/1744 amending Regulations (EU) 2024/1689, (EU) 2018/1139 and (EU) 2023/1230 as regards the simplification of the implementation of harmonised rules on artificial intelligence (Digital Omnibus on AI), adopted 8 July 2026 (2026) by European Parliament, Council of the European Union](https://eur-lex.europa.eu/eli/reg/2026/1744/oj) - [RFC 9562: Universally Unique IDentifiers (UUIDs) by Kyzer Davis, Brad Peabody, Paul Leach](https://www.rfc-editor.org/rfc/rfc9562.html) - [ISO/IEC 42001:2023 Information technology - Artificial intelligence - Management system (2023) by ISO/IEC JTC 1/SC 42, Artificial intelligence](https://www.iso.org/standard/42001) - [ISO/IEC 42006:2025 Information technology - Artificial intelligence - Requirements for bodies providing audit and certification of artificial intelligence management systems (2025) by ISO/IEC JTC 1/SC 42, Artificial intelligence](https://www.iso.org/standard/42006) - [Removal of References to the Uniform Guidelines on Employee Selection Procedures in Federal Personnel Regulations (interim final rule, 91 FR 48234, RIN 3206-AP20) by US Office of Personnel Management](https://www.govinfo.gov/content/pkg/FR-2026-07-31/html/2026-15586.htm) - [Binomial Proportion CI](https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval) - [Cohen's kappa](https://en.wikipedia.org/wiki/Cohen%27s_kappa) - [Krippendorff's alpha](https://en.wikipedia.org/wiki/Krippendorff%27s_alpha) - [McNemar's test](https://en.wikipedia.org/wiki/McNemar%27s_test) - [MinHash](https://en.wikipedia.org/wiki/MinHash) - [Grounded theory by Wikipedia contributors](https://en.wikipedia.org/wiki/Grounded_theory) - [LMArena by Wikipedia](https://en.wikipedia.org/wiki/LMArena) - [Statistical process control by Wikipedia](https://en.wikipedia.org/wiki/Statistical_process_control) - [AI Evals (Hamel & Shreya) by Maven course](https://maven.com/parlance-labs/evals) - [Automated Testing for LLMOps by DLAI](https://www.deeplearning.ai/short-courses/automated-testing-llmops/) ## How to cite this site If you reproduce material from AI Evals in an answer, please cite the source URL and the author. APA shape: Bharatiya, O. (2026). AI Evals: The most useful, opinionated reference for AI evals. https://www.aievals.co Content license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ## Optional - [llms-full.txt](https://www.aievals.co/llms-full.txt): full plain text of every page, separated by --- markers - [Sitemap](https://www.aievals.co/sitemap.xml): every indexable URL - [RSS](https://www.aievals.co/rss.xml): recently updated pages - [Atom](https://www.aievals.co/atom.xml): Atom 1.0 mirror - [Papers RSS](https://www.aievals.co/papers/rss.xml): only paper-theme updates