# Iris — stop shipping agents on vibes

> Iris is an open-source MCP server that scores AI agent output quality, catches safety failures (PII leaks, prompt injection, hallucination markers), and enforces cost budgets. Any MCP client lists its tools on connect — one command (`npx -y @iris-eval/mcp-server@0.20.0`), no SDK; your agent calls them, a host hook forwards each turn, or `iris-eval ingest` loads a trace file. MIT-licensed core, self-hosted, signed releases.

Current release: v0.20.0 (2026-10-03) — Iris says what it checked. A verdict tells a pass from a check that never ran, names who recorded the evidence it judged, and cannot be replaced by the agent it judges.

Iris registers 12 MCP tools (trace logging, heuristic eval, LLM-as-judge, semantic citation verification, rule lifecycle) and ships 26 built-in eval rules across completeness, relevance, safety, and cost. 10 of those rules read the agent's own tool calls and judge what it DID, not only what it wrote: a failed call the answer never acknowledges, a repeated one, a call its own tool's schema rejects, a file or URL cited that appears in nothing the agent read, an instruction obeyed out of a tool result, and a step budget. A ninth custom rule type, action_policy, states which tools an agent may call and with what arguments. All of it is deterministic and runs locally — no model call, nothing leaves the machine. It is verified on every CI run in Claude Code, and Gemini CLI; for Claude Desktop, Cursor, Devin Desktop (Windsurf), Continue, VS Code, Cline, Zed, and OpenAI Codex CLI the installer writes the configuration shape each client documents and that writer is tested on the shape, with nobody on the Iris side having watched the client connect (claimed — every row with its source and date at https://iris-eval.com/clients); and it runs in any other MCP client the same way.

Evaluator accuracy is published at https://iris-eval.com/proof: precision, recall and F1 with 95% confidence intervals for built-in rules — 13 measured for detection against labels a model gave by reading the failure (synthetic, model-labelled corpus; a human blind label is pending), and 13 checked against their own documented formula (a score there shows the code implements the formula, not that it detects the failure) — corpus 7c27824a2b98, generated 2026-10-04 from the source these numbers were measured on; reproduce with `npm run proof`.

Of 60 capability cells (10 evaluation questions by 6 subjects), 34 are answered by a shipped, measured thing, 18 are answered with a stated limit, 4 are open gaps and 4 do not apply — every answered cell names the rule, tool, resource, route, proof row or judge template behind it. The full map: https://iris-eval.com/capabilities.

## Getting Started

- [GitHub repository](https://github.com/iris-eval/mcp-server): source, README with per-client setup for every major MCP client, full tool schemas
- Claude Code plugin: `/plugin marketplace add iris-eval/mcp-server` then `/plugin install iris-eval@iris-eval` — MCP server + agent-eval skill in two commands
- [npm package](https://www.npmjs.com/package/@iris-eval/mcp-server): `@iris-eval/mcp-server` — runs via `npx -y @iris-eval/mcp-server@0.20.0`; `npx -y @iris-eval/mcp-server@0.20.0 install <client>` writes the config for one of: claude-code, claude-desktop, cursor, windsurf, continue, vscode, cline, zed, codex, gemini
- [Playground](https://iris-eval.com/playground): try agent eval in the browser, no install
- [Proof](https://iris-eval.com/proof): how accurate the built-in evaluators are — method, per-rule results with confidence intervals, and the command to reproduce them
- [Capabilities](https://iris-eval.com/capabilities): what Iris can judge and what it cannot yet — ten questions by six subjects, each answered cell naming its evidence
- [Security](https://iris-eval.com/security): security posture, signed releases, disclosure policy, measured issue-close times
- [OTel recipes](https://github.com/iris-eval/mcp-server/blob/main/docs/otel-recipes.md): one recipe per framework — Pydantic AI, Google ADK, LangGraph via LangSmith, CrewAI via OpenInference, the OpenAI Agents SDK (Python and JavaScript), LlamaIndex, AutoGen, Microsoft Agent Framework, Semantic Kernel, Vercel AI SDK, Mastra — the vendor's own setup lines with Iris as the OTLP endpoint, what Iris reads from each vocabulary, and the fixture in the repository that proves it

## Learn — Agent Eval Concepts

- [What is Agent Eval](https://iris-eval.com/learn/agent-eval): the category definition
- [Eval Gap](https://iris-eval.com/learn/eval-gap): the distance between what you test and what your agents do
- [Eval Tax](https://iris-eval.com/learn/eval-tax): the cost of not evaluating
- [Eval Drift](https://iris-eval.com/learn/eval-drift): silent quality decay over time
- [Eval Coverage](https://iris-eval.com/learn/eval-coverage): how much of agent behavior your rules observe
- [Eval Loop](https://iris-eval.com/learn/eval-loop): the improvement cycle
- [Eval-Driven Development](https://iris-eval.com/learn/eval-driven-development): evals as the spec
- [Output Quality Score](https://iris-eval.com/learn/output-quality-score): how scoring works
- [Self-Calibrating Eval](https://iris-eval.com/learn/self-calibrating-eval): rules that adapt

## Compare

- [Iris vs Langfuse](https://iris-eval.com/compare/langfuse)
- [Iris vs LangSmith](https://iris-eval.com/compare/langsmith)
- [Iris vs Helicone](https://iris-eval.com/compare/helicone)
- [Iris vs Braintrust](https://iris-eval.com/compare/braintrust)
- [Iris vs Arize](https://iris-eval.com/compare/arize)
- [Iris vs DeepEval](https://iris-eval.com/compare/deepeval)
- [Iris vs Confident AI](https://iris-eval.com/compare/confident-ai)
- [Iris vs Patronus AI](https://iris-eval.com/compare/patronus-ai)
- [Iris vs Promptfoo](https://iris-eval.com/compare/promptfoo)
- [Iris vs Galileo](https://iris-eval.com/compare/galileo)
- [Iris vs Opik](https://iris-eval.com/compare/opik)
- [Iris vs Weave](https://iris-eval.com/compare/weave)
- [Iris vs Judgment Labs](https://iris-eval.com/compare/judgment)
- [Iris vs Latitude](https://iris-eval.com/compare/latitude)

## Optional

- [Blog](https://iris-eval.com/blog): release notes, tutorials, agent eval essays
- [Pricing](https://iris-eval.com/pricing)
- [MCP Registry entry](https://registry.modelcontextprotocol.io): `io.github.iris-eval/mcp-server`
- [Docker image](https://github.com/iris-eval/mcp-server/pkgs/container/mcp-server): `ghcr.io/iris-eval/mcp-server`
- [llms-full.txt](https://iris-eval.com/llms-full.txt): expanded version of this file with tool reference and setup snippets