Documentation discovery benchmark
On 2026-10-19, we ran the same 20 hand-curated queries once against public API versions 0.7.0 and 0.8.0. Expected-provider matches rose from 8 to 15; expected source-URL matches rose from 5 to 6. This measures returned URLs, not successful agent tasks.
| Measure | 0.7.0 baseline | 0.8.0 later run |
|---|---|---|
| Expected source URL in first result | 5/20 (25%) | 6/20 (30%) |
| Expected source URL in first three | 5/20 (25%) | 6/20 (30%) |
| Expected provider selected | 8/20 (40%) | 15/20 (75%) |
| HTTP 404 / no match | 2/20 | 0/20 |
| API marked stale / freshness unknown | 14 / 2 | 19 / 0 |
| Client latency, median / p95 | 28.49 ms / 124.56 ms | 30.71 ms / 70.37 ms |
| Mean response body | 3046.75 bytes | 3111.75 bytes |
The runs began at 17:54 and 18:06 UTC against a changing production index, without a frozen database or controlled caches. Code and available data can both affect the difference; these runs cannot isolate the algorithm's contribution.
What needs work
The later run selected Stripe's own webhook documentation instead of a third-party integration. Provider selection also changed to Stripe, Supabase, Cloudflare and Next.js for several queries, but the right provider alone is insufficient: stripe idempotent requests, supabase authentication and nextjs middleware still returned no task-specific source links. The row-level security query returned a generic security URL rather than the expected RLS path.
Astro routing, OpenAI function calling and Anthropic tool-use queries still selected other publishers. The later resolver marked those candidates weak. Overall, 14 of 20 queries still failed the fixed source-URL rules. Read all results, including misses, before deciding whether this service fits a workflow.
The evaluator also has limitations: its Clerk token rule counted an account dashboard, while Cloudflare KV and R2 product URLs failed narrow path rules. Open-topic questions returned potentially relevant Render and Loopwork URLs outside the hand-picked expected providers. We preserved these results and rules rather than adjusting them to raise the score.
How to interpret this
Each run made one sequential pass with three sources requested, compact responses, a 24-hour freshness threshold and no refresh. Neither fetched destination pages. In the later run, 19 results were marked stale, one was within the threshold, and none had unknown freshness. Stored checks can exceed the requested freshness window. The threshold does not request a refresh or guarantee freshness. Latency includes this client's network and is not an availability guarantee. Stale does not prove a source is wrong.
These 20 developer-oriented tasks are hand-picked, not a representative sample of the index. A host and path match does not establish answer correctness, working code or agent task completion. There is no competing-service or ordinary-search baseline. Both runs contribute test traffic to our API counters and are not evidence of independent adoption.
Read every 0.7.0 attempt · Read every 0.8.0 attempt · Inspect the fixed task set · Reproduce the runs and inspect limitations