Written by Trevor Indrek Lasn @ trevorlasn.com --- title: I rebuilt 0xinsider around Polymarket sports and esports data description: 0xinsider is now Polymarket analytics for sports and esports. What 149,700 wallets and 411,770 large buys say, with every query public. topics: ['prediction markets', 'tech'] --- # I rebuilt 0xinsider around Polymarket sports and esports data I'm building [0xinsider](https://0xinsider.com) because I love data, and Polymarket has some of the best public data I've worked with. Every trade is tied to a wallet. Every fill settles on chain. Every market ends in a hard yes or no. That's millions of decisions with money behind them and a known answer at the end. I don't know another dataset like it. 0xinsider started in February as a live feed of large trades on Polymarket and Kalshi. It's now analytics for Polymarket sports and esports markets, because sports markets settle every day, and a market that settles is a data point with an answer. Kalshi is gone. Its trades are anonymous, so there's no wallet to follow. Here's the dataset today: None of it arrives clean. Raw fills aren't positions. You have to match entries to exits, handle splits, merges and redemptions, and mark every open position to market every day. I got that wrong for a while. My first P&L rollup added up per-market rows, and on some wallets it overstated the total by 7 to 10 times, sometimes with the wrong sign. Realized P&L now comes from Polymarket's own position accounting, and I [wrote up](https://0xinsider.com/research/polymarket-leaderboard-pnl-recomputed) why two tools can disagree on the same wallet. The first thing I wanted to know was whether the prices are even wrong. Mostly, they aren't. Across 411,770 large sports buys worth $8.2 billion, people paid 60.6¢ on average and their side won 60.7% of the time. That's a miss of two hundredths of a point. The one range that clearly loses is under 10¢, where long shots paid 6.5¢ and won 3% of the time. Sports turned out to be the well-priced part of Polymarket. Large buys outside sports miss their price by 6 to 10 points in most ranges. So if the prices are that good, who's making money? Not most people. Of the 36,149 wallets with 20 or more settled sports markets, 48.8% are in profit and the median wallet is down $8. The top 1%, 362 wallets, hold 68% of all the profit. The bottom 10% lost $401 million between them. That gap is what 0xinsider's grades try to measure. Every tracked wallet gets a grade from S to F, rebuilt every day from its settled P&L, and realized profit is about 95% of it. A grade like that is easy to compute and easy to fool yourself with. Win rate is the classic trap. A wallet that only buys at 85¢ wins about 85% of the time and has shown no skill at all. The number that holds up is the gap between how often your side won and the price you paid for it. So I tested it on 67,531 buys of $10,000 or more, using the grade each wallet already held on the day it traded, before the market settled. S, A and B wallets beat their price by 1.57 points. D and F wallets missed it by 2.06. C wallets and ungraded ones were indistinguishable from the market. The intervals are bootstrapped over markets rather than trades, because four wallets buying the same side of the same game are one observation. 1.57 points is small, and it should be. On a market this well priced, a study that found 10 points would be describing a bug in its own method. My favorite detail is that the sample keeps growing. Every trade in it sits on a market that already settled, so the universe gets bigger whenever an open market resolves. Three runs on the day I published returned 67,516, then 67,517, then 67,531 trades. The edge came out at 1.57 points all three times. Every query, raw output and bootstrap is [public](https://github.com/0xinsider/research). This one rebuilds the intervals above from a committed CSV, with no database and nothing to install past Python: ```bash git clone https://github.com/0xinsider/research && cd research/grade-vs-price && python3 bootstrap.py ``` About 20 seconds later: ``` cohort trades markets edge_pts 95% CI S/A/B 24610 5946 1.57 [+0.32, +2.82] C 9223 1986 0.53 [-1.93, +2.81] D/F 17106 4476 -2.06 [-3.61, -0.45] no grade 16592 5250 -0.13 [-1.81, +1.57] ``` On 0xinsider, all of this becomes a board. Every sports and esports game shows the moneyline on each side next to how much money from profitable wallets is sitting on it, live, across the NFL, NBA, MLB, NHL, soccer, tennis, UFC, CS2, LoL, Dota 2 and Valorant. Every wallet has a profile with its grade, P&L and the rest of its metrics. The boards, grades and profiles are free. If you'd rather pull the data yourself, there's a read-only REST API, an MCP server, and SDKs for Python and Go. The [research library](https://0xinsider.com/research) has the rest of the studies, and the [methodology page](https://0xinsider.com/transparency) says what every number is made from. None of this is financial advice. Every number here describes what already happened. --- --- title: I added Agentic Resource Discovery to 0xinsider so agents can find it description: ARD is a new proposal from Google, Microsoft and Hugging Face for how agents find tools. What's in it, what 0xinsider publishes, and what a registry returns. topics: ['llm', 'tech'] --- # I added Agentic Resource Discovery to 0xinsider so agents can find it Right now an agent only knows about the tools someone set up for it. You paste an MCP server into a config file, and that's its whole world. That works when there are five tools. It doesn't work when there are fifty thousand. [Agentic Resource Discovery](https://agenticresourcediscovery.org/spec/) is a proposal for fixing that. Version 0.91 came out on August 26, written by Junjie Bu at Google, R.V. Guha at Microsoft and Shaun Smith at Hugging Face, with contributors from GitHub, GoDaddy, Snowflake and others. Guha also worked on RSS and Schema.org, which is about the right résumé for this. The idea is to take tool discovery out of the model's context window and put it in a search service. A publisher puts a manifest at `/.well-known/ard.json`. A registry indexes it. An agent sends a plain-English question to the registry and gets back the entries that match, then connects to the one it needs. Nothing gets stuffed into the context up front. [0xinsider](https://0xinsider.com) has had a manifest for a while, and this week I rebuilt it to match v0.91. Here's the main entry: ```json { "identifier": "urn:air:0xinsider.com:mcp:prediction-market-intelligence", "displayName": "0xinsider MCP server", "type": "application/mcp-server-card+json", "url": "https://0xinsider.com/.well-known/mcp", "representativeQueries": [ "which Polymarket wallets placed the largest trades on tonight's NBA game", "grade this Polymarket wallet by settled P&L", "where is sharp money concentrated on this Premier League market", "show the Polymarket leaderboard for esports traders", "what sports games are on today and which Polymarket markets cover them" ] } ``` Four fields are required: `identifier`, `displayName`, `type`, and exactly one of `url` or `data`. The identifier is a URN anchored to your domain, `urn:air:::`. The type is a media type, so the same envelope can describe an MCP server, an [A2A agent](/blog/agent-2-agent-protocol-a2a) or a skill without ARD having to know anything about them. `representativeQueries` is the field I spent the most time on. It holds 2 to 5 things a person might actually ask, and it's what a registry builds its semantic index from. Writing them felt a lot like writing search keywords, except the reader is a model instead of Google. 0xinsider publishes nine entries: the product MCP server, the documentation MCP server, and seven agent skills covering things like wallet grades and large trades. The manifest carries a `trustManifest` that binds everything to `https://0xinsider.com`. The spec requires the publisher in every URN to match that identity, so a registry can check that an entry claiming to be 0xinsider really comes from 0xinsider.com. The official conformance tool is a single Python file with no dependencies: ```bash git clone https://github.com/ards-project/ard-spec && cd ard-spec python3 conformance/bin/conformance-test manifest https://0xinsider.com/.well-known/ard.json ``` It checks every entry's URN, the value-or-reference rule and the trust binding, then prints one verdict: ``` CONFORMANCE STATUS: PASS Validated with 0 critical specification errors and 0 warnings. ``` The first thing I got wrong had nothing to do with JSON. Our firewall challenges traffic that doesn't look like a browser, and verified crawlers like GPTBot get waved through. Registries aren't verified crawlers. So `/.well-known/ard.json` answered 429 to exactly the clients it exists for, while the older `ai-catalog.json` path right next to it answered 200. The fix was one line in a bypass rule. If you publish a manifest, fetch it from a server with curl, not from your own browser. Then I asked a registry about it. [ora.ai](https://ora.ai) runs one with the `POST /search` endpoint from the spec: ```bash curl -s -X POST https://ora.ai/api/ard/search \ -H 'Content-Type: application/json' \ -d '{"query":{"text":"Polymarket MCP server"},"pageSize":5}' ``` On September 23 it didn't return 0xinsider. Searching for "0xinsider" by name does find us, but as entries ora wrote itself, like `urn:air:ora.ai:product:0xinsider.com`, not the nine I published. The registry knows the site exists. It hasn't ingested the manifest yet. That's where ARD is today. The publishing side is ready: a manifest is a static JSON file, and the conformance tool tells you in a second if it's wrong. The search side, where registries crawl manifests and answer questions with them, is early. The spec still says "Proposal" at the top. I'm doing it anyway because 0xinsider is built for agents as much as for people. The data behind it is also a read-only REST API and an [MCP server](https://0xinsider.com/mcp). When someone asks an agent which Polymarket wallets are behind tonight's NBA game, I want it to find 0xinsider without anyone pasting a URL into a config file first. ARD is the first proposal I've seen that gets there without one company owning the directory. If you want to add one to your own site: - Write `/.well-known/ard.json` with one entry per thing an agent can use, and give each entry 2 to 5 real questions. - Run the conformance tool against the live URL. - Fetch it with curl from a server to make sure your firewall lets registries in. It's a small JSON file. The hard part is writing questions someone would actually ask. --- --- title: You have been invited to the 0xInsider.com Discord server! description: Join the 0xInsider.com Discord community — talk markets, odds, and predictions with fellow prediction market traders in real time. topics: ['prediction markets', 'tech'] --- # You have been invited to the 0xInsider.com Discord server! We just opened up a Discord server for prediction market enthusiasts, and we'd love for you to be part of it. It's a space to discuss market moves, debate odds, share forecasts, and swap strategies with other traders. Whether you want to dig into a specific market or just see what everyone's watching, there's a spot for you. ### What You'll Find Here - **Real-time market talk** — react to moves on Polymarket and Kalshi as they happen - **Odds and forecast debates** — argue your read, pressure-test someone else's - **Strategy swaps** — share what's working and learn what isn't - **Whale trade discussion** — break down the large trades [0xInsider](https://www.0xinsider.com) surfaces Whether you're deep into a specific market or just want to see what everyone's watching, there's a spot for you. ### Join Us Ready to talk markets, odds, and predictions with fellow traders? [Join the 0xInsider.com Discord](https://discord.gg/JxJsnbeYhn) See you inside! --- --- title: Ph.D in failure, Masters in getting back up description: Every time I bet on myself instead of taking a paycheck, and what each one taught me. topics: ['reflections'] --- # Ph.D in failure, Masters in getting back up I've failed a lot, more than most people would care to admit, and I'm proud of it. Everything below is something I bet on myself to build, and most of it didn't work. That's the point. Each failure taught me something a steady paycheck never could. My day jobs as an engineer and manager aren't on this list, only what I built for myself. **2010-2013 · pro gaming.** Semi-pro League of Legends. I started on top lane, then switched to jungle and climbed to around 2750 elo in Season 2, good enough to solo queue against pros and scrim with pros and semi-pros. I wanted to make it work, but the pieces never quite fit. Pro gaming was an opaque path back then with no clear way in, and I couldn't make a convincing case for it. My parents didn't see it as a real career, and I wasn't able to explain it well enough to change their minds. By Season 3 I put more into school, and that was that. It didn't pan out as a career, but that's around when I started learning to code. **Lesson:** talent without a path to get paid is just a hobby. I was early to a real industry, but being early without the conviction to back it is the same as being wrong. **2015-2022 · writing.** Writing tech articles, tutorials, and listicles, paid through the Medium Partner Program. That was the gig, and for a while it actually paid the bills. Then Medium lost ground in tech and the payouts dried up. The platform got flooded with low-effort, mass-produced filler, and the good writing drowned in the noise. **Lesson:** if you build your income on a platform you don't own, you're renting it. When Medium's economics changed, mine did too, and I had no vote. **2018 · startup 1.** A recipe site (lol). I wanted to learn Meteor.js and scratch my own itch: I wanted to cook more but never knew what to make, so I built a site to browse recipes. I shut it down not long after. The space was saturated with established brands that already owned the search rankings, and I had no edge to take them on. **Lesson:** know how it makes money before you build it. A content site needs enormous traffic just to clear minimum wage, and I'd confused shipping something with building a business. **2018 · startup 2.** Linktree, but for crypto addresses. I branded it at the time as one payment link for all your crypto: you put every address under a single link and shared that. I had no idea how to monetize it and wasn't business-savvy enough to figure it out. **Lesson:** a free utility anyone can clone is a feature, not a company. "Useful" and "fundable" are different questions, and I never asked the second. **2019 · startup 3.** A mashup of LinkedIn and Crunchbase. One place to look up a person and the company behind them at once: who works where, who raised what, who to reach out to. Handy for sales and recruiting on paper. In reality the data was a pain to keep fresh, I had no distribution, and I drifted off before working out who would actually pay for it. **Lesson:** reselling data you don't own is a treadmill, not a moat. The build was never the hard part. Distribution and a reason to pay were, and I'd skipped both. **2022 · startup 4.** Reddit, but for podcasts. You could submit podcast links, upvote and downvote them, and leave comments, so the best episodes would rise to the top instead of getting buried in someone's feed. I liked the idea of a community-curated front page for audio. It never reached the critical mass a voting site needs: with too few users there was nothing to vote on, and with nothing to vote on nobody stuck around. A classic cold-start problem, and I had no plan to solve it. **Lesson:** a community product is worthless until it's crowded, so the only question that matters is how you get the first thousand people. I had no answer. **2023 · startup 5.** Burned out on tech, so I pivoted to fashion design and launched two brands: a clothing label and a 3D-printed jewelry line. It actually went well for a while, but I shut both down. Not enough traction to justify keeping at it, and not enough passion to push through that. **Lesson:** work on something you care about deeply. Passion is the fuel for the unglamorous middle, and without it even a business that's working won't hold you. **2023 · startup 6.** A text-based dating site. I was in a relationship at the time, but I still thought there was a market for something more conversation-first than the swipe apps. I built it and then never launched it, because I didn't want my name attached to the dating scene. **Lesson:** if you won't put your name on it, you've already decided. I just figured that out after building it instead of before. **2024 · startup 7.** A hiring platform for developers. You could filter jobs by the tech stack you actually work in to find roles that fit, and build a profile so companies reached out to you instead of you applying to them. Too much competition in the space, and not enough passion on my end to outlast it. **Lesson:** entering a crowded market without an unfair edge is volunteering to lose slowly. "Better" isn't an edge, "different in a way that matters" is. **2025 · startup 8.** A dashboard for engineering managers to run their org. It tracked the metrics that actually tell you how a team is doing: cycle time, how long code takes to ship, PR review turnaround, deployment frequency, change failure rate, and lead time for changes. I gave up before launch. Somewhere along the way I realized I wasn't passionate about the problem, and forcing it felt worse than walking away. **Lesson:** only build problems you'd happily live inside for a decade, because you'll have to. I quit before launch, and quitting early beats quitting late. **2025 · startup 9.** You'd tell an AI what you wanted to learn, and it would point you to vetted resources and courses instead of the pile of low-effort tutorials out there. I built my own vector database stuffed with the good stuff to power the recommendations. I closed it because AI search got too good too fast: Perplexity, Claude, and Google now do the same thing well enough that there was no room to compete. **Lesson:** don't build in the path of where the giants are already heading. When your core feature is a roadmap item for Google and OpenAI, you're a placeholder, not a competitor. **2026 · startup 10.** [0xinsider](https://0xinsider.com), the intelligence layer for prediction markets. A coworker got me into it. In a random conversation she mentioned her husband builds Polymarket bots, and I wanted to see what all the fuss was about. I went down the rabbit hole and never came back. I'm still building it, I love it, and I plan to be at this one for many years. After everything above, this is the one I'm all in on. **Lesson:** everything before this is why it's different. A market I'm obsessed with, a clear way to get paid, an edge in the data, and the conviction to stay. The failures weren't detours, they were tuition. --- --- title: AEO and GEO for AI Overviews, ChatGPT, Claude, Gemini, and Perplexity description: What Answer Engine Optimization and Generative Engine Optimization mean, and how to get your site cited by AI Overviews, ChatGPT, Claude, Perplexity, and Gemini. topics: ['webdev', 'tech'] --- # AEO and GEO for AI Overviews, ChatGPT, Claude, Gemini, and Perplexity Search results don't look the way they did two years ago. Google now opens with AI Overviews, ChatGPT and Claude pull live web results into their answers, Perplexity built an entire product around it, and Gemini sits one tap away inside every Google surface. The page is no longer the destination. The page is a source the model is reading on your behalf. Two acronyms have shown up to describe the work of being visible inside those answers. **AEO** stands for Answer Engine Optimization, which is the work of being the source an "answer engine" uses when it returns a direct answer instead of a list of links. **GEO** stands for Generative Engine Optimization, which is the same idea framed around generative AI specifically: appearing inside answers a model writes from scratch using your page as a reference. Google's own [AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) treats both as variations of regular SEO. From their perspective, "optimizing for generative AI search is optimizing for the search experience, and thus still SEO." The ranking and quality systems that decide what shows up in a list of blue links are the same systems that decide what shows up inside an AI Overview. Improving for one improves the other. One reason this matters: each AI surface pulls from a different web index, but most of those indexes are downstream of the same crawl, rendering, and quality work. ``` Your page │ ▼ Search engine indexes (where the crawl lands) ┌────────────────────────────────────────────────────────┐ │ Google index ──▶ Google Search, AI Overviews, Gemini │ │ Bing index ──▶ Bing, Microsoft Copilot │ │ OpenAI index ──▶ ChatGPT Search │ │ Anthropic + Brave ▶ Claude web search │ │ Perplexity + Bing ▶ Perplexity answers │ └────────────────────────────────────────────────────────┘ │ ▼ The AI surface reads from the index, cites your page ``` The practical question is what you do to your site. The rest of this post walks through it, with sources for every recommendation. ### Eligibility comes before everything else Before any of the content work matters, the page has to be allowed to appear in AI features at all. Google's guide is explicit: a page is only eligible for AI features if it's eligible to appear as a regular search snippet ([source](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)). That means the URL needs to be indexed, the page needs to be crawlable in `robots.txt`, snippets need to be allowed (no `nosnippet`, no `max-snippet:0`), and the content has to load without requiring the crawler to execute heavy JavaScript first. Open Google Search Console and run a URL inspection on a page you care about. The "Test live URL" view shows you what Google sees, including the rendered HTML after JavaScript has executed. If the article body is missing from that rendered HTML, fix it before doing anything else. Google's [JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) covers the patterns that work and the ones that break crawling. Server rendering and static generation are the safest bets. A 30-second sanity check from your terminal, one per major AI crawler: ```bash # Google Search (feeds AI Overviews and Gemini grounding) curl -A "Googlebot/2.1 (+http://www.google.com/bot.html)" -I https://0xinsider.com/article # Bing (feeds Microsoft Copilot) curl -A "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" -I https://0xinsider.com/article # OpenAI search indexer (ChatGPT Search) curl -A "OAI-SearchBot/1.3; +https://openai.com/searchbot" -I https://0xinsider.com/article # Anthropic search indexer (Claude web search) curl -A "Claude-SearchBot/1.0 (+https://www.anthropic.com)" -I https://0xinsider.com/article # Perplexity indexer curl -A "PerplexityBot/1.0 (+https://perplexity.ai/perplexitybot)" -I https://0xinsider.com/article ``` A `200 OK` from a spoofed user agent isn't proof that the real crawler can reach the page. Bot operators block UA spoofing, so the only authoritative check is to verify the request against published IP ranges or reverse-DNS records. Google documents its [crawler verification process](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot), and OpenAI, Anthropic, and Perplexity all publish IP ranges in their bot docs. Use the `curl` test to catch obvious blocks (a `403`, `503`, or login-page redirect that suggests Cloudflare's bot-fight rule or a misconfigured WAF), then confirm against the official IP list for the bots that matter to you. The full list of bots worth thinking about, with what they do and the canonical reference: ``` Bot user agent Purpose Reference ────────────────────────────────────────────────────────────────────────── Googlebot Google Search index developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers Google-Extended Gemini Apps + Vertex AI Grounding + developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers AI training (separate from Search) Bingbot Bing index (feeds Copilot) bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0 GPTBot/1.3 OpenAI training developers.openai.com/api/docs/bots OAI-SearchBot/1.3 ChatGPT Search live index developers.openai.com/api/docs/bots ChatGPT-User ChatGPT user-initiated fetch developers.openai.com/api/docs/bots ClaudeBot Anthropic training support.claude.com/en/articles/8896518 Claude-SearchBot Anthropic search index (Claude search) support.claude.com/en/articles/8896518 Claude-User Claude user-initiated fetch support.claude.com/en/articles/8896518 PerplexityBot Perplexity search index docs.perplexity.ai/docs/resources/perplexity-crawlers Perplexity-User Perplexity user-initiated fetch docs.perplexity.ai/docs/resources/perplexity-crawlers Applebot-Extended Apple Intelligence training support.apple.com/en-us/119829 Meta-ExternalAgent Meta AI training developers.facebook.com/docs/sharing/webmasters/web-crawlers CCBot Common Crawl (feeds many models) commoncrawl.org/ccbot ``` A `robots.txt` you can copy that allows AI search visibility (the surfaces that cite your page) while opting out of training (the bots that scrape for model training data): ``` # Allow indexing for search and AI search surfaces User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / # Block AI training crawlers (does not affect Google Search inclusion) User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: CCBot Disallow: / # Fallback User-agent: * Allow: / Sitemap: https://0xinsider.com/sitemap.xml ``` The distinction matters. `GPTBot` and `ClaudeBot` are training crawlers, and blocking them does not affect search inclusion. `Google-Extended` is broader: it controls AI training **and** grounding inside Gemini Apps and Vertex AI Grounding, but does not affect Google Search ranking or AI Overview eligibility ([Google source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers)). The bots that determine whether your page can show up inside an AI answer are the search indexers: `Googlebot`, `Bingbot`, `OAI-SearchBot`, `Claude-SearchBot`, and `PerplexityBot` ([OpenAI source](https://developers.openai.com/api/docs/bots), [Anthropic source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), [Perplexity source](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)). Many sites accidentally block one of those and tank their visibility. Meta robots tags are the other lever, page-level rather than site-level: ```html ``` To opt out of `Google-Extended` (Gemini Apps and Vertex AI Grounding), use the `robots.txt` product token shown earlier. Google does not document `Google-Extended` as a robots meta tag, only as a `robots.txt` token ([source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers)). The snippet directives above are documented in Google's [meta tags reference](https://developers.google.com/search/docs/crawling-indexing/special-tags). ``` Page exists │ ▼ robots.txt allows crawl? ── no ──▶ invisible ▼ page renders without JS errors? ── no ──▶ invisible ▼ indexed in Search Console? ── no ──▶ invisible ▼ snippet allowed (no nosnippet)? ── no ──▶ regular search only ▼ quality + originality signals? ── weak ─▶ ranked, rarely cited │ ▼ Eligible to appear in AI Overviews and related surfaces ``` Every layer is a gate. The fancier optimization work only matters once all the gates are open. ### What gets cited is what a model can't write from training data alone Generative search rewards specificity. Models can summarize generic information without quoting anyone, so the pages that get cited are the ones that say something the model can't synthesize on its own. Google's guide tells creators to focus on "unique, valuable, people-first content" rather than commodity content that re-states what every other page on the topic already says ([source](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)). The deeper version of this advice lives in Google's [helpful content guidance](https://developers.google.com/search/docs/fundamentals/creating-helpful-content), which goes into how to demonstrate firsthand experience, real expertise, and original perspective. Here are two real versions of the same paragraph for an article about migrating to Next.js 16. Same topic, same word count, wildly different odds of being cited: ``` Commodity version: "Next.js 16 introduces async params, making route parameters asynchronous. This is a breaking change you should plan for when upgrading from Next.js 15. Make sure to await your params in dynamic routes." Distinctive version: "We migrated a 240-route Next.js 15 app to 16 last week. The async params change broke 47 pages in CI on the first run. The mechanical fix: wrap every `params.slug` access in `await params`. The catch we hit: dynamic API routes that destructure params in the function signature need the signature itself marked async, not just the body. Took 3 hours, almost all of it search-replace." ``` A model can produce the commodity version from training data alone, so it won't cite the source. There's nothing in there it couldn't write itself. The distinctive version has a number (47 broken pages), a specific catch (the function signature subtlety), and a time estimate (3 hours), none of which the model can generate without quoting the source. Even one of those details is often enough to flip a page from "training data summary" to "cited reference". ``` What the model sees about your topic │ ├──▶ Commodity content │ "Same overview 50 other pages have" │ │ │ ▼ │ Model synthesizes from training data │ │ │ ▼ │ Not cited │ └──▶ Distinctive content "Specific data, screenshot, opinion, result you tested in production" │ ▼ Model can't synthesize, must quote │ ▼ Cited in the answer ``` ### Clean technical structure helps the crawler and the model Semantic HTML matters. Use real heading levels in a sensible hierarchy, put the answer to the question the page is about near the top, and avoid burying content under preamble. A real before/after on the same blog post: ```html
How to migrate to Next.js 16
A practical guide
We migrated 240 routes last week...

How to migrate to Next.js 16

A practical guide to async params, Turbopack defaults, and the gotchas we hit.

Async params, in practice

We migrated 240 routes last week...

``` The second version gives the crawler clear structure (`article`, `h1`, `section`, `h2`) and the model clean boundaries for what's heading, lede, and body. Google's documentation on [page experience](https://developers.google.com/search/docs/appearance/page-experience) explains how Core Web Vitals feed into ranking, which feeds directly into AI feature eligibility. The thresholds Google publishes ([source](https://web.dev/articles/vitals)): ``` Metric Good Poor ───────────────────────────────────────────────────── LCP Largest Contentful Paint ≤ 2.5s > 4.0s INP Interaction to Next Paint ≤ 200ms > 500ms CLS Cumulative Layout Shift ≤ 0.1 > 0.25 ``` The numbers ranking algorithms look at are the 28-day field data from real Chrome users (CrUX), not a Lighthouse run on your laptop. Read them from `web-vitals` in JavaScript to align local testing with what Google's systems see: ```js onLCP(metric => console.log('LCP', metric.value, metric.rating)); onINP(metric => console.log('INP', metric.value, metric.rating)); onCLS(metric => console.log('CLS', metric.value, metric.rating)); ``` The AI optimization guide also pushes back on several "optimization hacks" circulating online. Adding an `llms.txt` file is not a ranking signal and isn't used by Google's AI features ([source](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)). Chunking content into tiny sections or rewriting every heading as a question is unnecessary, because models read context across the whole page. The guide also says structured data is useful where it powers a documented rich result, but it isn't required for AI feature visibility. Spend the time on real content quality and rendering instead. ``` What the crawler fetches ───────────────────────────────────────────────────────── Server-rendered HTML Client-only SPA shell ───────────────────── ─────────────────────

Title

Real content...

``` Test it inside Google's [Rich Results Test](https://search.google.com/test/rich-results) before deploying. The tool will tell you exactly which required fields are missing or malformed. If you run a local business or sell products, two unrelated surfaces matter more than schema. A verified [Google Business Profile](https://support.google.com/business/answer/3038063) feeds local AI answers with your hours, location, services, and reviews. A [Merchant Center](https://support.google.com/merchants/answer/188494) feed is what AI Overviews pull product information from. The AI optimization guide names both explicitly as the primary input for business and commerce results ([source](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)). ``` Type of result Source feed Where it shows ──────────────────────────────────────────────────────────────────── Local business ◀── Google Business Profile ──▶ Maps, local panel, (hours, location, reviews) local AI answers Products ◀── Merchant Center feed ────▶ Shopping cards, (price, stock, variants) product AI answers Recipes, FAQs, ◀── Schema.org JSON-LD ──────▶ Rich results, events, articles (on-page structured data) AI understanding ``` ### Agentic experiences are the next surface The newer wrinkle is autonomous agents browsing on the user's behalf (Claude with computer use, ChatGPT Operator, Perplexity's assistant). Google's AI optimization guide recommends sites consider how agents interpret their DOM, controls, and content ([source](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)). Sites with confusing markup, hidden controls, or essential information rendered only as images are hard for agents to operate. The accessibility work you'd already do for screen readers covers most of the same ground. A real before/after of an interactive control on a booking page: ```html
``` The second version tells an agent three things: it's a submit button, the action is "Confirm booking", and the icon is decorative. The first version tells it nothing. An agent that can't identify the booking confirmation gives up and picks a site it can operate. Form fields work the same way. An agent reads `name`, `id`, `aria-label`, and the surrounding `