Michał Suski

Chiang Mai SEO Conference · 13 November 2026

The story of 50M scraped LLM answers

For a year and a half we recorded what five AI engines actually answer, every day, from the real interface. Here is how to measure AI visibility without fooling yourself, which popular proxies don’t hold up, and what actually gets you cited.

Michał Suski on stage at Chiang Mai SEO Conference 2026, presenting the agenda of his talk on 50 million AI answers

Slides

The full talk, 56 slides

Use the arrow keys or swipe. Each slide comes with the line I said on stage.

Slide 1: The story: scraped 50M LLM answers. Story by Michał Suski
1 / 56We spent a year and a half recording what AI actually answers — every day, from the real interface. 1
Slide 2: 50M scraped AI answers, 5 engines, every day
2 / 56Fifty million answers. That's what we're going to argue with today. 2
Slide 3: 100k prompts a day, every day since March 2025
3 / 56Five engines, a hundred thousand prompts a day, scraped from what users really see — not from an API. 3
Slide 4: How much of what you hear about AI SEO survives contact with data?
4 / 56So here's the only question that matters: how much of what you hear about AI SEO survives contact with data? 4
Slide 5: Three acts: what moves an AI answer, false signposts, what gets you cited
5 / 56Three acts: first we fix how you measure, then we kill the fake signposts, then we show what actually gets you cited. 5
Slide 6: Act I: What moves an AI answer?
6 / 56Before asking what works, ask a simpler question: what moves an AI answer at all? 6
Slide 7: Change one thing, watch the answer change
7 / 56Run the same prompt twice and you share only 40% of brands — and every other change pulls you further down. 7
Slide 8: What if you just ask again? Brands present in every daily run
8 / 56Ask again and the answer changes; after twenty runs only 4% of brands show up every single time. 8
Slide 9: What if you wait a week? Overlap after 1, 7, 30 and 90 days
9 / 56Day-to-day is mostly noise, but over weeks it really drifts — overlap falls from 40% the next day to 28% after 90 days. 9
Slide 10: Reword it all you want: brand overlap by type of change
10 / 56Reword the question all you want — the answer keeps three quarters of its brands. Ask a different question and it keeps barely a third. 10
Slide 11: Track intents, not rewordings
11 / 56So stop paying to track a hundred variants of one question — track different intents instead. 11
Slide 12: What if you ask the API? Overlap with the chat interface
12 / 56If you track AI through the API, you're measuring a different product: only 29–42% of its top-3 brands make the chat's top 3. 12
Slide 13: What if you ask another engine? Overlap between engines
13 / 56Google's three surfaces move together; ChatGPT and Perplexity follow nobody — so one 'AI visibility' number is a fiction. 13
Slide 14: Noise you can’t remove, error you can stop adding
14 / 56Some of this noise you can't remove — but the measurement error you can stop adding yourself. 14
Slide 15: How to measure honestly: five rules
15 / 56So here are five rules for measuring without fooling yourself. 15
Slide 16: Act II: False signposts
16 / 56Because measuring is hard, everyone grabs a proxy — let's test the most popular ones. 16
Slide 17: Global top domains aren’t your top domains
17 / 56Reddit tops every global ranking, but only 9% of your niche's core sources come from that global top 20. 17
Slide 18: Find your niche’s core sources before you chase anyone else’s
18 / 56So before you chase Reddit or Wikipedia, find out which sources actually win in your niche. 18
Slide 19: Authority is a proxy, not a factor
19 / 56Two proxies everyone trusts: domain authority and domain age. 19
Slide 20: PageRank doesn’t get you cited
20 / 56Start with PageRank, the classic authority metric: against citations it's flat zero on every engine. 20
Slide 21: Centrality helps, barely
21 / 56Harmonic centrality is the one authority metric that always points up — but it's so small that covering the facts matters about eight times more. 21
Slide 22: Traffic gets you named, not cited
22 / 56And traffic: big sites get their brand named more often — but their pages don't get cited more. 22
Slide 23: Domain age: old sites win on volume, not on merit
23 / 56Old domains win on volume because they have more pages; inside one prompt, age barely matters and even flips sign. 23
Slide 24: The GEO checklist is decoration
24 / 56Stats, quotes, FAQ, table of contents — the GEO checklist everyone sells. 24
Slide 25: The checklist sits at the bottom of 20 features
25 / 56Every checklist item sits at the very bottom of 20 features; answering the query early is worth four to five times more. 25
Slide 26: Linking to Wikipedia won’t save you
26 / 56'Link to Wikipedia and other trusted sources so AI trusts you' — let's check. 26
Slide 27: Linking to Wikipedia does nothing: 0.031 across all pages
27 / 56Across all pages, external links correlate with being cited at 0.03 — practically nothing. 27
Slide 28: Linking to Wikipedia does nothing: 0.007 in blog posts
28 / 56And in blog posts it drops to 0.007 — that's noise, not a signal. 28
Slide 29: llms.txt is the new meta keywords. Schema isn’t.
29 / 56And the newest hype: llms.txt and schema — one of them is the new meta keywords. 29
Slide 30: llms.txt flips by engine
30 / 56llms.txt flips sign from engine to engine — plus 4 on Gemini, minus 2 on AI Mode. That's not a signal. 30
Slide 31: Schema is up on every engine
31 / 56Schema is up on every engine — but mostly because it marks sites someone actually maintains. 31
Slide 32: Big isn’t the same as cited
32 / 56Big isn't the same as cited. 32
Slide 33: Act III: What gets you cited
33 / 56Now the good news: the levers that actually hold up in the data. 33
Slide 34: Google AI is still Google: 95% of cited domains rank in Google’s top 10
34 / 5695% of cited domains already rank in Google's top 10 — SEO isn't dead, it's your entry ticket. 34
Slide 35: The first citation is the big one
35 / 56The first time your own site gets cited is the big one: plus 25% visibility — later citations add little on top. 35
Slide 36: How to write for AI: five moves
36 / 56Here's how to write for AI, in five moves — and the strongest single signal is matching the intent. 36
Slide 37: Answer in the first 50 words
37 / 56Almost a third of all citations come from the first 50 words — answer immediately, not after the intro. 37
Slide 38: Match intent or don’t bother
38 / 56A page that answers exactly what was asked is a source 42% of the time; an off-intent blog post, 13%. 38
Slide 39: Write in chunks AI can lift
39 / 56AI quotes passages, not pages — so every paragraph has to make sense on its own. 39
Slide 40: Facts win the source game
40 / 56The pages AI keeps citing cover a lot more of the facts than the ones it barely touches. 40
Slide 41: Every rank down, fewer facts
41 / 56And it's not just top versus bottom — coverage drops with every step down the ranking, from 39% in the top 3 to 26% by rank 50. 41
Slide 42: Unique angle: concrete detail wins
42 / 56Unique pages win, and what makes them unique isn't original research — it's concrete detail. 42
Slide 43: Two kinds of lists: theirs and your own
43 / 56Now the most controversial format in AI search: listicles — yours, and theirs. 43
Slide 44: Lists aren’t losing ground
44 / 56Everyone says AI stopped citing lists — but on the same reports, the share of both kinds barely moved all year. 44
Slide 45: Your list gets you named
45 / 56When your own list gets cited, you get named 16 percentage points more often — and higher up in the answer. 45
Slide 46: Put yourself first
46 / 56On your own list, put yourself first — you get named 38% of the time instead of 26%. 46
Slide 47: Cited vs named: two games
47 / 56Getting the page cited and getting your brand named are two different games — sometimes with opposite rules. 47
Slide 48: More lists, more presence
48 / 56Brands with four or more cited lists get mentioned in 38% of answers — build a set, not one page. 48
Slide 49: The slot is everything
49 / 56On someone else's list, the slot is everything: #1 gets named half the time, #10 barely at all. 49
Slide 50: Your list makes you in charge of the message
50 / 56Your list puts you in charge of the message. Their list makes the category visible. 50
Slide 51: Older content still gets cited
51 / 56And once a page stays cited, it keeps working for years — most citations go to pages older than six months. 51
Slide 52: The page fades, the name stays
52 / 56The page fades fast, but the name stays — 83% of the mention survives eleven weeks. 52
Slide 53: What 50M answers told us: measure honestly, ignore the proxies, pull the levers
53 / 56Measure honestly, ignore the proxies, pull the levers — that's the whole story of 50 million answers. 53
Slide 54: Meme: the industry looks at the shiny object while data stands right there
54 / 56Because the industry will always chase the next shiny object — while the data is standing right there. 54
Slide 55: Don’t trust me, check your brand
55 / 56Don't trust me — check your own brand; scan the code for a free AI visibility report. 55
Slide 56: Thank you
56 / 56Thank you — come find me if you want to argue with the data. 56

What the data says

Six numbers from the talk

From Surfer AI Tracker: about 50 million answers from ChatGPT, Perplexity, Gemini, Google AI Overviews and AI Mode.

40%
of brands are shared by two runs of the exact same prompt. After 20 runs, only 4% appear every time.
29–42%
of an API answer’s top-3 brands make the top 3 in the chat interface people actually use.
8.9%
of a niche’s core sources come from the global top 20 domains everyone tells you to chase.
95%
of cited domains already rank in Google’s top 10 for at least one keyword.
⅓
of all citations come from the first 50 words of a page.
42% vs 13%
how often a page that matches the intent is cited as a source, compared with an off-intent blog post.

The playbook

Measure honestly, ignore the proxies, pull the levers

The three acts of the talk as a checklist. Start with measurement: without it you can’t tell a lever from noise.

AI answers are noisy. Some of it you can’t remove; the measurement error you can stop adding yourself.

  • Rule

    Run every prompt many times.

    Two runs of the same prompt share only 40% of brands. After 20 runs, just 4% of brands appear every single time. One run is an anecdote.

  • Rule

    Track every day, judge over weeks.

    Day to day is mostly noise, but answers drift: brand overlap falls from 40% the next day to 28% after 90 days.

  • Rule

    Track intents, not rewordings.

    Rewording a question keeps about three quarters of its brands. A different intent keeps barely a third. A hundred variants of one question is money spent on the same answer.

  • Rule

    Scrape the real interface, not the API.

    Only 29–42% of an API answer’s top-3 brands make the top 3 in the chat people actually use. The API is a different product.

  • Rule

    Report each engine separately.

    Google’s three surfaces move together; ChatGPT and Perplexity follow nobody. A single “AI visibility” number averages things that don’t belong together.

Because measuring is hard, everyone grabs a proxy. These are the popular ones, checked against citations.

  • Myth

    “Get on Reddit and Wikipedia, they win everything.”

    They top global rankings, but only 8.9% of a niche’s core sources come from the global top 20. Find the sources that win in your niche first.

  • Myth

    “Domain authority gets you cited.”

    PageRank against citations is flat zero on every engine. Harmonic centrality is the one authority metric that points up, and covering the facts matters about eight times more.

  • Myth

    “Big sites get cited.”

    Big sites get their brand named more often, but their pages aren’t cited more. Big isn’t the same as cited.

  • Myth

    “Old domains are trusted more.”

    Old domains win on volume because they have more pages. Inside one prompt, domain age barely matters and even flips sign on ChatGPT and Perplexity.

  • Myth

    “Add stats, quotes, an FAQ and a table of contents.”

    Every checklist item sits at the bottom of 20 page features. Answering the query early is worth four to five times more.

  • Myth

    “Link to Wikipedia so AI trusts you.”

    External links correlate with being cited at 0.03 across all pages and 0.007 in blog posts. That’s noise.

  • Myth

    “Add llms.txt.”

    llms.txt flips sign from engine to engine, up on Gemini and down on AI Mode. That’s not a signal.

  • Partly

    “Add schema.”

    Schema is up on every engine, mostly because it marks sites someone actually maintains. Keep structured data on pages you look after; skip llms.txt.

What holds up in the data. For the full listicle playbook, see the follow-up talk Who writes the list?

  • Lever

    Rank in Google first.

    95% of cited domains rank in Google’s top 10 for at least one keyword. SEO isn’t dead; it’s the entry ticket.

  • Lever

    Go after the first citation of your own site.

    The first time your own site gets cited is worth about +25% visibility. Later citations add little on top.

  • Lever

    Answer in the first 50 words.

    Almost a third of all citations come from the first 50 words of a page. Answer immediately, not after the intro.

  • Lever

    Match the intent exactly.

    A page that answers exactly what was asked is a source 42% of the time; an off-intent blog post, 13%. The strongest single writing signal.

  • Lever

    Write in chunks AI can lift.

    AI quotes passages, not pages. Every paragraph has to make sense on its own.

  • Lever

    Cover the facts.

    Coverage drops with every step down the ranking: 39% of the facts in the top 3, 26% by rank 50.

  • Lever

    Be unique through concrete detail.

    Unique pages win, and what makes them unique isn’t original research. It’s specific, concrete detail.

  • Lever

    Get on lists, at the top.

    On someone else’s list your brand is named about half the time at #1 and barely at all at #10. On your own list, put yourself first: 38% instead of 26%.

  • Lever

    Build a set of lists, not one.

    Brands with four or more cited lists of their own are mentioned in 38% of answers, against 26% with none.

  • Lever

    Keep pages alive.

    57–74% of citations in AI Mode, AI Overviews, ChatGPT and Perplexity go to pages older than six months. The page fades fast, but the name stays: 83% of the mention survives eleven weeks.

AI instructions

Two audit prompts, backed by data

Two audit prompts: one for a page, one for how you measure. Every check comes with the number behind it, and the numbers travel with the prompt, so the model knows what matters most.

Audit a page

Paste a page and the query you want it cited for. The model checks it against eight data-backed checks and tells you what to fix first, and what to stop doing.

  1. Intent: does the page type match what the query asks? A “which” query needs a comparison, a “how” query needs steps.

    42% vs 13%how often an on-intent page vs an off-intent blog post is cited

  2. First 50 words: is there a direct, complete answer before any intro?

    ≈⅓of all citations come from a page’s first 50 words

  3. Passages: does each paragraph name its subject and make sense if quoted alone?

    PassagesAI quotes passages, not whole pages

  4. Fact coverage: list what a complete answer needs and mark which facts are missing.

    39% → 26%facts covered by top-3 vs rank-50 sources

  5. Concrete detail: are there specific numbers, settings, examples or first-hand results, or only generic claims?

    0.12correlation of a unique, concrete angle with citations

  6. Google: does the page rank in the top 10 for the query or close variants? If not, flag it as the first fix.

    95%of cited domains rank in Google’s top 10

  7. Structured data: is JSON-LD present and current (Article, Person as author, dateModified)?

    +3.7 ppcitation rate with vs without JSON-LD; Product markup +0.1 pp

  8. Wasted effort: flag llms.txt, “trust” links to Wikipedia, and stats, quotes, FAQs or a TOC added only for a checklist. Mark them low priority, not as fixes.

    ≈ 0llms.txt −0.04 pp without big hosts; external links 0.03; checklist items 4–5× weaker than an early answer

See the full prompt text
Audit the page below for how likely AI answer engines (ChatGPT, Perplexity, Google AI Overviews and AI Mode, Gemini) are to cite it as a source.

The checks come from Surfer AI Tracker data: about 50 million AI answers scraped daily from the real interfaces of five engines. They describe what cited pages have in common; treat them as strong signals, not guarantees.

INPUTS
- Target query: [the exact question people ask]
- Page: [paste the text or the URL content]
- Our brand: [Brand]
- Google position for the query, if known: [e.g. 7, or "not ranking"]

CHECKS (each with the data behind it)
1. Intent: does the page type match what the query asks? A “which” query needs a comparison, a “how” query needs steps. [data: 42% vs 13% how often an on-intent page vs an off-intent blog post is cited]
2. First 50 words: is there a direct, complete answer before any intro? [data: ≈⅓ of all citations come from a page’s first 50 words]
3. Passages: does each paragraph name its subject and make sense if quoted alone? [data: Passages AI quotes passages, not whole pages]
4. Fact coverage: list what a complete answer needs and mark which facts are missing. [data: 39% → 26% facts covered by top-3 vs rank-50 sources]
5. Concrete detail: are there specific numbers, settings, examples or first-hand results, or only generic claims? [data: 0.12 correlation of a unique, concrete angle with citations]
6. Google: does the page rank in the top 10 for the query or close variants? If not, flag it as the first fix. [data: 95% of cited domains rank in Google’s top 10]
7. Structured data: is JSON-LD present and current (Article, Person as author, dateModified)? [data: +3.7 pp citation rate with vs without JSON-LD; Product markup +0.1 pp]
8. Wasted effort: flag llms.txt, “trust” links to Wikipedia, and stats, quotes, FAQs or a TOC added only for a checklist. Mark them low priority, not as fixes. [data: ≈ 0 llms.txt −0.04 pp without big hosts; external links 0.03; checklist items 4–5× weaker than an early answer]

OUTPUT
1. A table: check | what you found (quote the page where useful) | pass / partly / fail | the fix.
2. The missing facts from check 4 as a list.
3. A rewritten first 50 words that answer the query directly.
4. The 5 fixes that matter most, in order of impact.
5. What to stop spending time on (from check 8), if anything.

Audit my tracking setup

Before you trust any AI visibility number. Describe how you collect answers and the model grades it against eight rules.

How much can you trust one run?

Of the brands you see across several daily runs of the same prompt, this share shows up in every one of them.

16%of brands in every run
16%of cited domains
  1. Run every prompt many times; never judge a brand from a single answer.

    40%of brands are shared by two runs of the same prompt

  2. Track every day, but read trends over weeks.

    40% → 28%brand overlap one day vs 90 days later

  3. Expect sources to churn faster than brands.

    37% → 13%cited-domain overlap one day vs 90 days later

  4. Track different intents, not rewordings of one question.

    −8 vs −19 ppbrand overlap lost by rewording vs by asking another engine

  5. Collect answers from the real chat interface, not the API.

    29–42%of API top-3 brands make the chat’s top 3 (chat vs itself: 49–54%)

  6. Report each engine separately, never one blended score.

    0.32–0.40how closely ChatGPT and Perplexity track other engines (Google surfaces: 0.58–0.70)

  7. Don’t use domain authority as a stand-in for AI visibility.

    ≈ 0correlation of PageRank and Domain Score with citations, on every engine

  8. Benchmark against your niche’s sources, not global top domains.

    8.9%of a niche’s core sources come from the global top 20

See the full prompt text
Review my AI visibility tracking setup against measurement rules from Surfer AI Tracker data (about 50 million AI answers from five engines, scraped from the real interfaces).

INPUTS
- Tool or method: [how answers are collected]
- Engines: [list]
- Prompts: [how many, how they were chosen]
- Runs per prompt and frequency: [e.g. once a week]
- Reporting: [one score, per engine, per prompt?]

RULES (each with the data behind it)
1. Run every prompt many times; never judge a brand from a single answer. [data: 40% of brands are shared by two runs of the same prompt]
2. Track every day, but read trends over weeks. [data: 40% → 28% brand overlap one day vs 90 days later]
3. Expect sources to churn faster than brands. [data: 37% → 13% cited-domain overlap one day vs 90 days later]
4. Track different intents, not rewordings of one question. [data: −8 vs −19 pp brand overlap lost by rewording vs by asking another engine]
5. Collect answers from the real chat interface, not the API. [data: 29–42% of API top-3 brands make the chat’s top 3 (chat vs itself: 49–54%)]
6. Report each engine separately, never one blended score. [data: 0.32–0.40 how closely ChatGPT and Perplexity track other engines (Google surfaces: 0.58–0.70)]
7. Don’t use domain authority as a stand-in for AI visibility. [data: ≈ 0 correlation of PageRank and Domain Score with citations, on every engine]
8. Benchmark against your niche’s sources, not global top domains. [data: 8.9% of a niche’s core sources come from the global top 20]

For each rule: say pass / partly / fail, why, and the cheapest fix. Then list the conclusions I’ve drawn so far that I shouldn’t trust yet, and what to change first.

Method

Where the numbers come from

All data comes from Surfer AI Tracker: about 100,000 prompts a day, scraped since March 2025 from the real interfaces of ChatGPT, Perplexity, Gemini, Google AI Overviews and AI Mode, not from APIs. Each slide lists its own sample and period in the footer.

Answers
About 50 million AI answers across five engines.
Noise tests
Repeated runs, daily re-runs, rewordings, API vs chat and engine-to-engine comparisons of the same prompts. Each slide lists its sample and period.
Proxies
Domain-level metrics (PageRank, harmonic centrality, traffic, age) and page features compared with how often pages are cited within the same prompt.
Listicles
238.6M citations in 8,677 reports, January to September 2026. Full detail on the Who writes the list? page.

Correlations and comparisons within prompts and niches, not proven causes. Test on your own pages and prompts before you roll anything out.

Want this talk at your event?

2027 is open, with the first events confirmed in March.