Methodology updates
What changed, and why
The methodology follows the research. When new studies change what we know about how AI engines find, select and cite sources, the methodology is revised and every page and tool on this site is updated to match. This log records each update by month.
All pages and tools are current as of November 2026.
Finding, reading, scoring and measuring
October moved the evidence to 2026 research. November checked how the engines themselves work against their own 2026 documentation, and turned that into new rules. What is new: finding and reading are separate jobs, with indexing as a precondition; how many searches an engine runs depends on the mode; which matching methods are actually documented; how fast a page can be indexed; two new opt-out controls; a “Found and Readable” check and Topic Match in the score; fan-out queries as context for the writer; and a fan-out log in measurement. The terms also changed: topic match, fan-out queries, prompt. Later in the month what counts as visibility changed. There are now six primary KPIs instead of five: Mention Rate (the brand is named) and ACF (one of your own pages is cited) lead visibility with equal standing and are never blended. Gate 1 became TSM’s own page quality standard of eight checks.
7
steps from indexing to citation, mapped to the four stages of an AI answer.
8
page-quality checks in Gate 1, each with a job in retrieval and its evidence stated.
23/30
to publish on the writing-techniques score (Gate 2), which now includes Topic Match. A TSM default; calibrate to your data.
What the new research showed
-
Finding and reading are separate jobs. Search crawlers build the index in advance, and the engine searches it when a prompt arrives. User-triggered fetchers then read the live page. A page that is not indexed is never found. A page whose facts need JavaScript contributes nothing when it is opened, and it can also be missing from the index of an engine whose search crawler does not run JavaScript.
🏷 OpenAI, Anthropic, Perplexity, Google crawler documentation, checked 2 Nov 2026
-
How many searches an engine runs depends on the engine and the mode. ChatGPT’s instant modes ran 1.06–2.36 fan-out queries per prompt, mostly in one round; GPT-5.4 Thinking ran 11.82 over 3.20 rounds. In real use, ChatGPT averaged 3.07 and Claude 1.80. None of these is a target.
📊 Jiang et al., arXiv 2610.15382, Oct 2026 📊 Amani et al., arXiv 2609.19244, Sep 2026
-
Each fan-out query is searched on its own. No engine documents combining them or publishes how it merges the results. Keyword matching is documented for Google and Perplexity, and Perplexity documents embeddings as the first stage of its retrieval. BM25 is a published formula that no engine says it uses.
🏷 Google, How Search works, checked 2 Nov 2026 🏷 Perplexity, Photon, 24 Sep 2026; pplx-embed, 26 Feb 2026
-
No engine publishes how often it crawls. Google says recrawling can take “a few days to a few weeks”. IndexNow notifies Bing, Naver, Seznam.cz, Yandex and Yep, does not guarantee crawling, and does not reach OpenAI, Anthropic or Perplexity. Your server logs are the only reliable record of crawler visits.
🏷 Google Search Central, checked 2 Nov 2026 🏷 IndexNow and Bing, checked 2 Nov 2026
-
Two new opt-out controls. Google Search Console’s generative AI control (worldwide from 31 Aug 2026) removes a site from AI Overviews, AI Mode and Discover AI features. Bing’s page-level
ai-grounding:disallowcovers Bing, Copilot and Microsoft 365; Bing honors it from 1 February 2027, and its older NoArchive and NoCache tags stop being read as a training or grounding preference after 1 March 2027. Google-Extended also governs grounding in the Gemini app and Vertex AI, and Google plans to support a shared robots.txt control for AI training in 2027.🏷 Search Console Help, checked 2 Nov 2026 🏷 Bing Search Blog, 30 Oct 2026 🏷 Google common crawlers, 30 Oct 2026 🏷 Google Search Central Blog, 30 Oct 2026
What changed in the methodology
How AI answers work
- BeforeEngines were described as splitting every prompt into several sub-queries, with keyword matching (BM25) as a step every engine runs.NowWhen an engine searches, it writes one or more fan-out queries; how many depends on the engine and the mode. Keyword matching is stated only where an engine documents it.
Content
- Before“Question match”: titles and paragraphs matched to the question.NowTopic match: the title, headings and opening or best paragraph carry the topic of the searches people and engines run, in meaning and in exact terms, whether the need is phrased as a question, a problem, a product or a solution. A question heading is one optional form.
- BeforeBefore publishing, a page was checked in two gates. Gate 1: a quality standard, pass or fail. Gate 2: five writing techniques, 5 points each, 25 in total; publish at 19 or more.NowGate 2 has a sixth technique, Topic Match: the title, headings and best paragraph carry the topic of the searches people and engines run. Six techniques, 5 points each, 30 in total. A page is ready to publish when it is found and readable, passes Gate 1, and scores 23 or more out of 30 in Gate 2. The 23 is a TSM default: a starting point to adjust once your own results show where it should sit. How the two gates work · Score a page with the Content Scorecard
- BeforeGate 1 scored the 14 named pillars of an outside page-quality framework, GEO-16: pass at 70% of the points and 12 of 14 hits.NowGate 1 is TSM’s own page quality standard: eight checks, each with its job (helps the page be found, be selected or shape the answer) and its evidence. They are dates and freshness, heading structure, sourced claims, accurate and specific figures, self-contained sections, content depth, comparison tables and internal links. A page passes at 17 of 24 points with a band of 2 or more on at least 6 of the 8 checks (a TSM default; calibrate to your data). The checks originate in GEO-16*; six of its pillars had no engine evidence, or evidence against an effect, and are no longer scored. Structured data is handled by the Technical stream, for entity clarity; the author byline in the E-E-A-T audit; disclosures in compliance review. The eight checks and their evidence
- BeforeBoth gates used fixed totals for every page.NowBoth gates count only the items that apply to the content type. On operational pages (returns, shipping, warranty), sourced claims and three techniques do not apply: Gate 1 is 15 of 21 points with 6 of 7 hits, and Gate 2 is 12 of 15.
- BeforeOne quality check asked for question-and-answer format where the prompt was a question.NowQuestion-and-answer format is not checked anywhere. A section is checked for standing on its own: a heading that names its topic, with the answer first.
- Before“Use the exact terms of the fan-out queries: product names, model numbers, standards.”NowUse the exact terms of the recurring unbranded fan-out queries: your own product names, model numbers, standards, ingredient and technique names. Competitor names in fan-out queries are read, never copied into a passage to match them; a comparison page is a separate decision.
- BeforeThe About page (the brand or entity anchor) was ranked with the other quick wins.NowThe brand or entity anchor is built first whatever its rank, because every piece links back to it.
- BeforeThe fan-out queries were grouped to plan pages.NowGrouped into context themes, the fan-out queries are context for the writer: they show what the engines look up for a prompt. A piece’s structure follows its content type, and a brief does not wait for fan-out data. How fan-out context works
Technical
- BeforePages were scored for quality whether or not an engine could find them.NowA “Found and Readable” check comes first: the page is indexed by the search crawlers, and every citable fact is in the HTML the server sends. A page that fails goes back to the Technical Stream. The Content Scorecard asks this first.
- BeforeGetting pages indexed was not a named duty.NowFast indexing is a Technical Stream duty: sitemaps with accurate dates, IndexNow for the engines that take it, recrawl requests in Search Console, and crawler visits read from server logs.
- BeforeAI crawlers were treated as one group.NowSearch crawlers, user-triggered fetchers and training crawlers are kept apart; the 2026 opt-out controls are checked before anything is blocked.
- BeforeLive AI fetchers were described as not reading JSON-LD, and schema as having no significant effect within a site.NowWhether a fetcher reads JSON-LD differs by engine (Grok and Kimi did in a 2026 test; ChatGPT’s and Claude’s fetchers did not), so every citable fact still goes in visible text. Schema evidence now carries its figures: +7.5% more citations across websites, −5.3% within one site (not significant), which the authors read as Simpson’s paradox. Markup favors depth over breadth: connected, accurate markup on the core types, linked to Wikidata. It serves two routes, entity clarity in the indexes and brand knowledge, and neither effect on citation has been measured.
- BeforeGlossary terms could be marked up with their own type, DefinedTerm.NowMarkup stays on the core types (Organization, Brand, Product, Person, Article, BreadcrumbList) and on types Google or Bing document. A definition goes in visible text, in the sentence that opens the section.
- Before“Expert quotes without Person schema” and “GTM-injected schema” were listed as common mistakes.NowThey are “expert content without verifiable credentials” (credentials are handed over, shown on the visible page and mirrored in Person markup) and “facts or schema injected client-side” (every citable fact in the served HTML).
Measurement
- BeforeSentinel prompts were the only log.NowSentinel prompts stay the outcome measure, beside a fan-out log of the searches the engine ran, taken from the APIs and Bing AI Performance.
- BeforeEach run (one sentinel prompt sent to one engine, with the answer saved) recorded the engine, the date and the location.NowEach run also records the model and the mode (for example instant or thinking), because modes run different numbers of searches. Pages not yet indexed are left out of citation judgments.
- BeforeACF counted a run when the brand appeared at all, named or cited.NowACF counts only the runs in which a page on your own domains is cited. Mention Rate, the share of runs in which the brand is named, cited or not, is measured over the same runs and never blended with it. “Citation Rate” is ACF itself. SOV-AI counts mentions, so it is read beside Mention Rate. How ACF is defined
- BeforeFive primary KPIs. ACF was the only lead visibility KPI, and Mention Rate was a supporting metric.NowSix primary KPIs. Mention Rate and ACF lead visibility with equal standing. Most AI answers are read without a click, and being named and being cited are mostly separate events: in one study 61.7% of appearances were a citation with no brand name in the answer, 25.1% a brand name with no citation and 13.2% both. 🏷 Semrush with Kevin Indig, 9 Jun 2026 🏷 BuzzStream, 29 Jun 2026 📊 Pew Research Center, 22 Jul 2025 The two lead KPIs
- BeforeNo-Search Brand Presence was described as leading ACF.NowIt leads Mention Rate: brands the model rarely names without search are also rarely named with search.
- BeforeA change was called “reliable” above about 1.4 times the margin.NowThat figure is the smallest change that can be told apart from noise at 95% confidence. To detect a real change four times in five, the change must be about 1.4 times larger still: about 5.3 points for a 50-query set.
- BeforeBrand-name queries for training contamination ran inside brand-direct monitoring.NowThree query sets are kept apart: performance sentinel queries (brand-absent, the only ones that feed ACF and SOV-AI), brand-direct queries (read quarterly) and risk detection queries (run weekly). The training-contamination queries belong to the risk set.
- BeforeAssisted-conversion deltas compared pages that engines cite with pages they do not.NowThey compare customer journeys that include an AI touchpoint with comparable journeys that include none. The page comparison stays as an optional second proxy. CTR-AI is defined as AI-referred sessions divided by citations of owned pages, computable only where an engine reports citation counts.
- BeforeStatcounter’s chatbot figures were described as reach or market share.NowThey are referral share: the share of referral clicks each chatbot sends to websites, not its share of users or of citations.
- BeforeThe last phase ended at “leading ACF within the competitive set”.NowIt ends when ACF is at the position the organization set as its goal within the competitive set.
Diagnosis and handoffs
- BeforeFailure categories A and B described streams as absent.NowA stream can be missing or ineffective: present on paper, but not doing its job.
- BeforeThe diagnostic tree checked entity markup as its own step, and authority by credentials and reference entries.NowStep 4 asks whether the page is indexed and retrieved for the target prompts, with entity markup as a sub-check. The authority test asks whether the model names the brand without search, whether the independent sources engines cite mention the brand, and whether a Wikidata item exists.
- BeforeThe flow model listed what moves between streams.NowIt also says how one planned piece moves: two companion briefs (one for the writer, one for Technical), one owner and one stage for every open item, and technical work that starts in parallel where it does not depend on final copy. Technical passes the fan-out log to Content.
- BeforeA claim that failed review was rejected, or reworded on the spot.NowA failed claim is sorted by cause. Where the kind of claim is not allowed, more evidence does not unlock it: a different wording is proposed, marked as a proposal and sent to the group that owns the wording, before compliance review. Where only the proof is missing, the claim waits for matching evidence and is not reworded. A reworded claim is a new claim, and a brief or a review never changes published or approved wording.
Risk and crisis
- BeforeThe five crisis levels were described by example.NowEach level has a definition that covers false and real events alike, from a minor inaccuracy (Level 1) to a crisis spreading across several engines and platforms at once (Level 5).
- BeforeThe response named a team, not an owner for each step.NowSeven steps each have one owner group: detect, classify, base response, choose the strategy, correct the sources, re-measure, close and review. Every incident at Level 3 or above closes with a written review that answers five questions.
- BeforeAuthority erosion was detected by a falling ACF.NowIt is detected by a falling self-citation rate: the share of brand-name queries in which your own domain is cited.
Removed
- Engine-behavior sources dated 2025 or earlier, such as Google’s “hundreds of searches” for Deep Search and late-2025 fan-out counts.
- BM25 as a step every engine runs.
- “Question match” and “sub-queries” as terms.
- GEO-16 as a scored framework, and its six pillars without engine evidence.
- DefinedTerm markup, and “intent-conditional citation” as a glossary term.
What changed on this site
- Platform & Retrieval Mechanics walks through every step of an AI answer, with an interactive animation.
- Every page and tool now uses “topic match”, “fan-out queries” and “prompt”.
- Every page and tool uses the new definitions of ACF and Mention Rate, shows both as lead visibility KPIs, and uses the eight quality checks.
* GEO-16 is the page-quality framework of Kumar & Palkhouski (arXiv 2509.10762, Sep 2025; preprint). Its correlations compare pages across websites: associations, not effects.
Evidence labels: 🔬 peer-reviewed · 📊 preprint or disclosed method · 🏷 vendor, agency or platform · 💡 TSM judgment.
The evidence moved to 2026 research
The methodology was rebuilt on research published in 2026, with older work kept only where it is foundational. The core idea holds: three streams that depend on each other. What changed is the evidence underneath it, how progress is measured, and which efforts are known to pay at each stage of an AI answer.
97%
of the pages ChatGPT retrieved were absent from Google’s top 50 for the same queries.
4
pipeline stages, with every element of the methodology mapped to the stage it acts on.
6
primary KPIs, read per engine, with no targets and no position weights.
What the new research showed
-
AI engines run their own searches, and retrieval works on meaning. Every page ChatGPT cited came from its own search. Content works mainly by matching, in meaning, the queries the engine runs, including the fan-out queries it writes itself. The pages it retrieved were largely not the pages that rank in Google.
📊 Chen & Koudas, arXiv 2610.15033, 10 Oct 2026; preprint; tested on API models, not the consumer apps
-
Engines run their own searches (fan-out queries). ChatGPT ran two or more fan-out queries on 89.6% of searches across 15,000 prompts, and 32.9% of the pages it cited surfaced only through a fan-out query.
🏷 AirOps, 12 Mar 2026
-
Engines remember brands. A brand the model already knew was written into ChatGPT’s own searches in 67% of cases, against 11% otherwise, and 46% of its mentions of brands the prompt did not name were supported by no cited page.
📊 Chen & Koudas, Oct 2026; preprint
-
Selection depends on topic match and recency. Among sources already retrieved, topic match, list position, an explicit price and a recent date decided which were cited, in all six models tested.
🔬 Vishwakarma, Kumar & Jamidar, SIGIR 2026
-
No single kind of optimization carries a page through. In a simulated search engine, rewriting body text alone lowered retrieval, reranking and citation; optimizing titles alone helped retrieval but lost ground at reranking.
🔬 Kim et al., SAGEO Arena, KDD 2026
-
Old lab percentages are not forecasts; the techniques still count as substance. The Princeton GEO study (2024) measured its well-known lifts, such as those from adding quotations or statistics, with the sources already placed in the model’s context, so retrieval was never tested. When later studies added a retrieval step, most rewrite tactics applied to existing pages (inserting statistics or quotations to influence an engine) did not help or did harm, and the gains shrank as more sites copied the same tactic.
This does not mean content has no effect. Across 45 studies, topic match and position in the model’s context were the most reproducible levers. Once a page is cited, pages with real statistics, definitions and comparisons shaped more of the answer. TSM keeps the techniques as content principles where they carry real information, and attaches no expected percentage to any of them.
🔬 Puerto et al., C-SEO Bench, NeurIPS 2025 📊 Martinez, arXiv 2607.14035, Jul 2026 📊 Zhang, He & Yao, arXiv 2604.25707, Apr 2026 (descriptive)
-
Answers change from run to run. Whether a brand appears is moderately stable; its position in the answer is not. Only presence measured over many runs is a reliable number.
🏷 SparkToro / Gumshoe, 27 Jan 2026 📊 Schulte et al., arXiv 2604.07585, Apr 2026
-
Retrievers favor the start of a passage. Dense retrievers found relevant text 15.6% less well, on average, when it sat later in the passage; rerankers were less affected.
🔬 Zeng et al., Findings of EMNLP 2025 📊 Yu, Han & Park, arXiv 2605.26578, 2026
-
Adding schema alone did not raise citations. In the only controlled test, on pages that were already heavily cited, adding JSON-LD produced no significant change in ChatGPT or AI Mode. Sites with schema are cited more, but that is a correlation.
🏷 Ahrefs, 11 May 2026 🏷 AirOps, 13 Apr 2026
-
Site owners now have first-party AI data. Bing reports AI citations and a sample of grounding queries; Search Console reports AI Overviews and AI Mode impressions; GA4 has an AI Assistant channel.
🏷 Bing, 10 Feb 2026 🏷 Google, Jun 2026 🏷 Google Analytics, 13 May 2026
What changed in the methodology
How AI answers work
- BeforeStage 1 described the engine expanding the prompt into searches.NowStage 1 starts with the engine deciding whether to search at all, and brands the model already knows are written into its searches.
- BeforeRetrieval was the Technical stream’s job; Content made pages worth selecting.NowAll three streams act on retrieval: Content carries the meaning, Technical makes pages readable, Business makes the brand known.
- BeforeMethods were listed without saying where they act.NowEvery method is tagged with the stage it acts on, with its evidence.
Content
- BeforeWriting techniques came with expected lifts, such as quotations raising visibility by about 40%.NowThe techniques are principles. Those lifts were measured with the sources already handed to the model and are shown only as history.
- BeforeGEO-16 was presented in five categories with impact percentages.NowGEO-16 uses the paper’s 14 named pillars, as a quality standard that predicts no citation rate.
- BeforeAnswer-first rested on featured-snippet studies.NowAnswer-first rests on retrieval research; the 25–60 word capsule is a TSM default, with its origins named.
- BeforeNo procedure for the engines’ fan-out queries.NowA fan-out procedure: collect the fan-out queries, group them, and decide for each whether to keep, revise or create a page, or earn coverage elsewhere.
- BeforeFAQ blocks, added statistics and quotations were recommended as optimizations.NowHeadings that name the topic, with direct answers, are core practice (a question heading is one optional form). Statistics and quotations are required in new content where they are real; retrofitting old pages with them is not an optimization.
Technical
- BeforeSchema was described as raising citations; FAQPage and HowTo were high priority.NowSchema: entity clarity; no proven citation effect; untested on pages not yet cited. FAQPage and HowTo are not GEO priorities.
- BeforeSome engines were described as executing JavaScript.NowTSM assumes no engine executes JavaScript, so every citable fact is in server-rendered HTML.
- BeforeCrawler guidance included fixed crawl-to-referral ratios.NowCrawler guidance follows each operator’s own documentation, includes CDN defaults, and states Crawl-delay support per operator.
- BeforeNo product-data vocabulary beyond the basics.NowSchema.org 30.1 Product and Offer terms, including the EU Digital Product Passport.
Business
- BeforeWhat the engines know about the brand was not measured.NowA quarterly brand knowledge audit: what each engine says about the brand with search turned off, and an action for each gap.
- BeforeWikipedia was treated as foundational for every engine.NowWikipedia’s weight differs by engine; Wikidata comes first.
- BeforeStream effort was set once for all engines.NowStream effort is weighted per engine, from where each engine’s citations actually land.
Measurement
- BeforeSix primary KPIs, including a position-weighted SOV-AI.NowFive primary KPIs. SOV-AI is a supporting metric, read quarterly, per engine, with no position weights.
- BeforeKPIs and stream metrics had targets.NowNo targets: every KPI is reported as a baseline, a change and a trend.
- BeforeA recommended number of tracked queries, with query weights.NowNo recommended number: the precision each set size gives is shown instead, and every query counts equally.
- BeforeDirect measurement was described as often impossible.NowFirst-party data is used: Bing AI Performance, Search Console AI reports and the GA4 AI Assistant channel.
- BeforeLocation was not controlled.NowThe declared and network location are fixed and recorded for every measurement run.
Removed
- Figures from 2024–2025 that newer data replaced, domain leaderboards and per-domain shares.
- Targets, position weights and numbers with no source.
- Percentage lifts presented as expected results.
What changed on this site
- The homepage and the knowledge pages carry the new evidence, and each page shows the month it was last updated.
Evidence labels: 🔬 peer-reviewed · 📊 preprint or disclosed method · 🏷 vendor, agency or platform · 💡 TSM judgment.