Ask ten people why their page never appears in an AI answer, and you will get ten theories, most about files you are supposed to upload or a magic word count. Almost none survive contact with what the search engines actually document.
Table of Contents
The useful way to think about AI search visibility is as a chain with four links. A page has to be in the index, reachable by the crawler that feeds the AI system, retrievable for the exact sub-question being asked, and quotable once retrieved. Break one link and nothing else you do matters. Fix them in order, and you stop guessing.
The step everyone skips: your page has to be retrieved first
Google describes its AI features as using retrieval-augmented generation, which it also calls grounding. The system does not answer from memory. It runs searches, pulls pages from the same index that powers normal results, reads the relevant parts, and writes an answer with links to what it used.
It also does something that changes how you should write: query fan-out. Google’s own example is a search for “how to fix a lawn that’s full of weeds”, which can generate side queries like “best herbicides for lawns” and “remove weeds without chemicals”. Those go out in parallel, and the pages that come back are what the answer gets built from. If the retrieval half of that is new to you, generative AI and chatbots explains how grounding differs from prompting and fine-tuning.
That single mechanism explains most of the confusion around AI search visibility. You are not optimising a page to match one keyword. You are making it the best available answer to one of perhaps six sub-questions that a model invented on the fly. A page that only addresses the head term has nothing to contribute to the fan-out, so it stays out.
The four gates a page has to pass
Gate one: indexed and eligible for a snippet
This sounds basic and is where a surprising number of pages fail. Google’s documentation is explicit that to be eligible for its generative AI features, a page must be indexed and eligible to be shown with a snippet, and the site must not have opted out of AI features in Search Console.
That last part is new enough that people forget it exists. Search Console now has a “Search generative AI” setting under Settings with three choices, and the default includes your site. If someone turned it off during an experiment, your pages are excluded from AI Overviews, AI Mode and generative AI features in Discover no matter how good they are.
Two more self-inflicted wounds live here: a noindex left on a template, or a nosnippet rule added years ago to stop scrapers. Both remove you from AI surfaces without touching normal rankings, which is why the problem looks like “AI search hates my site” when it is really a meta tag. If the crawl-and-index side is still fuzzy, how search engines actually work covers the pipeline these gates sit on.
Gate two: the right crawler is allowed in
Here is the distinction that separates a working checklist from a copied one. Google does not have a separate AI crawler. Its AI features read the normal index, which means Googlebot’s robots.txt rules and your usual crawl controls are the whole story. There is nothing extra to allow.
OpenAI is different, and its crawler documentation describes three identities people constantly mix up. OAI-SearchBot Surfaces sites in ChatGPT’s search features, and opting it out means your site will not be shown in ChatGPT search answers. GPTBot Is about training data. ChatGPT-User Fetches a page when a person’s question calls for it, and robots.txt may not apply to those user-triggered visits. So blocking GPTBot to stay out of training does nothing to search visibility, while blocking OAI-SearchBot quietly removes you from ChatGPT search. Perplexity runs much the same split, with PerplexityBot for indexing and Perplexity-User for user-triggered fetches. Both companies publish IP ranges and note that a robots.txt change can take about a day to apply.
The failure that matters most is not robots.txt at all. It is the firewall. A CDN or WAF rule that blocks “suspicious bots” will happily block OAI-SearchBot, and Search Console will not tell you, because it only knows about Google. The check is dull and worth doing: open your server or CDN logs, filter for OAI-SearchBot and PerplexityBot, and confirm they get 200s rather than 403s. If ChatGPT cites competitors and never you, look there first.
Access is also splitting into a second path. Google’s guide now covers agentic experiences, where browser agents read the rendered page, the DOM, and the accessibility tree rather than a text index. That is a different door into the same building, and AI agents in the workplace are a readable introduction to what those agents are asked to do.
Gate three: you answer a sub-question, not just the head term
Once a page is retrievable, the question is whether it contains the answer to the specific side query the model generated, and that is the part you control with writing.
Take a page targeting “AI search visibility”. The fan-out around that phrase is predictable: how do I check if AI crawlers can reach my site, does llms.txt work, why does my competitor get cited instead of me, how do I measure AI traffic. A page with one H2 called “Why AI search matters” answers none of those. A page with an H2 for each one, and a first sentence that answers the heading, gives the retrieval system something to grab. Put the answer in that first sentence, not after a warm-up paragraph: a model assembling an answer from six sources will not infer a point from text that circles it for four sentences.
Gate four: the passage survives being lifted out
The unit that gets quoted is a passage, not a page, which has a practical consequence: any sentence that only makes sense because of the paragraph above it cannot be cited.
Three habits fix this cheaply. Name the thing instead of saying “it”. Keep numbers attached to their units and source. Avoid “as we saw earlier”, because the reader of the extracted passage did not. None of this is a formatting trick; it is documentation discipline applied to paragraphs that get read out of order.
What Google says you can stop doing
This is where most AI search advice has drifted from the documentation, and where you can save real time.
Google’s guide to optimising for generative AI features has a myth-busting section, and the list is blunt. You do not need llms.txt or any other special file; Google says its Search does not use them, and that publishing one neither helps nor hurts. You do not need to “chunk” content into tiny pieces. You do not need to rewrite in a special style for AI. You do not need special schema.org markup, and chasing inauthentic brand mentions is not a strategy.
The llms.txt point deserves a caveat rather than a slogan. The file was proposed as a clean map of a site for language models, and some tools and agents may yet read it. Publishing one costs an afternoon and does no harm. Just do not expect it to move your visibility in Google or ChatGPT, and be sceptical of anyone selling it as the fix.
One item on that list cuts the other way: scaled content. Google names creating separate pages for every fan-out variation, purely to manipulate AI answers, as a violation of its scaled content abuse policy. Answer the questions inside one good page rather than manufacturing forty thin ones. The same logic kills the market for bought mentions. Google states that inauthentic mentions are no shortcut, because the quality and spam systems applying to normal search apply to AI features too.
The thing competitors underrate: be the source, not the summary
Every guide on this topic says to write “helpful, authoritative content”. That is correct and useless, because it does not say what to write instead of what you have. The concrete version: AI answers cite pages that state something first. A definition others copy. A table of specs nobody else assembled. A named method with steps in a fixed order. A number with a date and a source attached. Those pages get pulled into fan-out queries because the model wants a fact it can attribute, and a summary of five other pages is the last thing it needs. Google’s guide says the same without the jargon: unique point of view, non-commodity content, something that could not have been produced by summarising what already ranks.
Measuring it without fooling yourself
Three systems report on AI visibility and they do not agree, so start with the narrowest and most official number. Search Console’s generative AI performance report shows impressions from AI Overviews and AI Mode by page, country, device and date. No clicks, no position, no queries. It is a dedicated view, not an extra pile: the same impressions already sit in your main Performance report, so never add the two.
What it does not tell you is whether any of that visibility becomes a visit. Google Analytics 4’s “AI Assistant” channel covers ChatGPT, Gemini, Copilot, DeepSeek and Grok, and excludes AI Overviews and AI Mode, so clicks from Google’s AI features arrive looking like ordinary organic traffic. That gap is the whole measurement problem, and how to measure AI search traffic covers the practical way to report it. If the result has to sit in a wider marketing report, digital marketing technology covers where it fits.
Once a month, screenshot the Search Console report and look at two things: which pages are cited at all, and which are cited heavily with almost no clicks. Cited-but-not-clicked means the page answers completely in the snippet zone: good for brand visibility, bad for traffic.
A ninety-minute pass you can run on one page
If you want to test this on one URL this week rather than rebuild a site, work in this order.
- Confirm the page is indexed, has no
noindex, allows a snippet, and that the property includes AI features in the Search generative AI setting. - Check your CDN logs for
OAI-SearchBotandPerplexityBot. If they are blocked, fix that before writing a word. - Load the page with JavaScript disabled. If the answer text is gone, you are invisible to any crawler that does not execute scripts, and no AI crawler documentation promises that it does.
- Write down five sub-questions a real buyer would ask. If no H2 matches one, that is your rewrite.
- Read each H2 with only its first sentence. If it does not answer the heading alone, rewrite it.
- Add one thing nobody else has: a definition, a comparison table, a spec, a dated number.
Nothing on that list involves a new file format, a plugin or a service. AI search visibility turns out to be the work of being findable, reachable, specific and quotable, in that order. Get gate two wrong on a site behind an aggressive firewall and you can rewrite every heading for a year without moving a single citation.





