AI Search Audit: 13 Checks and Their Sources

Your site ranks fine on Google, yet ChatGPT and Perplexity never cite it when they answer questions in your niche. Before blaming the content, rule out the technical causes: AI crawlers may not get in at all, or they may get in and find an empty page. This checklist splits the work into 13 checks in 4 layers, notes how much official backing each one has, and runs 7 of them on linusseo.com with this site’s free tools.

Most AI search checklists treat every item as equally important, so llms.txt and AI-specific structured data sit next to robots.txt. The backing behind them is very different. Some checks come straight from OpenAI or Google documentation, some are industry habits, and Google’s 2026 guide says a few of them are not needed at all. So each check below says where its backing comes from: the vendor’s official docs, third-party research or common industry practice.

Key points

• An AI search audit has 4 layers: can crawlers get in, can they read the content, can they use it, and is anyone visiting or citing it. If the first two fail, nothing in the later layers matters.
• In 2026, check your CDN settings first, then robots.txt. Since September 15, 2026, Cloudflare’s Block options also block Googlebot.
• Crawlers come in three kinds: search, user-triggered and training. Blocking training crawlers does not stop AI search from citing you. OpenAI, Perplexity and Google say their user-triggered fetchers may ignore robots.txt.
• Google’s 2026 guide says you do not need llms.txt, content “chunking” or special structured data. Leave those for last.
• The fourth layer now has first-party data: Bing’s AI Performance report, GA4’s AI Assistant channel and Search Console’s generative AI report.

What changed for AI search audits in 2026

A technical audit used to have one reader, Googlebot. Now each AI company runs its own crawlers, and four things changed.

Crawlers split into three kinds with different rules. OpenAI’s and Anthropic’s crawlers each fall into three groups by purpose: a search crawler (such as OAI-SearchBot), a fetcher for pages a user asks the assistant to open (such as ChatGPT-User) and a training crawler (such as GPTBot). Perplexity runs only the first two and says PerplexityBot is not used to train models. Sites that block OAI-SearchBot are not shown in ChatGPT search answers, while blocking GPTBot only opts the content out of training. User-triggered fetchers are different again: OpenAI, Perplexity and Google say they may ignore robots.txt, and only Anthropic states that all of its bots follow it.

The blocking point moved from robots.txt to the CDN. Since July 2025, new domains joining Cloudflare are asked during setup whether to allow AI crawlers, and the default option is to block them. Since September 15, 2026, Cloudflare’s Block options also stop Googlebot, Bingbot and Applebot, because those crawlers collect data for both search and training. A correct robots.txt can still be overridden by one CDN toggle that removes the site from Google Search and AI search together.

Crawlers started proving who they are with signatures. Web Bot Auth lets a crawler prove its identity with a cryptographic signature instead of a User-Agent string, and the IETF chartered a working group for it in October 2025. ChatGPT’s agent browser already signs its requests. Since May 30, 2026, Shopify applies its strictest rate limits to unsigned bots.

The fourth layer has first-party data. In 2026 Bing Webmaster Tools launched an AI Performance report, GA4 added a default AI Assistant channel, and Search Console added a generative AI performance report and an opt-out control. Before that, the only ways to check AI citations were asking the assistants yourself or paying for a third-party tool.

For practitioners: 2024–2026 timeline

• 2024-12-17: Vercel and MERJ publish log research showing the major AI crawlers do not render JavaScript (Vercel).
• 2025-07-01: New domains joining Cloudflare are asked during setup whether to allow AI crawlers, with blocking as the default (Cloudflare).
• 2025-08-29: Shopify lets merchants authorize audit tools with Web Bot Auth (Shopify changelog).
• 2025-09-24: Cloudflare launches Content Signals with search, ai-input and ai-train (Cloudflare).
• Around 2025-12-09: OpenAI rewrites its crawler docs, and the robots.txt section no longer covers ChatGPT-User (PPC Land).
• 2026-02-10: Bing Webmaster Tools launches AI Performance in public preview (Bing).
• Around 2026-03-20: Google-Agent joins Google’s list of user-triggered fetchers (Google).
• 2026-05-13: GA4 adds an AI Assistant channel to the default channel group (GA4 Help).
• 2026-05-15: Google publishes its guide to optimizing for generative AI features (Google).
• 2026-05-30: Shopify tightens rate limits for unsigned bots (Shopify).
• 2026-06-03: Search Console launches the generative AI performance report and opt-out control, worldwide from August 31 (Google).
• 2026-09-15: Cloudflare’s Block options start covering Googlebot, Bingbot and Applebot (Cloudflare).

Four layers, in order of how badly things break

The layers come in order. If layer 1 fails, crawlers never get the page; if layer 2 fails, they get an empty shell. Fix those first; most of their checks are backed by official docs. Layer 3 mixes official requirements with checks that are only industry practice, so pick what is worth your time. Layer 4 is measurement, to confirm the earlier fixes worked.

Google’s AI Overviews and AI Mode use the regular index built by Googlebot. Google says there are no additional requirements to appear in them beyond being indexed and eligible for a snippet. Many checks below therefore matter most for ChatGPT, Perplexity and Claude, which do not use Google’s index.

Layer 1: can AI crawlers get in?

1. robots.txt treats the three kinds of crawler separately

Check that robots.txt does not block AI search crawlers by accident. A common mistake is trying to block training and adding OAI-SearchBot to the Disallow group as well, or blocking everything with User-agent: *. OpenAI, Anthropic and Perplexity each list their crawlers’ names and purposes in their official docs. The usual approach is to allow search and user-triggered crawlers and decide on training crawlers based on how you want your content used. Anthropic’s names are the easiest to mix up: Claude’s search crawler is Claude-SearchBot, and ClaudeBot is for training only.

Two more names are control tokens: they never fetch pages and only declare data use in robots.txt.

• Blocking Google-Extended affects training and grounding for Gemini and other systems, not Google Search or AI Overviews.
• Blocking Applebot-Extended affects only Apple’s model training, not Siri or Spotlight.

In this site’s AI Crawler robots.txt Checker, enter your domain and the path of a real article. It applies RFC 9309 to more than 30 crawlers and flags the ones whose operators say they may ignore robots.txt.

Result for this site: all 31 crawlers are allowed and no common mistakes were found.

2. The CDN, firewall and platform do not block AI crawlers

This is the most important check in 2026, and Google’s own documentation asks you to allow crawling in robots.txt “and by any CDN or hosting infrastructure”. Places to look:

• Cloudflare: open AI Crawl Control for the domain and check the Search, Agent and Training settings. Block or Block on pages with ads also blocks Googlebot, Bingbot and Applebot. If you only want to stop training, use Disallow AI Training.
• Other CDNs and firewalls: look for rules that block bot or AI categories, and for custom rules that match User-Agent strings.
• Shopify: bot protection rate-limits unsigned crawlers. To audit your own store with an SEO tool, generate a Web Bot Auth signature for the tool in the admin.

This site’s What AI Crawlers See tool requests a URL once each with a normal browser, Googlebot and 5 AI crawler User-Agents. If the browser gets a 200 and an AI crawler gets a 403 or a challenge page, a rule is blocking by User-Agent.

Result for this site: all 7 User-Agents got a 200 with the same bytes the browser received, so the hosting layer does not block these crawlers by User-Agent.

3. Verify real crawler visits against official IP ranges

The simulation in check 2 only changes the User-Agent, and its requests do not come from the crawlers’ published IP ranges. Many firewalls check both, so passing the simulation does not prove real crawlers get in. To confirm, filter your server or CDN access logs by each vendor’s published IP list and look at the status codes those requests received.

This site’s AI Crawler Log Analyzer takes an access log and separates real crawlers from impostors using the official IP lists. Anyone can fake a User-Agent, so counting by User-Agent alone can include impostors in your AI crawler numbers.

For practitioners: published crawler IP lists

• OpenAI: https://openai.com/searchbot.json, gptbot.json, chatgpt-user.json, plus adsbot.json added in 2026 (OAI-AdsBot only visits landing pages of ChatGPT ads).
• Perplexity: https://www.perplexity.ai/perplexitybot.json, perplexity-user.json.
• Google: verify Googlebot by reverse DNS or the official IP lists; Google-Agent has its own user-triggered-agents.json.
• Anthropic discourages IP blocking; its verification list is at claude.com/crawling/bots.json.
• Apple: reverse DNS *.applebot.apple.com, IP list at search.developer.apple.com/applebot.json.

Layer 2: once in, can they read the content?

4. Main content and internal links are in the raw HTML

In 2024 Vercel and MERJ analyzed access logs and found that none of the major AI crawlers render JavaScript; only Googlebot (which Gemini also uses) and Applebot do. One or two sources claim that OAI-SearchBot has started rendering JavaScript, but OpenAI has not said so publicly. Agents that operate a page for a user, such as ChatGPT’s agent browser and Google-Agent, are real browsers and do run JavaScript, but the crawlers that build AI search indexes still do not. Content, prices, reviews and FAQs loaded by scripts are therefore likely invisible to the crawlers of ChatGPT, Claude and Perplexity. This finding comes from third-party research; the AI companies have not described it themselves.

The second section of the What AI Crawlers See tool parses only the raw HTML without running JavaScript. It counts visible text, headings, links and JSON-LD, and shows the opening text an AI crawler reads. Paste the text you copied from the rendered page and it estimates how much of it the raw HTML covers.

Result for this site: the raw HTML of the English keyword research guide has about 5,000 words of visible text, 1 H1, 5 H2s, 16 H3s and 93 links. This site is static HTML generated by Hexo, so the text is in the HTML by design. Still, in the Semrush JS Impact report for the Site Audit guide, 191 of 193 pages changed after rendering, mostly in outlink counts and the word counts of tool pages, so even static sites are worth a spot check.

5. noindex, nosnippet and similar directives are not misused

These directives decide how much of a page AI features can use:

• Google: according to Google’s documentation on AI features, noindex removes the page; nosnippet, max-snippet and data-nosnippet limit the text AI Overviews and AI Mode can show.
• Bing and Copilot: Bing’s 2026 webmaster guidelines say NOARCHIVE keeps content out of Copilot answers, and NOCACHE limits Copilot to the URL, title and snippet. However, on October 30, 2026 Bing announced two new preferences, ai-training and ai-grounding, and after March 1, 2027 it will no longer read NOARCHIVE and NOCACHE as preferences for training or grounding.
• Apple: Apple’s updated documentation says nosnippet also keeps content from being used as context for AI answers.

Look for template-level settings, such as a theme that adds max-snippet:0 to every post. The third section of the What AI Crawlers See tool lists these directives for the page.

Result for this site: the meta robots on posts is index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1, with no restriction.

6. Search Console’s generative AI control is set to include

Search Console has a control that excludes a site from AI Overviews, AI Mode and AI features in Discover. It has been available worldwide since August 31, 2026. Google’s announcement says it is not used as a ranking signal for the rest of Search, but sites that opt out get no impressions or traffic from those AI features. When you take over a site, confirm the previous owner did not switch it off. It does not cover training, which is still controlled by Google-Extended.

Layer 3: once read, can they use it?

7. Structured data matches the visible content

Google’s guide says structured data is not required for generative AI search and there is no special schema.org markup to add. However, Google’s general structured data guidelines say not to mark up content that is not visible to readers, and Bing’s 2026 guidelines say markup must accurately reflect visible content; Bing adds that structured data may support clearer grounding but does not guarantee visibility. So this check is about consistency, and adding more markup does not help. Do price, stock, author and dates match between JSON-LD and the page, and does the same author or brand use the same @id across pages?

This site’s Schema Visualizer draws JSON-LD as a relationship graph and checks common errors.

To compare author and brand nodes across several pages at once, use the Entity Consistency Checker.

8. The page’s dates agree

A page usually carries several dates: datePublished and dateModified in JSON-LD, meta tags, the date shown on the page, lastmod in the sitemap and the HTTP Last-Modified header. Google says it uses lastmod only if it is consistently accurate. The idea that AI search prefers fresh content is only industry speculation with no official statement, so the point of this check is to keep systems from misreading a page’s age, not to chase citations by changing dates.

The fourth section of the What AI Crawlers See tool lists the dates from JSON-LD, meta tags, <time> elements, the visible text and the HTTP Last-Modified header side by side. It does not read the sitemap, so check lastmod by opening the sitemap separately.

Result for this site: the structured data, meta tags and visible dates agree. The HTTP Last-Modified header is another matter. This site is hosted on Vercel, and both a 2023 post and a profile photo that has not changed in weeks report a Last-Modified from the day of the latest deployment, so the header says nothing about when content changed.

9. Each section opens with its answer

The common advice is to answer in the first sentence under each heading, keep paragraphs short and cite sources for numbers, because AI answers often quote a single passage. This helps readers too, but the evidence is industry observation, and Google’s 2026 guide says you do not need to break content into small chunks or rewrite it for AI. Do it when you update old posts anyway.

This site’s AI Citability Checker splits an article at its H2s and H3s, checks each section’s first sentence, length, numbers and sources, and explains what backs each suggestion.

Result for this site: for the English keyword research guide, the opening sentence is a statement. The suggestions include a 355-word intro with no list or subheading and sections whose numbers have no source. The article does have a takeaways box, but the tool reported none because its list of labels does not include “Key points”, the label this site uses.

10. llms.txt (can wait until last)

llms.txt is a Markdown index at the site root that lists a site’s important pages for AI. There is no evidence that the major AI search products use it. Google’s John Mueller said in 2025 that no AI system currently uses llms.txt, Google’s 2026 guide lists it among things you do not need, and the crawler docs of OpenAI, Anthropic and Perplexity do not mention it. It can make sense for developer and API documentation, which coding assistants may read; online stores and content sites can skip it. You can check the format with this site’s llms.txt Validator & Generator.

Result for this site: this site has an llms.txt with 0 errors; the one tip is that its links do not point to Markdown versions. This site keeps it to test the tool and does not expect it to bring citations.

Layer 4: is anyone visiting or citing it?

11. AI crawlers and user-triggered fetches in your logs

Access logs answer two questions. First, did AI search crawlers visit, which pages did they fetch, and what status codes did they get? That verifies layer 1. Second, which pages did user-triggered fetchers such as ChatGPT-User, Claude-User and Perplexity-User open? According to OpenAI and others, these requests happen when a user asks something and the assistant needs to open a page. It follows that the pages they fetch most often are the pages AI actually uses in answers. That is this article’s inference; no vendor has published how these fetches relate to final citations.

12. AI Assistant traffic in GA4

According to GA4 Help, since May 13, 2026 GA4’s default channel group has an AI Assistant channel: when the referrer is a recognized AI assistant, the medium is set to ai-assistant. Google has not published the full list of assistants, and according to third-party tests the channel does not backfill and visits without a referrer still land in Direct. ChatGPT adds utm_source=chatgpt.com to its links, so ChatGPT traffic is easier to identify. In June 2026 GA4 also added a Source Group dimension, mainly to group sources such as social media by platform; it also has built-in groups for ChatGPT and Perplexity.

To cover sources the default channel misses (such as Kimi or Doubao), pick them in this site’s AI Traffic Attribution: GA4 / Shopify Rule Generator. It generates the regex for a GA4 custom channel group and versions for Looker Studio, GTM and server logs.

13. Search Console and Bing AI reports

• Bing Webmaster Tools AI Performance: Bing’s official blog announced the public preview in February 2026. It shows how often the site is cited in Copilot and Bing’s AI summaries, which pages are cited and the grounding queries (the phrases the AI used to retrieve content). Third-party reports say a citation share metric was added in June. There is no click data.
• Search Console generative AI performance report: Google announced it on the Search Central blog in June 2026, and it has been worldwide since August 31. It shows impressions in AI features by page, country and device. It has no queries or clicks, combines AI Overviews and AI Mode, and the same data is also counted in the regular performance report.

Both reports show which pages AI uses, but not clicks. For clicks, go back to GA4.

The 13 checks at a glance

Layer	Check	Backing	Tool on this site	linusseo.com result
Get in	1. robots.txt handles three kinds of crawler	Official docs	AI Crawler robots.txt Checker	31 allowed
Get in	2. CDN, firewall and platform do not block	Official docs	What AI Crawlers See	All 7 UAs got 200
Get in	3. Verify real crawlers by official IP	Official docs	AI Crawler Log Analyzer	Not checked
Read	4. Content and links in raw HTML	Third-party research	What AI Crawlers See	About 5,000 words
Read	5. noindex, nosnippet not misused	Official docs	What AI Crawlers See	No restriction
Read	6. Search Console generative AI control	Official docs	Search Console	No data shown
Use	7. Structured data matches visible content	Official docs	Schema Visualizer, Entity Consistency Checker	Not checked separately
Use	8. Dates agree	Official docs	What AI Crawlers See	Agree; Last-Modified unusable
Use	9. Sections open with the answer	Industry practice	AI Citability Checker	Some suggestions
Use	10. llms.txt	Industry practice	llms.txt Validator & Generator	0 errors
Cited	11. AI fetches in logs	Official docs	AI Crawler Log Analyzer	Not checked
Cited	12. GA4 AI Assistant traffic	Official docs	AI Traffic Attribution: GA4 / Shopify Rule Generator	No data shown
Cited	13. Search Console and Bing AI reports	Official docs	Search Console, Bing	No data shown

What this means for you

• Check the CDN and firewall before robots.txt. On Cloudflare, open AI Crawl Control and make sure no Block option is stopping Googlebot. On Shopify, watch how bot protection treats third-party tools.
• Block training crawlers only and allow search crawlers. If you do not want your content used for training, block training crawlers and tokens like Google-Extended, and allow search and user-triggered crawlers. This does not stop AI search from citing you.
• Keep the content in the raw HTML. If a front-end framework or app loads your content, spot-check at least the product, category and article templates.
• Leave the checks that are only industry practice (checks 9 and 10) for last, and connect the layer 4 data first. A few weeks after fixing the first three layers, check the Bing and Search Console AI reports and GA4’s AI Assistant channel for changes.

FAQ

Does blocking Google-Extended remove my site from AI Overviews?

No. Google-Extended controls Gemini training and grounding and does not affect Google Search, AI Overviews or AI Mode. To leave AI Overviews and AI Mode, use Search Console’s generative AI control, or nosnippet and noindex, which also affect regular results.

Can robots.txt stop ChatGPT from opening my pages?

Not completely. robots.txt controls OAI-SearchBot and GPTBot, but OpenAI says ChatGPT-User may not follow it, and the same goes for Perplexity-User and Google-Agent. Only Anthropic states that Claude-User follows robots.txt. To block those fetchers, you have to block their official IP ranges at the server or CDN.

Everything passes. Why am I still not cited?

This checklist only checks whether AI can get and use your content. Whether you get cited also depends on whether your content says something others do not, whether other sites mention your brand, and how each AI system picks its sources.

Appendix: sources

Claim	Backing	Source
Sites that block OAI-SearchBot are not shown in ChatGPT search answers; GPTBot is for training	Official docs	OpenAI crawler docs
ChatGPT-User may not follow robots.txt (rewritten Dec 2025)	Official docs	OpenAI crawler docs; PPC Land 2025-12-09
All three Anthropic bots follow robots.txt; ClaudeBot is for training	Official docs	Anthropic Help Center
Perplexity-User generally ignores robots.txt	Official docs	Perplexity docs
Google’s user-triggered fetchers, including Google-Agent, generally ignore robots.txt	Official docs	Google crawler docs
Google-Extended does not affect inclusion in Google Search	Official docs	Google common crawlers
Applebot-Extended does not crawl; it only controls training use	Official docs	Apple Support
No additional requirements for AI Overviews or AI Mode; allow crawling in robots.txt and the CDN	Official docs	Google: AI features and your website
No need for llms.txt, special markup or chunking; structured data is not required	Official docs	Google generative AI optimization guide
Do not mark up content that is not visible to readers; markup must accurately reflect visible content	Official docs	Google structured data general guidelines; Bing Webmaster Guidelines
The Search Console generative AI control opts a site out of AI Overviews, AI Mode and AI Overviews in Discover, and is not a ranking signal for regular results	Official docs	Google blog, 2026-06-03
The Search Console generative AI report has impressions only, no queries or clicks	Official docs	Search Central blog, June 2026
Cloudflare’s Block options cover Googlebot, Bingbot and Applebot from 2026-09-15	Official docs	Cloudflare blog, 2026-09-15
Bing and Copilot share crawling and indexing; NOARCHIVE keeps content out of Copilot	Official docs	Bing Webmaster Guidelines
Bing is moving to ai-training and ai-grounding preferences; after 2027-03-01 it no longer reads NOARCHIVE and NOCACHE as training or grounding preferences	Official docs	Bing blog, 2026-10-30
GA4’s default channel group has AI Assistant with medium ai-assistant	Official docs	GA4 Help: what’s new
Sitemap lastmod is used only if consistently accurate	Official docs	Google: build a sitemap
The major AI crawlers do not render JavaScript	Third-party research	Vercel × MERJ, Dec 2024
Pages that user-triggered fetchers open most are the pages AI uses in answers	This article’s inference	Based on vendors’ descriptions of user-triggered fetchers
Sections that open with the answer get cited more; AI prefers fresh content	Industry practice	Industry observation and correlation studies; no official statement
llms.txt improves citations in other AI search products	Industry practice	No supporting evidence; Mueller, June 2025 said no AI system currently uses it