# Cloudflare Wrote Two Thirds of YC B2B AI Crawler Blocks (/blog/ai-crawler-study)
import { Step, Steps } from 'fumadocs-ui/components/steps';
When a Y Combinator B2B company blocks AI crawlers, it usually did not write the block itself. Cloudflare did.
We fetched the robots.txt file of every active B2B company with a website in Y Combinator's public directory, 2,243 sites in all, on September 11, 2026. Of the 2,207 sites we could reach, 168 (7.6%) block at least one AI crawler by name. 112 of those 168 (66.7%) carry Cloudflare's managed robots.txt section: one setting, and the same list on every site.
> **October 2026 update:** We fetched the same sites again on October 1, 2026. Cloudflare's section is now gone from 112 of the 114 files that carried it, and on 63 of those 112 sites the homepage refuses a request that names GPTBot while serving a browser. The figures below are the September 11 reading. The [second reading](#second-reading-october-2026) has what changed.
A robots.txt file is a plain-text file at the root of a website that tells crawlers which pages they may fetch. Crawlers choose whether to honour it.
Below is what we found, what it means for your own site, and exactly how we counted, including what this method cannot see.
> **Key Takeaways**
>
> * 168 of 2,207 reachable sites (7.6%) block at least one AI crawler by name, and Cloudflare's managed robots.txt section wrote 112 of those 168 blocks (66.7%).
> * Cloudflare's list blocks Google-Extended, which Google says controls grounding in Gemini Apps as well as Gemini training. Read in full, 104 of the 114 files carrying the section (91.2%) refuse both; 9 re-allow Google-Extended with a rule of their own, and 1 carries a section that does not name it.
> * 39 of the 114 sites carrying Cloudflare's section (34.2%) also publish an llms.txt file, a file that exists to help AI read a site.
> * 164 of 2,207 sites (7.4%) block at least one training or dataset crawler while allowing all 3 answer-engine crawlers. 0 do the reverse. For Gemini, that split is not available: one Google token covers both.
> * 222 of the 1,822 robots.txt files we collected (12.2%) name `anthropic-ai` or `Claude-Web`, which Anthropic no longer documents. Only 18 of them block those names.
> * October 1, 2026: Cloudflare's section is gone from 112 of the 114 files, and sites blocking an AI crawler by name fell from 168 of 2,207 (7.6%) to 65 of 2,200 (3.0%). On 63 of those 112 sites, the homepage now refuses GPTBot at the network edge.
## Second Reading, October 2026
*Added October 2, 2026. The rest of this page is the September 11, 2026 reading and stays as published.*
We fetched the same 2,243 sites again on October 1, 2026, with the same fetcher and the same analysis. The largest change is not one that site owners wrote. Cloudflare's managed section left robots.txt.
| | September 11, 2026 | October 1, 2026 |
| --------------------------------------------------- | ------------------ | --------------- |
| Reachable sites | 2,207 | 2,200 |
| Sites serving a robots.txt file | 1,822 | 1,799 |
| Sites blocking at least one AI crawler by name | 168 (7.6%) | 65 (3.0%) |
| Files carrying Cloudflare's managed section | 114 | 2 |
| Managed sections that block an AI crawler | 112 | 0 |
| By-name blocks written outside Cloudflare's section | 56 | 65 |
So the fall from 7.6% to 3.0% is almost entirely Cloudflare's section disappearing. Blocks that site owners wrote themselves went up slightly, from 56 sites to 65.
### What Happened to the 112 Files
Of the 114 files that carried the section in September, 112 no longer do. Read site by site:
* 82 still serve a robots.txt file, now without the section
* 20 return a 404 for robots.txt, so Cloudflare's section was the whole file
* 8 answer with an HTML page instead of a file
* 1 returned a server error and 1 could not be reached
The 2 files that still carry the section block nothing. One holds only a Content-Signal line, and the other carries a comment saying its owner reversed the block on purpose.
Only 4 of the 112 sites still block an AI crawler by name in robots.txt, with rules of their own.
### The Block Moved to the Network Edge on 63 of the 112 Sites
A robots.txt line is a request. A block at the CDN is enforced, and it never shows up in the file. So on October 2, 2026, we asked each homepage directly.
We requested the homepage of each of the 112 sites three times: as a desktop Chrome browser, as GPTBot (OpenAI's training crawler), and as OAI-SearchBot (OpenAI's search crawler), using the user-agent strings on [OpenAI's crawler page](https://developers.openai.com/api/docs/bots). We did the same for 112 comparison sites from the same sample that were served through Cloudflare on both fetches and never carried the managed section.
| Homepage response on October 2, 2026 | 112 sites that lost the section | 112 comparison sites |
| ---------------------------------------------- | ------------------------------- | -------------------- |
| Browser 200, GPTBot refused, OAI-SearchBot 200 | 63 (56.3%) | 3 (2.7%) |
| All three answered 200 | 46 (41.1%) | 107 (95.5%) |
| All three refused or challenged | 2 | 1 |
| Other | 1 | 1 |
On 63 of the 112 sites, the homepage served the browser and refused GPTBot with a 403. Among the comparison sites, 3 of 112 did. All 63 refusals came from Cloudflare's servers, and 61 of them were the same 25-byte reply: "Your request was blocked."
OAI-SearchBot was served on all 109 of the 112 sites that served the browser. That matches September's section, which never blocked an answer-engine crawler.
On 46 of the 112, all three requests got the same page, so as far as this check can see, the block ended. 10 of those 46 are no longer served through Cloudflare at all, going by the server header.
The two "other" rows: one site failed its secure connection for the browser and OAI-SearchBot (a 525) and refused GPTBot, and one comparison site served the browser and answered both crawler requests with a 429 (too many requests).
A user-agent string is not the crawler. The real GPTBot comes from OpenAI's published IP addresses, and our requests did not. A site that checks crawler addresses can treat our request differently from the real one, in either direction. A 403 here shows the edge refuses requests that name GPTBot. It does not show what the real GPTBot receives. We also did not run this check in September, so we cannot say when each edge block began.
### What Cloudflare Says About the Change
In a [blog post on September 15, 2026](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), under "What changes on September 15?", Cloudflare wrote: "Managed Robots.txt will be deprecated in favor of Bot Preference Sync. Customers who enabled Managed Robots.txt will migrate to the new system."
Its [post introducing Bot Preference Sync](https://blog.cloudflare.com/bot-preference-sync/) (August 21, 2026, updated September 15) says the new feature writes its own section to robots.txt, opening with `# BEGIN Cloudflare Bot Preference Sync`. For "existing customers who are using the legacy managed robots.txt feature", it says Cloudflare will "prompt you to review and confirm your preferences to transition".
On October 1, 0 of the 1,799 robots.txt files in our sample carried that new marker. The old section left 112 files, and the new one appeared on none.
Cloudflare's documentation does not say why the old section left without the new one in its place. Its [managed robots.txt page](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/), last updated August 3, 2026, still describes the setting with no mention of a deprecation. We found no documentation page for Bot Preference Sync, and Cloudflare's changelog has no entry about robots.txt between September 11 and October 2, 2026.
The same September 15 post describes the edge side. Previous Training selections of "Block or Block on pages with ads will migrate to Disallow AI Training", and under that setting, "Every other training crawler is blocked, including the training-only crawlers run by Amazon, Anthropic, Meta, and OpenAI." Our data cannot see which setting each site has, so we do not claim that this is what refused GPTBot on the 63 sites.
### The Gemini Opt-Out Did Not Move to the Edge
In September, 104 of the 114 files carrying Cloudflare's section refused Google-Extended. On October 1, 5 of those 104 sites still refuse it in robots.txt.
That one cannot move to the edge. Google says Google-Extended "doesn't have a separate HTTP request user agent string", so there is no request for a CDN to refuse. A robots.txt line is the only way to say it. On the other 99 sites, the robots.txt opt-out from Gemini grounding and training is gone, whether or not the owner meant it to be.
### What to Check on Your Own Site Now
**Read your robots.txt again.** Search for `BEGIN Cloudflare Managed content` and `BEGIN Cloudflare Bot Preference Sync`. If you switched the old setting on and neither is there now, your file changed without anyone at your company editing it.
**Read your Cloudflare security settings.** Look at what Search, Training, and Agent are set to. If your edge refuses AI crawlers, that is where the block lives, and robots.txt does not show it.
**Request your homepage as a crawler.** Send a request with GPTBot's user-agent string from [OpenAI's crawler page](https://developers.openai.com/api/docs/bots) and compare the status code with a normal browser visit. A 403 means your edge refuses requests that name that crawler.
**Decide on Google-Extended in writing.** If you want to stay out of Gemini grounding and training, it has to be a robots.txt line. No edge setting can say it for you.
If you would rather have your crawler settings fixed for you, that is the first job of our [technical SEO service](/services/technical-seo).
## Cloudflare Wrote 112 of the 168 AI Crawler Blocks
In 112 of the 168 sites that block an AI crawler by name, the block comes from a section that opens with the comment `# BEGIN Cloudflare Managed content`.
That section is what Cloudflare serves when a site owner turns on its managed robots.txt setting. [Cloudflare's documentation](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) says that with the setting on, "Cloudflare generates and maintains a robots.txt file that instructs known AI crawlers to stay away from your content."
The section appears in 114 of the 1,822 robots.txt files (6.3%), and 112 of those 114 files (98.2%) block at least one AI crawler by name. For context, 839 of the 2,207 reachable sites (38.0%) are served through Cloudflare, going by the server header.
In 113 of the 114 files that carry it, the section blocks the same 9 crawlers: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, CloudflareBrowserRenderingCrawler, Google-Extended, GPTBot, and meta-externalagent. It never blocks an answer-engine crawler such as OAI-SearchBot, Claude-SearchBot, or PerplexityBot.
### Someone chose the setting, Cloudflare chose the list
Someone at each company turned the setting on. But Cloudflare decided which crawlers that means, and Cloudflare keeps the text current.
The evidence is in the file. 113 of the 114 files with the section (99.1%) carry `use=reference`, a Content Signals value Cloudflare [announced on July 1, 2026](https://blog.cloudflare.com/content-independence-day-ai-options/) it would add for "all customers who have already enabled managed robots.txt." So the list on your site can change without anyone at your company editing a file.
### People wrote the rest
In this sample, Cloudflare is the only platform writing AI crawler rules at any scale. Among the blocks Cloudflare did not write, only 1 came from another platform or plugin setting.
People wrote or pasted in the other blocks. Here is how they break down, out of the 168:
* 29 short hand-written lists with no marker
* 16 short lists whose comments state a policy
* 7 long lists of 15 or more agents with no marker
* 3 lists typed inside an SEO plugin's editor markers, where the plugin was used but the owner wrote the rules
No site blocks exactly Cloudflare's 9 crawlers without also carrying its markers.
## The Same Setting Also Opts You Out of Gemini Grounding
Cloudflare frames the setting as a training choice. In its dashboard, the toggle is called "Set your preference to block training in robots.txt".
One of the 9 crawlers it blocks is Google-Extended. According to [Google's crawler documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) (updated July 14, 2026), Google-Extended controls whether content "may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding ... in Gemini Apps and Grounding with Google Search on Vertex AI."
Google-Extended is a control token, not a separate crawler: Google says it "doesn't have a separate HTTP request user agent string". Google Search is governed by a different token, Googlebot.
Cloudflare's section names Google-Extended in 113 of the 114 files that carry it. Read in full, 104 of those 114 files (91.2%) refuse it, so those sites ask Google not to use their pages for Gemini grounding as well as Gemini training. Of the other 10, 9 add a rule of their own that lets Google-Extended back in, and 1 carries a section that does not name it.
That may be the most practical finding here. A setting presented as a training preference also decides whether Gemini may ground its answers in your pages.
## A Third of Cloudflare's Blockers Also Publish llms.txt
An llms.txt file is a Markdown file at `/llms.txt`, proposed at [llmstxt.org](https://llmstxt.org/), that summarizes a site for AI systems. Its only purpose is to help AI read the site.
39 of the 114 sites carrying Cloudflare's section (34.2%) also publish a valid llms.txt. Across all 168 sites that block an AI crawler by name, 84 (50.0%) do.
So a third of the sites carrying a section that tells GPTBot and ClaudeBot to stay out also publish a file written for AI readers. That is not always a contradiction: Cloudflare's list leaves answer-engine crawlers alone, and a site may want those to read it. But it is a reason to check that the two files say what you mean.
Overall, 908 of the 2,207 reachable sites (41.1%) publish a valid llms.txt.
We report adoption only. We make no claim that llms.txt helps a site get cited, and [Ahrefs' study of llms.txt files](https://ahrefs.com/blog/llmstxt-study/) (June 15, 2026) found they are rarely read. If you are deciding whether to publish one, our [llms.txt guide](/guides/llms-txt) walks through that decision.
One signal was consistent. No robots.txt file that declares `ai-train=yes` in a Content-Signal line also blocks GPTBot: 0 of the 1,822.
## No Site Blocks AI Answer Engines While Allowing Training
We grouped the 13 crawlers in this study by what their operators say they are for: training and dataset crawlers, answer-engine crawlers, and fetchers that act when a person asks. The full list, with each operator's page, is in the methodology below.
164 of the 2,207 reachable sites (7.4%) block at least one training or dataset crawler and none of the 3 answer-engine crawlers (OAI-SearchBot, Claude-SearchBot, and PerplexityBot). The reverse, blocking answer engines while allowing training, occurs on 0 sites.
That split is available for OpenAI and Anthropic, which publish separate tokens for training and for search. It is also available for Perplexity, which says PerplexityBot "is not used to crawl content for AI foundation models."
It is not available for Gemini. Google-Extended is the one token Google offers for Gemini, and it covers training and grounding together, so a site that blocks it has refused both.
Here is the share of the 2,207 reachable sites that block each crawler site-wide, whether by naming it or through a rule for all crawlers:
Read the user-fetcher figures with care. OpenAI says of ChatGPT-User that "robots.txt rules may not apply", and Perplexity says Perplexity-User "generally ignores robots.txt rules", because a person asked for the page. A block there is a weak instruction.
## Allow Lists Still Name Crawlers Anthropic No Longer Documents
222 of the 1,822 robots.txt files (12.2%) name `anthropic-ai` or `Claude-Web`. [Anthropic's crawler page](https://support.claude.com/en/articles/8896518) (dated April 7, 2026) documents three agents: ClaudeBot, Claude-User, and Claude-SearchBot. Neither older name is on it.
Most are not block lists. Only 18 of the 222 files block those names.
Most of the others name them in a group with an explicit `Allow: /`. These are sites trying to let Claude in, using names Anthropic no longer documents. None of these names comes from Cloudflare's section.
## What This Means for Your B2B Site
These figures describe one group of companies, but the checks apply to any site. Start with these six:
**Check whether a CDN setting speaks for you.** Open `/robots.txt` on your own domain and search for `BEGIN Cloudflare Managed content`. If it is there, the list of blocked crawlers is Cloudflare's, and it changes when Cloudflare changes it.
*October 2026:* by October 1 the section had left 112 of the 114 files that carried it. Search for `BEGIN Cloudflare Bot Preference Sync` as well, and see the [second reading](#second-reading-october-2026).
**Decide per crawler, not per category.** Training, search, and user-triggered fetching are separate jobs, and most operators give each its own token. Our [AI crawlers guide](/guides/ai-crawlers) lists the token for each one.
**Know what Google-Extended covers.** Blocking it refuses Gemini training and grounding in Gemini Apps together. It is not the Google Search token; Googlebot is.
**Make robots.txt and llms.txt agree.** If you publish an llms.txt while blocking the crawlers of the same companies, decide which of the two you mean.
**Use the names operators document today.** If your file names `anthropic-ai` or `Claude-Web`, check that it also names ClaudeBot, Claude-User, and Claude-SearchBot the way you intend.
**Look past robots.txt.** Your CDN or firewall can block a crawler that your robots.txt allows, and those rules never show up in the file.
## Methodology
This section sets out the sample, the definitions, and the rules behind every figure above.
### Sample and source
The sample is every company in Y Combinator's public directory that is marked active, carries the industry B2B, and lists a website. We read the directory on September 10, 2026, through the community-maintained mirror at [yc-oss.github.io](https://yc-oss.github.io/api/companies/all.json). We kept one record per unique host, with `www.` removed: 2,243 sites.
### Collection
We made one capture pass, on September 11, 2026. For each site we requested `/robots.txt` over HTTPS, following redirects, and retried on the `www.` host if the first request failed at the network level. When robots.txt answered, we requested `/llms.txt` once from the same origin. Every request carried a user agent that names RankZeroResearch and links to rankzero.io.
### Denominators
Every figure on this page names its denominator the first time it appears. These are the counts behind them:
* 2,243 sites in the sample
* 36 unreachable: a network error, or a server error (status 500 or above) on robots.txt
* 2,207 reachable, the denominator for every site-level figure
* 1,822 of the reachable sites served a robots.txt file (HTTP 200 and not an HTML page), the denominator for file-level figures
* 385 reachable sites had no file (any other response, or an HTML page in its place), which we count as allowing every crawler
### What "blocked" means
A crawler is blocked when, under the robots.txt standard ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309)), it may not fetch the root path `/`. We read the group that names the crawler or, if no group names it, the group for all crawlers (`User-agent: *`). Groups naming the same crawler are merged, the longest matching rule wins, and `Allow` wins a tie. Rules for specific paths, such as a disallowed `/checkout`, do not count.
"Blocked by name" means the deciding group names the crawler. A rule for all crawlers can block too, but it is not an AI-specific decision, so the headline figures count blocks by name only. Counting those rules as well, 170 of the 2,207 sites (7.7%) block at least one of the 13 crawlers, and 4 sites block every crawler they do not name individually.
### The crawler list
We fixed the list on September 11, 2026, and checked every name on its operator's own page that day. The purposes below are the operators' own words.
Training and dataset crawlers:
* [GPTBot](https://developers.openai.com/api/docs/bots) (OpenAI): crawls content "that may be used in training our generative AI foundation models"
* [ClaudeBot](https://support.claude.com/en/articles/8896518) (Anthropic): collects web content "that could potentially contribute to their training"
* [Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) (Google): a control token for Gemini training and grounding, quoted in full above
* [Applebot-Extended](https://support.apple.com/en-us/119829) (Apple): lets publishers "opt-out from having their content used to train generative foundation models"
* [CCBot](https://commoncrawl.org/ccbot) (Common Crawl): gathers an open web crawl, whose stated aim is that "free access to web crawl data encourages collaboration and interdisciplinary research"; the page does not mention training
* [meta-externalagent](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) (Meta): crawls "for use cases such as training foundation AI models or improving products by indexing content directly"
* Bytespider (ByteDance): ByteDance publishes no page for it, and the third-party bot directories Known Agents and Dark Visitors list it as a training crawler
Meta's and Common Crawl's pages describe more than training, and ByteDance publishes nothing, which is why we call this group "training and dataset crawlers".
Answer-engine crawlers:
* [OAI-SearchBot](https://developers.openai.com/api/docs/bots) (OpenAI): "used to surface websites in search results in ChatGPT's search features"
* [Claude-SearchBot](https://support.claude.com/en/articles/8896518) (Anthropic): "navigates the web to improve search result quality for users"
* [PerplexityBot](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) (Perplexity): "designed to surface and link websites in search results on Perplexity"
User-triggered fetchers:
* [ChatGPT-User](https://developers.openai.com/api/docs/bots) (OpenAI): acts on user requests, and "robots.txt rules may not apply"
* [Claude-User](https://support.claude.com/en/articles/8896518) (Anthropic): "When individuals ask questions to Claude, it may access websites using a Claude-User agent"
* [Perplexity-User](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) (Perplexity): fetches on a user's request and "generally ignores robots.txt rules"
We also measured [Googlebot](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) and [Bingbot](https://blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat), the main crawlers for Google and Bing search, as a reference, and the two names Anthropic no longer documents, `anthropic-ai` and `Claude-Web`. None of them is among the 13.
### How we attributed authorship
For each blocking file, we looked for text that a platform or tool writes, and applied these rules in this order:
1. **Cloudflare:** the file carries the section that opens with `# BEGIN Cloudflare Managed content`.
2. **Another platform or plugin setting:** the file carries a marker that a website builder, a CMS plugin, or an ad-network plugin writes. We read each marker from the tool's source code or a live file on September 11, 2026.
3. **Long list:** 15 or more blocked agents in named groups, with no marker.
4. **SEO plugin wrapper:** the rules sit inside an SEO plugin's editor markers. The plugin was used, but the owner typed the rules, so we do not credit the plugin.
5. **Stated policy:** the file's comments state a policy, for example by mentioning "AI training" or "answer engine".
6. **Hand-written:** everything else.
We also checked for agent names that only a widely shared community block list carries, and no site matched. Cloudflare's marker is exact. The other markers were read from source, but only 1 blocking site in this sample carries one, so they are barely tested here. A Content-Signal line on its own cannot identify its author, because several platforms write identical lines, so we never attribute a file on that line alone.
### What counts as a valid llms.txt
A valid llms.txt means `/llms.txt` on the same origin answered HTTP 200 with a `text/plain` or `text/markdown` content type, was not an HTML page, and opens with a Markdown heading (`# Title`), as [llmstxt.org](https://llmstxt.org/) specifies. We did not judge quality: some valid files hold little more than a title.
### Second reading (October 2026)
The second fetch ran on October 1, 2026, with the same fetcher, the same list of 2,243 sites, and the same analysis script, unchanged. 2,200 sites were reachable and 1,799 served a robots.txt file.
The homepage check ran once, on October 2, 2026. Each site's homepage got one request per user agent, with a 20-second timeout, one retry on a network error only, and every header identical except the user agent. A refusal means a 401, 402, 403, 406, 429, or 451 status, or a Cloudflare challenge page.
The comparison set is 112 sites from the same sample, one per site that lost the section. Each was served through Cloudflare on both fetches, served a robots.txt file in September, and carried no managed section on either date. 103 come from the same Y Combinator batch as their pair; for the other 9, no such site was left, so we took the nearest eligible site in the directory's order.
### Minimum group size
We give no percentage for any group smaller than 10 sites.
## What This Study Cannot See
Every method has blind spots. These are ours:
* **robots.txt is a request, not enforcement.** Blocks set at a CDN, firewall, or server never appear in the file, so every blocking figure here is a floor. Cloudflare says the same about its own lines: items in robots.txt "signal a website owner's preference, rather than issuing blocks directly."
* **Cloudflare's September 15, 2026 defaults will not show up here.** In its [July 1, 2026 post](https://blog.cloudflare.com/content-independence-day-ai-options/), Cloudflare wrote: "For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads." Site owners opt out in their Security settings. Those are security settings enforced by Cloudflare, not robots.txt text, so our next fetch may show little change. If so, the change will be real but invisible to a robots.txt study like this one. *Correction, October 2, 2026:* the next fetch did show a change. Cloudflare's managed section left 112 of the 114 files, and a homepage check found GPTBot refused at the edge on 63 of those 112 sites.
* **The Cloudflare share depends on the crawler list.** We count 13 named crawlers. Counting more bots as AI, such as Amazonbot, which Cloudflare's list blocks but our 13 do not include, changes the denominator.
* **We cannot see intent.** We can see that a site switched Cloudflare's setting on. We cannot see whether the owner knew which crawlers it blocks.
* **This is one population.** Y Combinator's active B2B companies lean AI-forward. These figures describe them, not B2B software in general or the web. This edition has no ecommerce comparison.
* **We read each file once.** Page-level meta tags and HTTP headers are out of scope, and files change. *October 2026:* the files have now been read twice, on September 11 and October 1, 2026.
* **The October homepage check sends a crawler's name, not the crawler.** Our requests carried GPTBot's and OAI-SearchBot's user-agent strings from an address that is not OpenAI's, so a site that verifies crawler addresses may answer the real bot differently.
## Conflict of Interest
RankZero sells done-for-you AI SEO services and runs a free AI SEO audit tool, which checks AI crawler access among other things. We have a commercial interest in site owners paying attention to this topic.
To keep that interest out of the numbers, every figure comes from a fresh public fetch, not from our audit data. We make no claim that any robots.txt or llms.txt setting improves citations or rankings.
## Data, Code, and the Next Edition
The aggregate data and the code behind every figure (the fetcher, the robots.txt parser, and the analysis script) are available on request: write to [support@rankzero.io](mailto:support@rankzero.io). The format for a public release is still being decided.
The second edition is due in late September 2026. We will fetch the same 2,243 sites again after Cloudflare's September 15 change and report what moved.
*October 2026:* the second fetch ran on October 1, 2026, and the homepage check on October 2. Both are reported in the [second reading](#second-reading-october-2026) above.
## Check Your Own AI Crawler Access
The quickest check is to open your own `/robots.txt` and look for the Cloudflare marker.
For a fuller read, run our [free AI SEO audit](/tools/ai-seo-audit) on your domain. It checks your robots.txt and whether AI crawlers can reach your pages, with no signup. If you would rather go through the results with us, [book an AI search audit call](/services).