When a Y Combinator B2B company blocks AI crawlers, it usually did not write the block itself. Cloudflare did.
We fetched the robots.txt file of every active B2B company with a website in Y Combinator's public directory, 2,243 sites in all, on September 11, 2026. Of the 2,207 sites we could reach, 168 (7.6%) block at least one AI crawler by name. 112 of those 168 (66.7%) carry Cloudflare's managed robots.txt section: one setting, and the same list on every site.
A robots.txt file is a plain-text file at the root of a website that tells crawlers which pages they may fetch. Crawlers choose whether to honour it.
Below is what we found, what it means for your own site, and exactly how we counted, including what this method cannot see.
Key Takeaways
- 168 of 2,207 reachable sites (7.6%) block at least one AI crawler by name, and Cloudflare's managed robots.txt section wrote 112 of those 168 blocks (66.7%).
- Cloudflare's list blocks Google-Extended, which Google says controls grounding in Gemini Apps as well as Gemini training. Read in full, 104 of the 114 files carrying the section (91.2%) refuse both; 9 re-allow Google-Extended with a rule of their own, and 1 carries a section that does not name it.
- 39 of the 114 sites carrying Cloudflare's section (34.2%) also publish an llms.txt file, a file that exists to help AI read a site.
- 164 of 2,207 sites (7.4%) block at least one training or dataset crawler while allowing all 3 answer-engine crawlers. 0 do the reverse. For Gemini, that split is not available: one Google token covers both.
- 222 of the 1,822 robots.txt files we collected (12.2%) name
anthropic-aiorClaude-Web, which Anthropic no longer documents. Only 18 of them block those names.
Cloudflare Wrote 112 of the 168 AI Crawler Blocks
In 112 of the 168 sites that block an AI crawler by name, the block comes from a section that opens with the comment # BEGIN Cloudflare Managed content.
Show the data
| Author of the block | Sites |
|---|---|
| Cloudflare's managed section | 112 |
| Short hand-written list | 29 |
| Short list with a stated policy | 16 |
| Long list, 15+ agents, no marker | 7 |
| Typed inside an SEO plugin's markers | 3 |
| Another platform or plugin setting | 1 |
That section is what Cloudflare serves when a site owner turns on its managed robots.txt setting. Cloudflare's documentation says that with the setting on, "Cloudflare generates and maintains a robots.txt file that instructs known AI crawlers to stay away from your content."
The section appears in 114 of the 1,822 robots.txt files (6.3%), and 112 of those 114 files (98.2%) block at least one AI crawler by name. For context, 839 of the 2,207 reachable sites (38.0%) are served through Cloudflare, going by the server header.
In 113 of the 114 files that carry it, the section blocks the same 9 crawlers: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, CloudflareBrowserRenderingCrawler, Google-Extended, GPTBot, and meta-externalagent. It never blocks an answer-engine crawler such as OAI-SearchBot, Claude-SearchBot, or PerplexityBot.
Someone chose the setting, Cloudflare chose the list
Someone at each company turned the setting on. But Cloudflare decided which crawlers that means, and Cloudflare keeps the text current.
The evidence is in the file. 113 of the 114 files with the section (99.1%) carry use=reference, a Content Signals value Cloudflare announced on July 1, 2026 it would add for "all customers who have already enabled managed robots.txt." So the list on your site can change without anyone at your company editing a file.
People wrote the rest
In this sample, Cloudflare is the only platform writing AI crawler rules at any scale. Among the blocks Cloudflare did not write, only 1 came from another platform or plugin setting.
People wrote or pasted in the other blocks. Here is how they break down, out of the 168:
- 29 short hand-written lists with no marker
- 16 short lists whose comments state a policy
- 7 long lists of 15 or more agents with no marker
- 3 lists typed inside an SEO plugin's editor markers, where the plugin was used but the owner wrote the rules
No site blocks exactly Cloudflare's 9 crawlers without also carrying its markers.
The Same Setting Also Opts You Out of Gemini Grounding
Cloudflare frames the setting as a training choice. In its dashboard, the toggle is called "Set your preference to block training in robots.txt".
One of the 9 crawlers it blocks is Google-Extended. According to Google's crawler documentation (updated July 14, 2026), Google-Extended controls whether content "may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding ... in Gemini Apps and Grounding with Google Search on Vertex AI."
Google-Extended is a control token, not a separate crawler: Google says it "doesn't have a separate HTTP request user agent string". Google Search is governed by a different token, Googlebot.
Cloudflare's section names Google-Extended in 113 of the 114 files that carry it. Read in full, 104 of those 114 files (91.2%) refuse it, so those sites ask Google not to use their pages for Gemini grounding as well as Gemini training. Of the other 10, 9 add a rule of their own that lets Google-Extended back in, and 1 carries a section that does not name it.
104 of 114 files carrying Cloudflare's section refuse Google-Extended, the token that covers Gemini training and grounding
Of the other 10: 9 re-allow Google-Extended with a rule of their own, and 1 carries a section that does not name it.
Show the data
| Files carrying Cloudflare's section | Count |
|---|---|
| Refuse Google-Extended | 104 |
| Re-allow it with their own rule | 9 |
| Section does not name it | 1 |
| Total | 114 |
That may be the most practical finding here. A setting presented as a training preference also decides whether Gemini may ground its answers in your pages.
A Third of Cloudflare's Blockers Also Publish llms.txt
An llms.txt file is a Markdown file at /llms.txt, proposed at llmstxt.org, that summarizes a site for AI systems. Its only purpose is to help AI read the site.
39 of the 114 sites carrying Cloudflare's section (34.2%) also publish a valid llms.txt. Across all 168 sites that block an AI crawler by name, 84 (50.0%) do.
So a third of the sites carrying a section that tells GPTBot and ClaudeBot to stay out also publish a file written for AI readers. That is not always a contradiction: Cloudflare's list leaves answer-engine crawlers alone, and a site may want those to read it. But it is a reason to check that the two files say what you mean.
Overall, 908 of the 2,207 reachable sites (41.1%) publish a valid llms.txt.
We report adoption only. We make no claim that llms.txt helps a site get cited, and Ahrefs' study of llms.txt files (June 15, 2026) found they are rarely read. If you are deciding whether to publish one, our llms.txt guide walks through that decision.
One signal was consistent. No robots.txt file that declares ai-train=yes in a Content-Signal line also blocks GPTBot: 0 of the 1,822.
No Site Blocks AI Answer Engines While Allowing Training
We grouped the 13 crawlers in this study by what their operators say they are for: training and dataset crawlers, answer-engine crawlers, and fetchers that act when a person asks. The full list, with each operator's page, is in the methodology below.
164 of the 2,207 reachable sites (7.4%) block at least one training or dataset crawler and none of the 3 answer-engine crawlers (OAI-SearchBot, Claude-SearchBot, and PerplexityBot). The reverse, blocking answer engines while allowing training, occurs on 0 sites.
That split is available for OpenAI and Anthropic, which publish separate tokens for training and for search. It is also available for Perplexity, which says PerplexityBot "is not used to crawl content for AI foundation models."
It is not available for Gemini. Google-Extended is the one token Google offers for Gemini, and it covers training and grounding together, so a site that blocks it has refused both.
Here is the share of the 2,207 reachable sites that block each crawler site-wide, whether by naming it or through a rule for all crawlers:
Show the data
| Crawler | Share of sites |
|---|---|
| Training and dataset crawlers: Bytespider | 7.1% |
| Training and dataset crawlers: CCBot | 7.0% |
| Training and dataset crawlers: GPTBot | 6.0% |
| Training and dataset crawlers: Google-Extended | 5.9% |
| Training and dataset crawlers: Applebot-Extended | 5.6% |
| Training and dataset crawlers: meta-externalagent | 5.6% |
| Training and dataset crawlers: ClaudeBot | 5.5% |
| Answer-engine crawlers: OAI-SearchBot | 0.2% |
| Answer-engine crawlers: Claude-SearchBot | 0.2% |
| Answer-engine crawlers: PerplexityBot | 0.2% |
| User-triggered fetchers: ChatGPT-User | 0.3% |
| User-triggered fetchers: Perplexity-User | 0.2% |
| User-triggered fetchers: Claude-User | 0.1% |
| Classic search, for reference: Googlebot | 0.1% |
| Classic search, for reference: Bingbot | 0.1% |
Google-Extended also controls Gemini grounding, not only training.
Read the user-fetcher figures with care. OpenAI says of ChatGPT-User that "robots.txt rules may not apply", and Perplexity says Perplexity-User "generally ignores robots.txt rules", because a person asked for the page. A block there is a weak instruction.
Allow Lists Still Name Crawlers Anthropic No Longer Documents
222 of the 1,822 robots.txt files (12.2%) name anthropic-ai or Claude-Web. Anthropic's crawler page (dated April 7, 2026) documents three agents: ClaudeBot, Claude-User, and Claude-SearchBot. Neither older name is on it.
Most are not block lists. Only 18 of the 222 files block those names.
Most of the others name them in a group with an explicit Allow: /. These are sites trying to let Claude in, using names Anthropic no longer documents. None of these names comes from Cloudflare's section.
What This Means for Your B2B Site
These figures describe one group of companies, but the checks apply to any site. Start with these six:
Check whether a CDN setting speaks for you. Open /robots.txt on your own domain and search for BEGIN Cloudflare Managed content. If it is there, the list of blocked crawlers is Cloudflare's, and it changes when Cloudflare changes it.
Decide per crawler, not per category. Training, search, and user-triggered fetching are separate jobs, and most operators give each its own token. Our AI crawlers guide lists the token for each one.
Know what Google-Extended covers. Blocking it refuses Gemini training and grounding in Gemini Apps together. It is not the Google Search token; Googlebot is.
Make robots.txt and llms.txt agree. If you publish an llms.txt while blocking the crawlers of the same companies, decide which of the two you mean.
Use the names operators document today. If your file names anthropic-ai or Claude-Web, check that it also names ClaudeBot, Claude-User, and Claude-SearchBot the way you intend.
Look past robots.txt. Your CDN or firewall can block a crawler that your robots.txt allows, and those rules never show up in the file.
Methodology
This section sets out the sample, the definitions, and the rules behind every figure above.
Sample and source
The sample is every company in Y Combinator's public directory that is marked active, carries the industry B2B, and lists a website. We read the directory on September 10, 2026, through the community-maintained mirror at yc-oss.github.io. We kept one record per unique host, with www. removed: 2,243 sites.
Collection
We made one capture pass, on September 11, 2026. For each site we requested /robots.txt over HTTPS, following redirects, and retried on the www. host if the first request failed at the network level. When robots.txt answered, we requested /llms.txt once from the same origin. Every request carried a user agent that names RankZeroResearch and links to rankzero.io.
Denominators
Every figure on this page names its denominator the first time it appears. These are the counts behind them:
- 2,243 sites in the sample
- 36 unreachable: a network error, or a server error (status 500 or above) on robots.txt
- 2,207 reachable, the denominator for every site-level figure
- 1,822 of the reachable sites served a robots.txt file (HTTP 200 and not an HTML page), the denominator for file-level figures
- 385 reachable sites had no file (any other response, or an HTML page in its place), which we count as allowing every crawler
What "blocked" means
A crawler is blocked when, under the robots.txt standard (RFC 9309), it may not fetch the root path /. We read the group that names the crawler or, if no group names it, the group for all crawlers (User-agent: *). Groups naming the same crawler are merged, the longest matching rule wins, and Allow wins a tie. Rules for specific paths, such as a disallowed /checkout, do not count.
"Blocked by name" means the deciding group names the crawler. A rule for all crawlers can block too, but it is not an AI-specific decision, so the headline figures count blocks by name only. Counting those rules as well, 170 of the 2,207 sites (7.7%) block at least one of the 13 crawlers, and 4 sites block every crawler they do not name individually.
The crawler list
We fixed the list on September 11, 2026, and checked every name on its operator's own page that day. The purposes below are the operators' own words.
Training and dataset crawlers:
- GPTBot (OpenAI): crawls content "that may be used in training our generative AI foundation models"
- ClaudeBot (Anthropic): collects web content "that could potentially contribute to their training"
- Google-Extended (Google): a control token for Gemini training and grounding, quoted in full above
- Applebot-Extended (Apple): lets publishers "opt-out from having their content used to train generative foundation models"
- CCBot (Common Crawl): gathers an open web crawl, whose stated aim is that "free access to web crawl data encourages collaboration and interdisciplinary research"; the page does not mention training
- meta-externalagent (Meta): crawls "for use cases such as training foundation AI models or improving products by indexing content directly"
- Bytespider (ByteDance): ByteDance publishes no page for it, and the third-party bot directories Known Agents and Dark Visitors list it as a training crawler
Meta's and Common Crawl's pages describe more than training, and ByteDance publishes nothing, which is why we call this group "training and dataset crawlers".
Answer-engine crawlers:
- OAI-SearchBot (OpenAI): "used to surface websites in search results in ChatGPT's search features"
- Claude-SearchBot (Anthropic): "navigates the web to improve search result quality for users"
- PerplexityBot (Perplexity): "designed to surface and link websites in search results on Perplexity"
User-triggered fetchers:
- ChatGPT-User (OpenAI): acts on user requests, and "robots.txt rules may not apply"
- Claude-User (Anthropic): "When individuals ask questions to Claude, it may access websites using a Claude-User agent"
- Perplexity-User (Perplexity): fetches on a user's request and "generally ignores robots.txt rules"
We also measured Googlebot and Bingbot, the main crawlers for Google and Bing search, as a reference, and the two names Anthropic no longer documents, anthropic-ai and Claude-Web. None of them is among the 13.
How we attributed authorship
For each blocking file, we looked for text that a platform or tool writes, and applied these rules in this order:
- Cloudflare: the file carries the section that opens with
# BEGIN Cloudflare Managed content. - Another platform or plugin setting: the file carries a marker that a website builder, a CMS plugin, or an ad-network plugin writes. We read each marker from the tool's source code or a live file on September 11, 2026.
- Long list: 15 or more blocked agents in named groups, with no marker.
- SEO plugin wrapper: the rules sit inside an SEO plugin's editor markers. The plugin was used, but the owner typed the rules, so we do not credit the plugin.
- Stated policy: the file's comments state a policy, for example by mentioning "AI training" or "answer engine".
- Hand-written: everything else.
We also checked for agent names that only a widely shared community block list carries, and no site matched. Cloudflare's marker is exact. The other markers were read from source, but only 1 blocking site in this sample carries one, so they are barely tested here. A Content-Signal line on its own cannot identify its author, because several platforms write identical lines, so we never attribute a file on that line alone.
What counts as a valid llms.txt
A valid llms.txt means /llms.txt on the same origin answered HTTP 200 with a text/plain or text/markdown content type, was not an HTML page, and opens with a Markdown heading (# Title), as llmstxt.org specifies. We did not judge quality: some valid files hold little more than a title.
Minimum group size
We give no percentage for any group smaller than 10 sites.
What This Study Cannot See
Every method has blind spots. These are ours:
- robots.txt is a request, not enforcement. Blocks set at a CDN, firewall, or server never appear in the file, so every blocking figure here is a floor. Cloudflare says the same about its own lines: items in robots.txt "signal a website owner's preference, rather than issuing blocks directly."
- Cloudflare's September 15, 2026 defaults will not show up here. In its July 1, 2026 post, Cloudflare wrote: "For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads." Site owners opt out in their Security settings. Those are security settings enforced by Cloudflare, not robots.txt text, so our next fetch may show little change. If so, the change will be real but invisible to a robots.txt study like this one.
- The Cloudflare share depends on the crawler list. We count 13 named crawlers. Counting more bots as AI, such as Amazonbot, which Cloudflare's list blocks but our 13 do not include, changes the denominator.
- We cannot see intent. We can see that a site switched Cloudflare's setting on. We cannot see whether the owner knew which crawlers it blocks.
- This is one population. Y Combinator's active B2B companies lean AI-forward. These figures describe them, not B2B software in general or the web. This edition has no ecommerce comparison.
- We read each file once. Page-level meta tags and HTTP headers are out of scope, and files change.
Conflict of Interest
RankZero sells done-for-you AI SEO services and runs a free AI SEO audit tool, which checks AI crawler access among other things. We have a commercial interest in site owners paying attention to this topic.
To keep that interest out of the numbers, every figure comes from a fresh public fetch, not from our audit data. We make no claim that any robots.txt or llms.txt setting improves citations or rankings.
Data, Code, and the Next Edition
The aggregate data and the code behind every figure (the fetcher, the robots.txt parser, and the analysis script) are available on request: write to [email protected]. The format for a public release is still being decided.
The second edition is due in late September 2026. We will fetch the same 2,243 sites again after Cloudflare's September 15 change and report what moved.
Check Your Own AI Crawler Access
The quickest check is to open your own /robots.txt and look for the Cloudflare marker.
For a fuller read, run our free AI SEO audit on your domain. It checks your robots.txt and whether AI crawlers can reach your pages, with no signup. If you would rather go through the results with us, book an AI search audit call.