We Checked How 93 Major Websites Treat AI Crawlers: Who Blocks, Who Publishes llms.txt

This post shares the numbers from our check on 25 October 2026, the limits of the sample and what they mean for your own robots.txt. It also explains the difference between training crawlers and agents that fetch pages for a user, and it ends with a 10-minute checklist. It follows our audit of the language setup of 123 websites, which also looked at 123 well-known brands.

How we ran the check

We took 123 well-known consumer and B2B brands. For each one we made a single plain HTTP GET to /robots.txt, /llms.txt and /llms-full.txt. The requests used a generic browser-like user agent, came from one server location and followed redirects. We ran everything on 25 October 2026.

A crawler counts as fully blocked when its own group in robots.txt has Disallow: / and no Allow: /. When the file does not name the crawler, the wildcard (*) group decides. We looked for 14 AI crawlers.

93 of the 123 sites returned a readable robots.txt. The other 30 refused the request, had no file or returned a page that is not a robots file. Every robots.txt figure below uses the 93 as its base. The llms.txt figures use all 123.

The numbers at a glance

Who blocks which crawler

Blocking is concentrated in a handful of crawlers. CCBot, the Common Crawl crawler, leads with 10 sites. GPTBot and ClaudeBot follow with 9 each, Bytespider and Amazonbot with 8 each, Applebot-Extended and meta-externalagent with 7 each and Google-Extended with 6.

The crawlers that act for a user sit at the bottom of the list. PerplexityBot is fully blocked by 3 sites, ChatGPT-User and OAI-SearchBot by 2 each.

Only 20 of the 93 sites name any of the 14 AI crawlers, so most robots.txt files say nothing about them. 11 sites block at least one of GPTBot, ClaudeBot, CCBot, Google-Extended and anthropic-ai. One file carries a wildcard rule that blocks every crawler, and our definition applies the wildcard group to any crawler a file leaves unnamed.

76 of the 93 sites declare a sitemap in robots.txt. The file already carries sitemap work for most of them, which makes it a natural home for AI rules as well.

Who publishes llms.txt

29 of the 123 sites (24%) publish an llms.txt. We opened all 29, and every one is a real text or markdown file. Only 3 of the 123 also publish an llms-full.txt, the longer variant.

The cross-check gives one clear result: none of the 9 sites that block GPTBot publish an llms.txt. In this sample, the sites that shut out the best-known training crawler and the sites that write a guide for language models are separate groups. We did not look at why.

Limits of this data

Read the figures with these limits in mind.

• One day. Everything reflects 25 October 2026, and these files change.
• One server location. Sites can answer differently from other locations.
• Homepage domains only. We fetched the files from each brand's main domain.
• Large, well-known brands. The sample says little about small sites.
• 30 sites unreadable. They refused the request, had no file or returned a page that is not a robots file. The robots.txt figures cover the other 93.
• robots.txt records intent. It states what a site asks crawlers to do. Enforcement happens elsewhere.
• No test of llms.txt effects. We did not test whether llms.txt changes anything. It is a community proposal and has no standard behind it.

Training crawlers and user-driven agents are different

AI crawlers fall into two groups, and the consequences of blocking differ.

• Training collectors: GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended collect content for model training, or control whether it may be used for that.
• Search and user agents: OAI-SearchBot, ChatGPT-User and PerplexityBot search or fetch pages on behalf of a person asking a question.

Blocking a training crawler does not by itself remove your site from search-style AI answers, because those answers rely on other crawlers. Google-Extended is a control token for Google's AI uses, and Googlebot keeps crawling for Search under its own rules.

So decide per crawler. A company that wants to appear when people ask an assistant for a supplier can allow the search and user agents and still block training collection. A publisher that licenses its archive may block all of them. Both are valid choices, and each follows from your goals.

How robots.txt groups work

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines and continues with Disallow and Allow rules. A crawler follows the group that names it. When no group names it, the crawler follows the wildcard group, User-agent: *. A named group wins over the wildcard group, and the crawler ignores the wildcard rules once it finds its own.

This file blocks two training crawlers and leaves everything else open:

• User-agent: GPTBot
• User-agent: ClaudeBot
• Disallow: /
• User-agent: *
• Allow: /
• Sitemap: https://www.example.com/sitemap.xml

The first two lines share one group, and in a real file a blank line separates the groups. Copy crawler names exactly from each vendor's documentation, because a misspelled name applies to nobody. A lone Disallow: / under User-agent: * shuts out every crawler that respects the file. That is the rule one site in our sample uses.

A minimal llms.txt

llms.txt is a community proposal for a plain markdown file at /llms.txt that points language models to your most useful pages. It is a proposal and no standard, and we did not test whether it changes anything. The skeleton is short:

• # Your Site Name
• > One-line summary of what the site offers.
• ## Section name
• - [Page title](https://www.example.com/page/): a short note on what the page covers

Add a few sections, such as products, guides and contact, and keep each link to a page you want read. Globaprom publishes llms.txt and llms-full.txt in four languages. If you run a multilingual site, list the pages of each language version. Our hreflang guide covers how search engines match those versions, and our page on multilingual software covers building them.

A 10-minute checklist for your site

• Open /robots.txt. Confirm that it loads and reads as a robots file.
• List the AI crawlers you name. Compare your list with the table above. A crawler you do not name follows the wildcard group.
• Decide per crawler. Allow or block each one against your goals: training collection, AI search visibility and user-triggered fetches.
• Check /llms.txt. Either it exists and lists your key pages, or its absence is a deliberate choice.
• Declare your sitemap. Add a Sitemap: line, as 76 of the 93 readable files in our sample do.
• Note the date. Crawler names change, so book a review for a few months from now.

Frequently Asked Questions