AI Crawlers, robots.txt and llms.txt: Controlling How AI Reads Your Site in 2026

Every day, bots from OpenAI, Google, Anthropic, Perplexity and others visit websites to read pages for AI training, AI search and live answers. Understanding AI crawlers and robots.txt lets you decide which of these bots can read your site, which parts they see, and what you give up by blocking them. This guide covers the practical controls in 2026, including the much-discussed llms.txt file.

Quick Answer

AI crawlers are bots that read web pages for AI model training, AI search indexes or live answers. You control most of them through robots.txt rules that name each bot's user-agent, such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. llms.txt is a proposed, optional convention for guiding AI tools to your key content, not an official standard.

What Are AI Crawlers, and Why Are They Different?

An AI crawler is an automated program that fetches web pages so an AI company can use the content. AI crawlers do three different jobs, and the job matters more than the company.

• Training crawlers collect public text to train or improve future AI models. Blocking them affects what future models may learn, not what an assistant can find today.
• AI search crawlers build an index that an assistant searches when it answers questions with citations. Blocking these can remove your pages from those cited answers.
• User-triggered fetchers visit a page because a person asked the assistant to open or read it. They behave more like a browser acting for a user than like a crawler.

If you are new to how answer engines choose sources, read our guide to generative engine optimization first.

Which AI Crawler User-Agents Should You Know in 2026?

Names change, so confirm against each company's official documentation before editing live rules.

OpenAI

• GPTBot: collects content that may be used to train OpenAI models.
• OAI-SearchBot: used for surfacing websites in ChatGPT search results.
• ChatGPT-User: fetches pages when a ChatGPT user's request needs them.

Google

• Googlebot: the main search crawler. Google Search features, including AI Overviews, rely on Googlebot's index.
• Google-Extended: not a separate bot but a robots.txt product token. Disallowing it tells Google not to use your content for training and grounding its Gemini models; Google states it does not affect inclusion or ranking in Google Search.

Anthropic, Perplexity and others

• ClaudeBot (Anthropic) collects training data, with separate agents documented for search and user-initiated fetching.
• PerplexityBot builds Perplexity's search index, while Perplexity-User handles user-requested fetches.
• CCBot (Common Crawl) builds a public archive widely used as AI training data.

How Do You Control AI Crawlers With robots.txt?

robots.txt is a plain-text file at the root of your domain (for example, yoursite.com/robots.txt) that tells crawlers which paths they may or may not request. Each group starts with a User-agent line followed by Allow or Disallow rules.

Step-by-step setup

1. Open your current file by visiting /robots.txt on your domain.
2. Decide your policy per job, not per company. Write down whether you want training, AI search and user fetches allowed.
3. Add named groups. For example, a group with User-agent: GPTBot and Disallow: / blocks OpenAI training, while a separate group for OAI-SearchBot with Allow: / keeps you eligible for ChatGPT search.
4. Protect private paths for everyone. Disallow areas such as /cart, /checkout, /my-account or internal search results under User-agent: *.
5. Test it. Use the robots.txt report in Google Search Console to confirm Google can read the file, and recheck the syntax after every change.

Mistakes that quietly hurt visibility

• A leftover Disallow: / under User-agent: * from a staging site, which blocks every well-behaved bot, including Googlebot.
• Assuming blocking Google-Extended removes you from AI Overviews. It does not, because AI Overviews are part of Google Search.

What robots.txt Cannot Do

robots.txt is a request, not a lock. Reputable companies follow it, but a rogue scraper can ignore it, and it cannot remove content from an already-trained model.

For content that must stay private, such as student records or client price lists, use logins or server-level blocking. Some hosts and CDNs also offer bot management that blocks crawlers by verified IP ranges.

For Google specifically, snippet controls matter more than robots.txt. The nosnippet and max-snippet meta directives, and the data-nosnippet attribute on a section of HTML, limit what Google can show from your page, and Google applies them to AI features in Search as well.

What Is llms.txt, and Should You Add One?

llms.txt is a proposed convention, suggested in 2024 by Jeremy Howard of Answer.AI, for placing a Markdown file at /llms.txt that gives large language models a clean summary of a site and links to its most useful pages. It is not an official web standard, and it is not a replacement for robots.txt, because it grants or blocks nothing.

The proposed format is simple:

• An H1 with the site name and a short blockquote summary.
• H2 sections listing important links, each with a one-line note.
• Optionally, a longer companion file, often called llms-full.txt, with full page text.

Adoption has grown among documentation and SaaS sites, and some SEO plugins generate the file. However, there is no public confirmation from Google, OpenAI or other major providers that llms.txt is used as a ranking or citation signal in their search products. Treat it as a low-cost experiment, not a strategy.

A sensible approach for small businesses

1. Fix crawlability, indexing and page clarity first. These clearly matter.
2. If you add llms.txt, keep it short and accurate: services, locations, key guides and contact page.
3. Check server logs to see whether any bot actually requests it.

Should a Local Business Block AI Crawlers?

For most local businesses, stay visible. A sweet shop in Model Town or a sports goods exporter in Basti Nau wants AI assistants to describe it correctly when someone asks for recommendations. Blocking AI search crawlers removes you from that answer while competitors stay in it.

A balanced policy:

• Allow search crawlers and AI search agents (Googlebot, Bingbot, OAI-SearchBot, PerplexityBot) so you can be found and cited.
• Decide case by case on training crawlers. Publishers of paid or highly original content may block them; most service businesses gain little.
• Protect private and thin areas with Disallow rules and logins.

Document your choice so the next developer does not undo it, and extend the same review into a full SEO audit.

Key Takeaways

• AI crawlers do three different jobs: training, AI search and user-triggered fetching. Decide your policy per job.
• robots.txt controls well-behaved bots by user-agent, but it is voluntary and is not a security tool.
• Blocking Google-Extended does not remove your site from Google Search or AI Overviews.
• llms.txt is a proposed, optional convention with no confirmed ranking effect; treat it as a small experiment.
• Most local businesses benefit from allowing AI search crawlers so assistants can recommend them accurately.

Learn Technical and AI Search SEO at techcadd Jalandhar

Editing robots.txt, reading server logs and planning AI crawler policy are practical skills covered alongside technical SEO, AEO and GEO in the digital marketing course in Jalandhar at techcadd. The 3-month track builds SEO foundations, the 6-month track adds analytics, AEO and GEO, and the 9-month track goes further with automation and capstone projects. You can also explore the dedicated SEO course in Jalandhar. Fees vary by track and modules, so confirm details with a counsellor and book a free demo class.

Frequently Asked Questions

Does blocking GPTBot remove my website from ChatGPT?

Not necessarily. OpenAI documents GPTBot for training, while OAI-SearchBot is used for ChatGPT search results. Blocking GPTBot alone signals you do not want content used for training, but to remain eligible for cited ChatGPT search answers you should keep OAI-SearchBot allowed in robots.txt.

Will blocking Google-Extended stop my pages appearing in AI Overviews?

No. Google-Extended controls whether content is used to train and ground Gemini models, and Google states it does not affect Google Search. AI Overviews are a Search feature built on Googlebot's index, so snippet controls like nosnippet are the relevant tools there.

Is llms.txt an official standard?

No. llms.txt is a proposed convention introduced in 2024 that suggests a Markdown file summarising a site for language models. It has growing but uneven adoption, and major AI search providers have not publicly confirmed using it as a ranking or citation signal, so treat it as optional.

Can robots.txt protect private content from AI tools?

No. robots.txt is a public, voluntary request that reputable bots follow and bad actors can ignore. Anyone can also read the file, so listing secret paths can reveal them. Protect private information with logins, passwords or server-level access rules instead.

Where do I edit robots.txt on WordPress or Shopify?

On WordPress, SEO plugins such as Rank Math and Yoast include a robots.txt editor, or you can upload a file to the site root. On Shopify, you customise the robots.txt.liquid template in the theme code editor. Always test changes in Google Search Console afterwards.

Should a small business in Jalandhar block AI crawlers?

Usually not the search-related ones. Local shops, clinics and institutes in Jalandhar want AI assistants to find and describe them accurately. Allow AI search agents, protect private sections, and make a separate, considered decision about training crawlers based on how valuable and original your content is.