What is robots.txt?
robots.txt is a plain text file at the root of your website, for example yourdomain.com/robots.txt. It tells crawlers which parts of your site they may visit. Each group of rules starts with a User-agent line naming the crawler, followed by Allow and Disallow lines.
It's a polite request, not a lock. Reputable crawlers follow it. Badly behaved scrapers ignore it, so anything truly private belongs behind a login.
The main AI crawlers and what they do
Most big AI companies now run separate bots for training their models and for search, meaning fetching pages to answer a user's question with a citation. That split is what makes a sensible policy possible.
- GPTBot (OpenAI): collects content that may be used to train OpenAI's models.
- OAI-SearchBot (OpenAI): finds pages for ChatGPT search results. Block it and you're unlikely to be cited there.
- ChatGPT-User (OpenAI): fetches a page when someone asks ChatGPT to look at it.
- ClaudeBot (Anthropic): collects content that may be used to train Claude models.
- Claude-SearchBot and Claude-User (Anthropic): search indexing and pages fetched at a user's request.
- PerplexityBot and Perplexity-User: Perplexity's search index and user-requested fetches.
- Google-Extended: not a separate crawler but a robots.txt token. It controls whether Google may use your content for Gemini models. It does not affect Google Search or AI Overviews.
- Applebot-Extended: a similar token for Apple's AI training.
- CCBot (Common Crawl): builds an open web archive that many AI models are trained on.
Should I block AI crawlers?
It depends on what you want from AI tools:
- You want AI assistants to recommend you (most businesses): allow the search and user bots. Allowing training bots is your choice, and it may help AI models know your brand.
- You publish original content and don't want it used for training: block the training bots but keep the search bots, so you can still be cited.
- Private or paid content: don't rely on robots.txt at all. Put it behind a login.
Example: allow all AI search, block training
# Search and user-requested fetches: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# Model training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
# Everyone else, including Googlebot
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlExample: welcome every AI crawler
If you want maximum visibility, you don't need AI-specific rules at all. A site with no rules that block them is open to every crawler. The simplest version is:
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlHow to check you're not blocking AI crawlers by accident
Many sites block AI bots without anyone deciding to. Common causes are security plugins, a CDN or hosting setting that blocks AI bots with one click, and old robots.txt rules copied from a template. Check three places:
- Open yourdomain.com/robots.txt and read every User-agent group.
- Look at your CDN or firewall settings for bot-blocking or AI-blocking options.
- Check your server logs for AI bot visits that get a 403 (forbidden) response.
Pagelyt's GEO check reads your robots.txt and flags AI crawlers that are blocked. See how to unblock AI crawlers, or run a free scan to check yours now.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
Blocking GPTBot stops your content being collected for training. ChatGPT search uses a different bot, OAI-SearchBot, so allow that one if you want to be cited in ChatGPT search.
Does blocking Google-Extended affect my Google rankings?
No. Google-Extended only controls whether your content is used for Gemini models. It has no effect on Google Search rankings or on AI Overviews.
Do AI crawlers respect robots.txt?
The major AI companies say their automated crawlers follow robots.txt. User-triggered fetches may be treated differently, and some lesser-known scrapers ignore it, so use firewall rules if you need a hard block.
Where does the robots.txt file go?
At the root of each host, for example yourdomain.com/robots.txt. Subdomains such as shop.yourdomain.com need their own file.
Can robots.txt remove a page from Google?
No. Blocking crawling doesn't remove a page that's already indexed. Use a noindex tag instead, and leave the page crawlable so Google can see the tag.