Guides · Updated 2026-08-26 · Markdown
How to allow GPTBot and other AI crawlers in robots.txt
If you want ChatGPT, Claude or Perplexity to be able to quote your site, the first requirement is boring: they have to be allowed to fetch it.
The short answer
Add this to the robots.txt at the root of your domain:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
That is it. No meta tag, no header, no plugin.
What each token actually controls
They are not interchangeable, and the difference matters more than most guides admit.
| Token | Operator | What it controls |
|---|---|---|
GPTBot |
OpenAI | Crawling for model training |
OAI-SearchBot |
OpenAI | Indexing for ChatGPT search results |
ChatGPT-User |
OpenAI | A live fetch triggered by a user's question |
ClaudeBot |
Anthropic | Crawling for model training |
Claude-User |
Anthropic | A live fetch triggered by a user's question |
PerplexityBot |
Perplexity | Indexing for Perplexity answers |
Google-Extended |
Whether Gemini may use your content. Not a crawler — a permission token only |
The practical consequence: blocking GPTBot keeps you out of training data but does not keep you out of ChatGPT answers, because those use OAI-SearchBot and ChatGPT-User. Many sites block the training crawler and assume they have opted out of everything. They have not. Others block everything and wonder why they never get cited.
Decide per token, not per company.
Not mentioning them is not the same as allowing them
If your robots.txt says nothing about GPTBot, the crawler falls back to your User-agent: * group. If that group is permissive, you are allowed — by accident rather than by decision.
That is a fragile position. The moment somebody adds a broad Disallow for an unrelated reason, you are out, and nobody will notice for months. Naming the crawlers explicitly makes the decision visible in the file, and visible to the next person who edits it.
The part robots.txt cannot fix
This is where most sites actually fail. Your robots.txt can say Allow: / while your infrastructure returns 403 to the same crawler.
Common causes:
- Bot protection at the CDN. Cloudflare's Bot Fight Mode, AWS WAF bot rules and similar features act before your application does. They do not read your robots.txt.
- Managed robots.txt features. Some platforms offer to generate a robots.txt that signals your content may not be used for AI training. If you enable that, it overwrites whatever you wrote.
- Geo or rate-limit rules that treat datacenter IP ranges as suspicious. Every AI crawler is datacenter traffic.
The only way to know is to request your own homepage with the crawler's user agent and compare the status code to a normal browser request:
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
https://example.com/
A 403, 429 or 503 where a browser gets 200 means something in front of your application is blocking, no matter what robots.txt says.
Verify it
Three things have to be true at once: the file exists and is reachable, the crawler is named in it, and the live request actually returns 200.
Checking all three by hand takes a while. Run a free scan — five checks, under 60 seconds, no login.
A product by The Autopilot — https://the-autopilot.com