AI crawler access
17/25
AI crawlers get through in principle, but your access rules are not a deliberate choice.
robots.txt
Your robots.txt is not cleanly retrievable (status 200). AI crawler access is therefore ungoverned.GPTBot
GPTBot is not mentioned in your robots.txt — access is ungoverned.ClaudeBot
ClaudeBot is not mentioned in your robots.txt — access is ungoverned.PerplexityBot
PerplexityBot is not mentioned in your robots.txt — access is ungoverned.Google-Extended
Google-Extended is not mentioned in your robots.txt — access is ungoverned.Live test GPTBot
GPTBot gets the same response as a browser (status 200).Live test ClaudeBot
ClaudeBot gets the same response as a browser (status 200).Live test PerplexityBot
PerplexityBot gets the same response as a browser (status 200).Google-Extended
Google-Extended is only a robots.txt token, not a crawler of its own, so there is no live test for it. It controls whether Gemini may use your content.
Next step
Name GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended explicitly in your robots.txt with "Allow: /" — and check your CDN or firewall so no bot rule intercepts them first.