AI crawler access
17/25
AI crawlers get through in principle, but your access rules are not a deliberate choice.
robots.txt
robots.txt found (5 lines). Rules are being evaluated.GPTBot
GPTBot is not mentioned in your robots.txt — access is ungoverned. It is allowed, but only by default rather than by decision.ClaudeBot
ClaudeBot is not mentioned in your robots.txt — access is ungoverned. It is allowed, but only by default rather than by decision.PerplexityBot
PerplexityBot is not mentioned in your robots.txt — access is ungoverned. It is allowed, but only by default rather than by decision.Google-Extended
Google-Extended is not mentioned in your robots.txt — access is ungoverned. It is allowed, but only by default rather than by decision.Live-Test GPTBot
GPTBot gets the same response as a browser (status 200).Live-Test ClaudeBot
ClaudeBot gets the same response as a browser (status 200).Live-Test PerplexityBot
PerplexityBot gets the same response as a browser (status 200).Google-Extended
Google-Extended is only a robots.txt token, not a crawler of its own, so there is no live test for it. It controls whether Gemini may use your content.
Next step
Name GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended explicitly in your robots.txt with "Allow: /" — and check your CDN or firewall so no bot rule intercepts them first.