AI crawler access
17/25
AI crawlers get through in principle, but your access rules are not a deliberate choice.
robots.txt
robots.txt found (6 lines). Rules are being evaluated.GPTBot
GPTBot is not mentioned in your robots.txt — access is ungoverned. It is allowed, but only by default rather than by decision.ClaudeBot
ClaudeBot is not mentioned in your robots.txt — access is ungoverned. It is allowed, but only by default rather than by decision.PerplexityBot
PerplexityBot is not mentioned in your robots.txt — access is ungoverned. It is allowed, but only by default rather than by decision.Google-Extended
Google-Extended is not mentioned in your robots.txt — access is ungoverned. It is allowed, but only by default rather than by decision.Live test GPTBot
GPTBot gets the same response as a browser (status 200).Live test ClaudeBot
ClaudeBot gets the same response as a browser (status 200).Live test PerplexityBot
PerplexityBot gets the same response as a browser (status 200).Google-Extended
Google-Extended is only a robots.txt token, not a crawler of its own, so there is no live test for it. It controls whether Gemini may use your content.
Next step
Name GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended explicitly in your robots.txt with "Allow: /" — and check your CDN or firewall so no bot rule intercepts them first.