AI crawler access
25/25
AI crawlers reach your content, and you decided that on purpose.
robots.txt
robots.txt found (90 lines). Rules are being evaluated.GPTBot
GPTBot is named explicitly in your robots.txt and allowed on the homepage.ClaudeBot
ClaudeBot is named explicitly in your robots.txt and allowed on the homepage.PerplexityBot
PerplexityBot is named explicitly in your robots.txt and allowed on the homepage.Google-Extended
Google-Extended is named explicitly in your robots.txt and allowed on the homepage.Live test GPTBot
GPTBot gets the same response as a browser (status 200).Live test ClaudeBot
ClaudeBot gets the same response as a browser (status 200).Live test PerplexityBot
PerplexityBot gets the same response as a browser (status 200).Google-Extended
Google-Extended is only a robots.txt token, not a crawler of its own, so there is no live test for it. It controls whether Gemini may use your content.
Next step
Name GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended explicitly in your robots.txt with "Allow: /" — and check your CDN or firewall so no bot rule intercepts them first.