Free tool — AI crawler check
Is your robots.txt blocking the AI crawlers that matter?
One misplaced Disallow and ChatGPT, Perplexity and Claude stop citing you. Check 16 AI bots against your robots.txt — free, instant, no signup.
16 AI bots · robots.txt only, nothing else fetched · really free
Three kinds of AI crawlers — three different decisions.
Training
GPTBot · ClaudeBot · CCBot
Crawl your content to train future models. Blocking them is a legitimate licensing stance — it does not remove you from today's AI answers.
AI Search
OAI-SearchBot · PerplexityBot
Index your site so answer engines can cite and recommend you. Blocking these is the AEO equivalent of de-indexing yourself from Google.
Assistant
ChatGPT-User · Perplexity-User
Fetch a page live when a user asks about it. Blocking them means the assistant can't read your pricing or docs mid-conversation.
The AEO-friendly default
Allow search + assistant bots, decide consciously about training bots. To unblock an engine, remove its Disallow line — or add an explicit group:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Questions.
What does this tool check?
It fetches your robots.txt and evaluates it against the user-agent tokens of the AI crawlers that matter — OpenAI, Anthropic, Perplexity, Google-Extended, Common Crawl and more. For each bot you see whether your site allows it, blocks it completely, or blocks only parts of it.
Should I block AI crawlers?
It depends on the crawler type. Blocking training bots (GPTBot, CCBot) keeps your content out of future model training — a legitimate choice. Blocking search and assistant bots (OAI-SearchBot, PerplexityBot, ChatGPT-User) makes you invisible in AI answers: those bots are how ChatGPT and Perplexity cite and recommend you.
Does blocking GPTBot affect my Google rankings?
No. GPTBot, Google-Extended and the other AI tokens are separate from Googlebot. Blocking them has no effect on classic Google Search rankings — Google-Extended exists precisely so you can opt out of Gemini training without touching Search.
My site has no robots.txt — is that bad?
No robots.txt means every crawler is allowed by default. That is the most AI-visible configuration, though you give up control. Adding a robots.txt lets you allow answer engines while opting out of training crawlers.