Free tool — robots.txt generator
Decide which AI crawlers may use your site.
Toggle 16 AI bots — training, search, assistant — and get a correct robots.txt snippet to merge into your site. Deterministic, built in your browser, no signup.
16 AI bots · never touches your Googlebot rules · really free
Training
Crawl your content to train future models. Blocking = licensing stance.
GPTBotOpenAIClaudeBotAnthropicanthropic-aiAnthropicGoogle-ExtendedGoogleApplebot-ExtendedAppleMeta-ExternalAgentMetaCCBotCommon CrawlBytespiderByteDance
AI Search
Index you for AI answers and citations. Blocking = invisible in AI search.
OAI-SearchBotOpenAIClaude-SearchBotAnthropicPerplexityBotPerplexityAmazonbotAmazon
Assistant
Fetch pages live when a user asks. Blocking = the assistant cannot read you.
ChatGPT-UserOpenAIClaude-UserAnthropicClaude-WebAnthropicPerplexity-UserPerplexity
0/16 blocked
# AI crawler rules — merge into your robots.txt
# generated with rankrobot.ai/tools/robots-generator
# Nothing to add: all AI crawlers inherit your default rules.
# Toggle a bot to "block" (or enable explicit allow groups) to generate rules.
Merge into yoursite.com/robots.txt — then verify with the AI Crawler Check.
Access is only step one.
Letting the right crawlers in doesn't mean AI recommends you. Run the free visibility scan and see who AI actually names for your niche — you, or your competitors.
[ Run the free AI visibility scan ]A snippet that can't break your SEO.
Most robots.txt generators rewrite your whole file — and one wrong User-agent: * line later, Google stops crawling you. This generator deliberately emits only AI-bot groups: blocked bots share one group with Disallow: /, and your existing rules for classic crawlers stay untouched. Per RFC 9309, a crawler follows the most specific matching group — so a group naming GPTBot affects GPTBot and nothing else.
Example: block training, keep AI search
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /OAI-SearchBot and PerplexityBot are not named — they keep crawling under your default rules, so those assistants can still cite you.
The trade-off nobody mentions
Blocking training bots keeps your content out of future models — but those same models answer your customers' questions. Sites that block everything are opting out of the channel where buying decisions increasingly happen. Decide per category, not all-or-nothing: that's why the toggles above are grouped by what each bot actually does.
Questions.
What does this generator produce?
A robots.txt snippet covering 16 AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more. Bots you block get a shared group with Disallow: /. You merge the snippet into your existing robots.txt; it never touches your rules for classic crawlers like Googlebot.
Which AI bots should I block?
Most sites that care about AI visibility allow search and assistant bots (OAI-SearchBot, PerplexityBot, ChatGPT-User — they are how ChatGPT and Perplexity cite you) and decide consciously about training bots (GPTBot, CCBot, Bytespider). Use the "block training only" preset for exactly that split.
Does blocking AI bots hurt my Google rankings?
No. These tokens are separate from Googlebot. Google-Extended exists precisely so you can opt out of Gemini training without affecting Google Search. Blocking any of the 16 bots here has no effect on classic rankings.
What does "explicit allow groups" do?
By default, allowed bots are simply not mentioned — they inherit the rules of your existing User-agent: * group. With explicit allow groups, allowed bots get their own Allow: / group. Per RFC 9309 a specific group replaces the * rules entirely for those bots — a clear AEO statement, but be careful if your * group protects private paths.
Where do I put the snippet?
Append it to the robots.txt at the root of your domain (yoursite.com/robots.txt). If you have no robots.txt yet, the snippet alone is a valid file. Then verify the result with our AI Crawler Check.