Can you block AI crawlers, and should you?
Yes, you can block them in robots.txt, but if you want AI to recommend you in 2026, you need to allow the crawlers that fetch content for answers. Whether you also let your content be used to train models is a separate choice.
Crawlers for search and answers
Allow these if you want to be named in AI answers:
- OAI-SearchBot (ChatGPT’s search) and ChatGPT-User (when a user asks ChatGPT to open a page).
- Claude-SearchBot (Claude’s search) and Claude-User (when a user asks Claude to open a page).
- PerplexityBot (Perplexity).
- Googlebot and Bingbot (Google’s AI answers, Gemini and Copilot build on their indexes).
Crawlers for training
- GPTBot (OpenAI), ClaudeBot (Anthropic) and Google-Extended (Gemini) control whether your content may be used to train models.
- Blocking Google-Extended does not affect Google’s AI answers in search. Those are governed by Googlebot.
- For most companies that want to be known and recommended, allowing them is an advantage.
Check your own robots.txt
Open yourdomain.com/robots.txt and look for Disallow lines under the names above. Some hosting companies and security services block AI crawlers by default, so check your firewall settings too.
Want to know where you stand today?
Get a free AI snapshot