Skip to content
mirok

AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended)

AI crawlers are the bots AI companies use to collect web pages, either to train their models or to feed their answer engines in real time. They identify themselves with a user agent such as GPTBot, ClaudeBot or PerplexityBot, and most of them follow robots.txt.

Why do AI crawlers matter for your website?

These bots partly decide whether your content can be read and cited by AI assistants. Companies often run several bots for different purposes. According to their public documentation, OpenAI uses GPTBot for training-related collection, OAI-SearchBot for ChatGPT search and ChatGPT-User for visits triggered by a user. Anthropic uses ClaudeBot, Perplexity uses PerplexityBot.

Google-Extended is a special case: it is not a separate bot but a robots.txt token that lets you opt out of your content being used for certain Gemini model uses. It does not affect your presence in Google Search, AI Overviews included, which remains handled by Googlebot.

Blocking a training bot and allowing a search bot are therefore two different decisions. Many sites block everything by reflex and disappear from answers where they could have been cited.

How should you manage AI crawlers on your site?

  1. 1Inventory the bots already visiting your site from your server or CDN logs.
  2. 2Decide, bot by bot, what you accept: search and citation, training, or nothing.
  3. 3Express those decisions in robots.txt with one User-agent block per bot and explicit Allow / Disallow rules.
  4. 4Check that your firewall, CDN or bot protection does not silently block the bots you have allowed.
  5. 5Read each vendor's official documentation: bot names and purposes change over time.
  6. 6Review these rules regularly, for example every quarter.

What mistakes should you avoid with AI crawlers?

  • Copying an "anti-AI" robots.txt found online without measuring its impact on your visibility.
  • Treating robots.txt as a security measure: it is an instruction well-behaved bots follow, not a lock.
  • Confusing Google-Extended with Googlebot and blocking Google Search by mistake.
  • Forgetting CDN protection rules, which sometimes return 403 errors to AI crawlers.