AI crawler
Definition
A bot operated by an AI company (OpenAI, Anthropic, Google, Perplexity, and others) that fetches web content specifically to train models or answer live user queries.
Defined in The Agentic Commerce Glossary.
How else it gets framed
The same term carries different emphasis depending on the surface it’s used on. Both readings are correct.
A bot operated by an AI company that fetches web pages to feed model training (GPTBot, CCBot), a live answer index (OAI-SearchBot, PerplexityBot), or both. Distinct from user-triggered fetchers like ChatGPT-User, which retrieve a specific page on demand when an assistant needs it mid-conversation.
Should You Block AI Crawlers? A Merchant Decision Guide →Where AI crawler comes up
The guides that put this term to work, rather than just defining it.
Want to know where you actually stand?
Submit your store and get a /100 agentic-commerce readiness score across catalog, schema, product pages and checkout — plus three fixes you can ship yourself.