Can AI crawlers read your website?
A site that GPTBot, OAI-SearchBot, ClaudeBot, or PerplexityBot cannot fetch cannot be cited, however good its pages are. This check reads your robots.txt for wholesale blocks against each AI user-agent, then fetches your homepage as GPTBot and PerplexityBot to catch CDN and firewall rules that turn bots away at the edge.
FAQ
Questions
- Which AI crawlers does this check?
- GPTBot (ChatGPT training and retrieval), OAI-SearchBot (ChatGPT search results), ClaudeBot (Claude), PerplexityBot (Perplexity), Google-Extended (the robots token for Gemini and Google AI grounding), and Bingbot (the index Microsoft Copilot runs on). The live probe fetches your homepage as GPTBot and PerplexityBot; the others are checked in robots.txt.
- Why would my site block AI crawlers without my knowing?
- Two common ways. Some CDNs and security plugins block AI bots by default, so the rule was never something you chose. And some SEO plugins or hosting templates ship a robots.txt that disallows GPTBot wholesale, added in 2023 when blocking training crawlers was the fashion. Either way, the engine that could have cited you cannot read you.
- Should I block AI crawlers?
- If you want to be recommended when customers ask AI who to hire, no. Blocking GPTBot keeps your content out of ChatGPT's training and, more importantly for a local business, out of the retrieval that produces its live answers. Yext found 86% of AI citations came from brand-managed sources; you cannot be one of them if the crawler is turned away.
- What does this tool not check?
- It does not check whether your content is good enough to be cited, whether your listings are complete, or how often AI actually names you. The free AI visibility report at /ai-visibility does the last part by asking ChatGPT live about your industry and city.
