Crawlability
Crawlability is how easily automated bots from search engines and AI companies can discover, fetch, and read the pages on a website, determined by robots.txt rules, server responses, HTTPS, sitemaps, and whether content is visible without running JavaScript.
Crawlability is the answer to a mechanical question: can a bot get to your pages and read what's on them? Before any engine can rank you, cite you, or recommend you, its crawler has to discover the URL, fetch it without being blocked, receive a successful response, and parse the content. Each of those steps can fail quietly, and when one does, the page effectively doesn't exist for that engine.
The common failure points are unglamorous. A robots.txt rule or CDN setting blocks the crawler. The server returns an error or a redirect loop. The site isn't on HTTPS. Pages aren't linked from anywhere a crawler would find them and there's no sitemap. Or the content is rendered entirely by JavaScript after the page loads, so a crawler that doesn't execute scripts sees an empty shell. Modern search crawlers handle JavaScript reasonably well; many AI crawlers are less patient, and a plain HTML page is the safe bet.
For AI visibility specifically, crawlability is the first gate. Retrieval-augmented engines search an index and read what comes back; a page that was never crawled isn't in the index and can't be retrieved. Ahrefs found 88% of ChatGPT's cited URLs across 1.4 million prompts came from its general web-search retrieval. That retrieval only reaches pages OAI-SearchBot could fetch. Crawlability isn't a ranking factor for AI answers; it's the precondition for having a chance at all.
Small sites have a second problem: discovery speed. A new page on a low-authority domain can sit unindexed for weeks because crawlers visit infrequently. Publishing a pricing page in March doesn't help you get cited in March if nothing fetches it until May. Sitemaps help, internal links help, and the IndexNow protocol, which Bing honors and which feeds Copilot, lets a site push a notification the moment a page changes instead of waiting to be found.
Tibly-hosted sites are built to be crawlable by default: server-rendered HTML, HTTPS, a robots.txt that allows all crawlers, a generated sitemap that can't drift from the live page list, and an IndexNow ping every time a page is published or changed. The AI-readiness audit checks the same properties on sites hosted elsewhere and reports what's blocking the bots.
Keep exploring the glossary
- AI CrawlerAn AI crawler is an automated bot such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, or Google-Extended that fetches web pages so an AI company can train its models or retrieve current information to answer user questions.
- robots.txt for AI Crawlersrobots.txt for AI crawlers means using the robots.txt file at your site's root to allow or block named AI bots such as GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot, deciding which AI systems may read your pages for training or live answers.
- IndexNowIndexNow is an open protocol, created by Microsoft Bing and Yandex, that lets a website notify participating search engines the moment a page is added, updated, or removed, so the change is crawled within hours.
- AI-Readiness AuditAn AI-readiness audit is a check of a website that tests whether AI crawlers can reach it, whether it serves HTTPS, whether facts like pricing, hours, and service area appear in plain text, and whether basic structured data is present.
Common questions about Crawlability
- How do I check if my site is crawlable by AI engines?
- Read your robots.txt for rules blocking GPTBot, OAI-SearchBot, ClaudeBot, or PerplexityBot; fetch your homepage using one of their user-agent strings and confirm you get a real page; confirm HTTPS; and view your page source to make sure the main content is in the HTML, not injected by JavaScript. Tibly's audit does all four.
- Does crawlability affect whether AI recommends me, or just whether it can?
- Just whether it can. Crawlability is a gate, not a score. A crawlable site with vague content still won't get named; an uncrawlable site with perfect content can't be. Clear the gate first, then work on what the engines find when they get through.