GPTBot

GPTBot is the crawler OpenAI uses to gather public web content for training its models. It announces itself in the user-agent header as GPTBot, and OpenAI publishes the IP ranges it crawls from so you can verify a request is genuine. Like any well-behaved bot, it reads your robots.txt file and follows rules written for its name.

The key thing to understand is what GPTBot is not. According to OpenAI's documentation, GPTBot is the training crawler. A different bot, OAI-SearchBot, is what fetches pages for ChatGPT's search features, and a third, ChatGPT-User, handles requests a user triggers directly in a conversation. Blocking GPTBot tells OpenAI not to use your content for training. It doesn't stop ChatGPT from reading your pages when someone asks it who does foundation repair in Denver.

That distinction got lost in 2023, when GPTBot was announced and thousands of sites added a blanket block. If your site or your hosting provider did that and never updated the rule, you may be blocking the training crawler while leaving search open, which is fine, or blocking every OpenAI bot, which isn't. Worth checking. The robots.txt file is at yourdomain.com/robots.txt and takes a minute to read.

For a small business, allowing GPTBot is a judgment call about whether you mind your public pages contributing to model training. There's no measured visibility benefit to allowing it and no measured penalty to blocking it, because training data and live retrieval are different pipelines. What does matter for being recommended is whether OAI-SearchBot can read you, and whether the facts it finds are clear.

Tibly's AI-readiness audit lists which OpenAI bots your robots.txt allows and blocks, and fetches your homepage as GPTBot to catch blocks that live in a CDN or firewall rather than the file. The point is to make the choice deliberately instead of inheriting a 2023 default. A Tibly-hosted site serves a robots.txt that allows all crawlers.