Short answer: OpenAI documents separate controls for OAI-SearchBot, which supports ChatGPT search results, and GPTBot, which may collect content for model training. A site owner can allow the former while disallowing the latter in robots.txt. That controls one platform's documented crawler behavior; it does not guarantee a mention, citation, or visit. OpenAI's crawler documentation sets out the roles and their independence.
Decide which public pages you actually want discoverable
A service page, product explanation, and useful article may be intended for public discovery. A client portal, quote, draft, or private document should be protected by access control. Do not use a robots rule as the only safeguard for sensitive material: robots.txt is a crawler instruction, not an authentication system. Before editing it, make a list of genuinely public paths and any areas that must require login or be removed from public hosting.
For Google AI Overviews and AI Mode, a separate OpenAI crawler rule is irrelevant. Google's AI search guidance says its AI features use the Search index; a page must be crawlable, indexed, and eligible for a snippet to be considered. Google also states that eligibility does not guarantee indexing or serving. Keep platform-specific controls separate so a change for one bot does not accidentally block another.
Check the full access path
Read the deployed robots.txt, not just the copy in a repository. Check whether the web application firewall, CDN, or bot-protection service blocks the crawler's requests even when the text file allows them. OpenAI recommends allowing OAI-SearchBot and its published IP ranges if you want the site eligible for ChatGPT search results, and notes that a robots change can take about 24 hours to affect its systems. Its crawler page publishes the current agent description and IP-range link; use that maintained source rather than copying a permanent IP list into an article.
ChatGPT-User is a separate agent used for some user-initiated visits, and OpenAI says it is not the agent that determines Search inclusion. A successful user-triggered fetch therefore does not prove OAI-SearchBot can crawl the site. OpenAI documents the distinction. Equally, a request in server logs does not prove the page was cited; it only shows access.
Measure the result with restraint
After the rule change, verify that intended pages return a successful response and that protected paths remain protected. Monitor crawler logs and available first-party search reports. A ChatGPT search answer may cite the site, omit it, or use another source; no public setting forces the system to use a page. If a product fact is wrong on the public page, granting crawl access only makes that wrong fact easier to retrieve. Fix the content first.
ARCHV can include crawler access, public-page structure, and content governance in a website brief. Our website service page makes technical scope and ongoing responsibility explicit. Use a website estimate to name the platform, CDN, public sections, and access policy that need review; the crawler decision belongs to the site owner.