The checklist
- Search crawlers allowed. Make sure robots.txt does not block OAI-SearchBot, Bingbot or Googlebot.
- Training crawlers decided. GPTBot (OpenAI) and Google-Extended (Google's AI training control) can be allowed or blocked separately from search.
- Firewall checked. Some hosts and security tools block AI crawlers by default. Ask your host or check the settings.
- Structured data. Add LocalBusiness or Organization, Service and FAQPage markup that matches the visible page.
- Clear answers on page. Lead each service page with a short, direct answer to the main question.
- Sitemap current. Submit it in Google Search Console and Bing Webmaster Tools.
- llms.txt, optional. A plain-text guide to your most important pages. It is a proposal, not an adopted standard, so treat it as a low-cost extra.
Search vs. training
| Crawler | Operator | Purpose |
|---|---|---|
| OAI-SearchBot | OpenAI | Surfacing sites in ChatGPT search results |
| GPTBot | OpenAI | Collecting content that may be used for training |
| ChatGPT-User | OpenAI | Visiting pages when a user's request needs them |
| Google-Extended | A control for use in Google's AI models; not a separate crawler |
Blocking training while allowing search is a legitimate choice. If your content has licensing value, that decision connects to selling data to AI companies.
Search allowed, training your call
robots.txtyourbusiness.com
User-agent: OAI-SearchBotSearch: lets pages appear in ChatGPT
Allow: /
User-agent: GPTBotTraining: allow or block, your call
Disallow: /
User-agent: Google-ExtendedGoogle's AI training control
Disallow: /
Sitemap: https://yourbusiness.com/sitemap.xmlKeep it current
Sources
- Overview of OpenAI crawlers · OpenAI
- The /llms.txt file proposal · llmstxt.org