Discoverability · Check robots-txt-ai-bots
robots.txt with the studio AI crawler allowlist
A robots.txt file that explicitly allows the modern AI crawlers the studio considers canonical: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, and Google-Extended. Each gets its own User-Agent block with `Allow: /`.
Why it matters
The case for ai bot allowlist
ChatGPT, Claude, and Perplexity honor robots.txt strictly. If your robots.txt doesn't list these user agents OR lists them with `Disallow:`, the assistant may skip your site when answering homeowner queries about your services. Many default Wix/Squarespace/GoDaddy templates either omit robots.txt entirely or only have `User-Agent: *` which doesn't explicitly cover the modern AI bots.
How this audit checks it
What we actually look at
We GET `https://yoursite.com/robots.txt`, parse the User-Agent blocks, and count how many studio-standard bots are explicitly allowed. PASS requires the full current studio allowlist, WARN for partial coverage, FAIL for none.
Copy-pasteable fix
Snippet to drop in
User-Agent: *
Allow: /
User-Agent: GPTBot
Allow: /
User-Agent: OAI-SearchBot
Allow: /
User-Agent: ChatGPT-User
Allow: /
User-Agent: ClaudeBot
Allow: /
User-Agent: Claude-Web
Allow: /
User-Agent: PerplexityBot
Allow: /
User-Agent: Google-Extended
Allow: /
Sitemap: https://yoursite.com/sitemap.xmlSteps
How to apply it
- Save the snippet as robots.txt at your site root
- Replace the sitemap URL with your actual domain
- Visit https://yoursite.com/robots.txt and confirm it loads
- Keep each crawler in its own User-Agent block so site builders and crawlers parse the intent cleanly
Common mistakes
What goes wrong
- Disallowing the AI bots (`Disallow: /` under their User-Agent block) - this blocks them rather than allowing
- Trusting the platform default - Wix and Squarespace defaults frequently lack explicit AI-bot blocks
- Mixing case incorrectly - `User-Agent: gptbot` works but the canonical form is `GPTBot`
Other checks in the baseline
Continue your audit walk-through.
Next step
See how your site scores.
Run the free 13-point audit on your URL — takes ~10 seconds, shows exactly which checks pass and fail with copy-pasteable fixes.