How to Let Claude Read Your Site

• Claude-SearchBot: builds the search index Claude uses. Anthropic says blocking it stops your content being indexed for search, which reduces your visibility in results.
• ClaudeBot: collects content that may be used to train Anthropic's models. Separate decision, no effect on whether you appear in answers.
• Claude-User: visits a page when a Claude user asks a question that needs it. Because a person asked for it, robots.txt rules may not hold it back either way, so it isn't a reliable control.

For visibility, the one that matters is Claude-SearchBot. ClaudeBot is the training one.

Blocking ClaudeBot is a reasonable choice if you don't want your writing in training data, and it costs you nothing in visibility. The problem starts when all three get blocked together by one switch, which is what most "block AI bots" settings do. Then Claude has nothing of yours to read, and answers about your industry come from your competitors.

1. Check your robots.txt. Open yoursite.com/robots.txt in your browser. Look for groups like this:

Delete the lines for the crawlers you want to allow. Nothing needs to go in their place. With no rule against it, a crawler is allowed. If you want to stay out of training but remain visible, keep the ClaudeBot block and remove the Claude-SearchBot one.

2. Check bot protection on your CDN or hosting. If robots.txt is clean and Claude still can't get in, something is refusing the crawler at server level. On Cloudflare that's usually the Block AI Bots setting or a firewall rule.

Note that rules must be repeated for each subdomain you want covered.

If robots.txt looks right and your security settings are off, ask your hosting provider whether they filter crawler traffic. That's the one place a site owner can't see from the outside.