How this works
It's three or four lines long, almost nobody reads it, and it quietly decides what AI assistants are allowed to see.
The rule that causes trouble looks like this:
Disallow: /services/That line tells every assistant to skip your services pages. It doesn't matter how good those pages are. They won't be opened.
Why it matters
Nobody blocks their own services pages on purpose. These rules get there by accident:
- A developer blocked a section during a rebuild and never removed the rule.
- A plugin or app added rules of its own during setup.
- Someone copied a robots.txt file from another site, along with its paths.
- A rule meant to hide
/adminwas written loosely enough to catch real pages too.
The file is never shown to you in day-to-day work, so a wrong line can sit there for years.
How to fix
1. Read your file. Open yoursite.com/robots.txt in your browser. Anyone can see it, including you. It's short.
2. Go through the Disallow lines one at a time. For each, ask: do I want this part of my site to appear in AI answers? Checkout pages, admin areas and account pages are reasonable to block. Services, prices, locations, about and contact pages are not.
3. Delete the rules that block pages you want found. Nothing goes in their place. With no rule against it, a path is allowed. Changes take effect immediately.
4. Check the whole file while you're there. Two more things live in it. One is a Sitemap: line pointing to your sitemap, and it should be there. The other is rules naming specific AI crawlers, such as User-agent: GPTBot; those decide which assistants get in at all.
Google's documentation on robots.txt explains the syntax if you want to go deeper.
One limit worth knowing: robots.txt only asks crawlers not to visit. It isn't security. Anything genuinely private needs a password, not a Disallow line.
Where to change this on your platform
WordPress
the file is normally managed by your SEO plugin: editing robots.txt in Yoast SEO.
Shopify
generated from a theme template you can edit: editing robots.txt.liquid.
Wix
there's a built-in editor: editing your site's robots.txt file.
Squarespace
you can't edit this file by hand. Squarespace writes it for you, based on your page and crawler settings.
If you can't find where the rules come from
Some hosting panels and security plugins inject rules into robots.txt without showing them in your SEO settings. If a line keeps coming back after you delete it, that's what's happening. Ask your host, or disable security plugins one at a time to find the source.
