How this works
Meta's own documentation lists what each one does:
- meta-webindexer: improves Meta AI search quality. Allowing it helps Meta AI cite and link to your content. This is the one that decides whether you appear in answers.
- meta-externalagent: crawls for AI model training and for indexing to improve products.
- meta-externalfetcher: fetches a single page when a user asks for it. Meta notes it may bypass robots.txt, because a person requested it.
- facebookexternalhit: fetches shared links to build the title, description and thumbnail in a link preview.
- meta-externalads: supports advertising products.
Why it matters
Two traps here, both common.
The first: blocking meta-externalagent to stay out of AI training, and assuming that's the one that controls Meta AI answers. It isn't. meta-webindexer is. You can block training and still be cited.
The second is worse. Blocking facebookexternalhit breaks link previews. Every time someone shares your site on Facebook, Instagram or Messenger, the post appears as a bare link with no image and no description. Clicks drop and nobody connects it to a robots.txt line added months ago.
How to fix
1. Check your robots.txt. Open yoursite.com/robots.txt in your browser and look for groups naming any of the crawlers above:
User-agent: meta-webindexer
Disallow: /Delete the lines for the ones you want to allow. Nothing needs to go in their place. Meta says changes can take up to 24 hours, because crawlers cache the file.
2. Check bot protection on your CDN or hosting. If robots.txt is clean and Meta still can't get in, something is refusing the crawler at server level. On Cloudflare that's usually the Block AI Bots setting or a firewall rule.
Where to change this on your platform
WordPress
robots.txt is normally managed by your SEO plugin: editing robots.txt in Yoast SEO.
Shopify
the file comes from a theme template: editing robots.txt.liquid.
Wix
there's a built-in editor: editing your site's robots.txt file.
Squarespace
you can't edit the file by hand. One checkbox writes it for you: Request that AI models exclude your site, under Settings → Crawlers. It blocks Meta-ExternalAgent and FacebookBot, but not meta-webindexer, so Meta AI can still cite you.
If you can't find the setting
If robots.txt looks right and your security settings are off, ask your hosting provider whether they filter crawler traffic. That's the one place a site owner can't see from the outside.
