AI scrapers hammer WordPress sites, copying content without credit or payment. Blocking every bot sounds tempting, but it also locks out Googlebot and Bingbot. Rankings drop and traffic dries up.

Some crawlers prove who they are. Google and Bing publish IP ranges and support reverse DNS checks. Many AI scrapers fake user agents and rotate residential proxies, so simple user-agent rules don’t work.

Robots.txt looks helpful, but it’s a request, not a rule. Honest crawlers follow it, but most AI scrapers don’t. A blanket disallow in robots.txt wipes pages from indexes and guts organic traffic in days.

A smarter approach uses selective AI access control. Verified search engines and real visitors get through. Unknown bots get slowed or challenged. High-value paths, like premium articles or API endpoints, require programmatic payments with tools like PayLayer before access. Permissioned AI access protects content from freeloaders while keeping SEO intact.

Why blocking all bots hurts SEO and how selective access solves it

WordPress can welcome real search engines and still keep scraper bots from chewing through premium content. The setup below lets Google and other trusted crawlers index the site, slows or blocks unknown bots, and offers a paid path for AI agents that want access to protected data.

Robots.txt Configuration
Robots.txt works like a polite gatekeeper. It gives well-known crawlers room to index the site and nudges unknown agents away from high‑value areas. A practical example:

“`plaintext

Allow trusted search engines full access

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Applebot
Allow: /

Block unknown bots from sensitive paths

disallow: /premium-articles/
disallow: /wp-json/wp/v2/
disallow: /cart/
disallow: /checkout/

AI access policy – directs bots to payment info page

AI access policy: https://yoursite.com/ai-access

“`
This keeps SEO healthy for real engines while steering unknown crawlers away from valuable routes.

Rate Limiting & Firewall Rules
Scrapers hit fast and hard, so rate limits slow them down before they drain resources.

• Regular visitors: about 120 requests per 5 minutes
• Verified Googlebot: roughly 600 requests per 5 minutes
• Unknown agents that spike past limits receive HTTP 429 responses and a temporary block
  Cloudflare or another WAF helps flag non-browser agents, bursty traffic, and HEAD-only probes. Reverse DNS checks and ASN filters keep trusted bots whitelisted so indexing continues without friction.

PayLayer Integration Concept
Premium routes – like /wp-json/wp/v2/posts or /product-category/premium/ – sit behind a paid gate for AI agents.

• First contact returns 402 Payment Required plus instructions for paid access
• After payment, the system issues a signed token with time limits (for example, 24 hours) and usage caps (for example, 1,000 objects)
  Humans browse as usual. Verified search engine crawlers bypass the gate, so indexing stays intact. This converts unapproved scraping into a clear, controlled access model while protecting top-tier content.