How to require Claude to pay before crawling selected WordPress content

Claude AI often pulls content from public web pages to answer questions, which chips away at the value of gated posts. It usually sends an HTTP GET request to the page URL or uses a retrieval tool, then parses the returned HTML or text. If a paywall notice or a 402 Payment Required response comes back instead of the real content, Claude summarizes that message. It doesn’t slip past the wall.

Some teams assume a robots.txt file keeps Claude out, but it doesn’t. Robots.txt suggests crawling rules, it doesn’t block access. Real paywalls act differently. They return specific status codes or challenges until the request shows valid proof of payment in headers or query parameters.

Programmatic paywalls solve this. They detect when an AI or agent tries to fetch content and trigger a payment challenge. With PayLayer as the gatekeeper, the system verifies payment on the spot before returning anything behind those URLs.

WordPress sites don’t need complex setup to do this. Site owners can require payment before Claude gets through to selected posts or pages. The process is straightforward, and a short walkthrough explains how to make Claude pay before it crawls protected WordPress content.

Why Claude can read your pages and how a programmatic paywall fixes it

A Claude-only AI paywall on WordPress with PayLayer keeps things simple. Site owners charge Claude for access without bothering human visitors. PayLayer identifies Claude by checking Anthropic-specific User-Agent patterns against an allowlist, and if that’s unclear, it checks IP or ASN. Normal browsers never get blocked. Only Claude hits the paywall, while other AIs get handled with separate rules.

Here’s the flow:

1. Install and activate the PayLayer.org plugin on the WordPress site.
2. In the PayLayer dashboard, create a product like “AI crawl access,” and set a price per request or per 1,000 words, for example $0.02.
3. In WordPress, mark chosen posts or pages as “AI gated.” When Claude fetches those URLs, the site returns HTTP 402 plus a PayLayer payment challenge. Regular visitors still get HTTP 200 and see the page as usual.

Claude receives a JSON challenge with the price, a resource ID, and a unique nonce for security. After paying through the PayLayer API, Claude retries the page with an authorization token in the headers. The plugin verifies the token on the server, then serves the full HTML.

WordPress logs every paid access with timestamps, URLs, and charges. These records support audits and help adjust pricing over time.