x402 paywall test: no search crawler paid the one-cent price

What happened

Suganthan Mohanadasan priced one page on his own site at a cent and watched five payments settle on the morning of September 15. Every one came from an agent he controls. He published the test at Search Engine Journal on September 17.

The demo runs at paid.suganthan.com on a Cloudflare Worker, roughly 300 lines of Hono code plus the x402 middleware, live since August 8. One route, /research/agentic-seo/, charges $0.01 per crawl. A second route publishes every paid crawl with timestamps, payer addresses and transaction hashes.

The exchange takes four steps. An agent requests the page and gets HTTP 402 back with a PAYMENT-REQUIRED header describing the price, network and asset. The agent signs an authorisation for the exact USDC amount and retries with a PAYMENT-SIGNATURE header. Coinbase’s facilitator verifies the signature, settles on chain, and the Worker returns the content with a PAYMENT-RESPONSE receipt.

None of it was real money. The payments used testnet USDC on Base Sepolia, where the tokens have no value. The buyer wallet held zero ETH and needed only the testnet USDC it was spending.

One payment came from Claude Code running on Mohanadasan’s own machine. A hook on its WebFetch tool caught the 402, checked the price against a $0.05 per-call cap and a $0.25 daily allowance, paid, and passed the content and receipt back as context.

His verdict is not to start charging this month expecting search crawlers to pay. GPTBot, ClaudeBot, PerplexityBot and Googlebot do not turn up with wallets.

Why it matters

HTTP reserved status 402 for payment in 1997. X402 gives it a job just as pricing turns into a dashboard toggle. Cloudflare’s Pay Per Crawl sits in closed beta, its Monetization Gateway is behind a waitlist announced July 1, and the Wallets handles it opened on August 4 still cannot send, receive or hold funds. The x402 Foundation launched under the Linux Foundation on July 14 with 40 members, including Visa, Mastercard, Google, AWS, Stripe and Cloudflare.

Enforcement is the real departure from robots.txt. A robots.txt rule works only when the crawler chooses to honour it. A 402 withholds the bytes until payment clears.

The cost of that enforcement lands on the bots most sites want. Training crawlers and retrieval bots both get the 402 and nothing else, so a priced resource drops out of the systems that cite it while earning nothing. Mohanadasan tested because of the gap between how much gets crawled and how little traffic comes back, the same pressure now putting bot traffic ahead of human readers at Wikipedia.

Volume numbers on this market deserve suspicion. Decrypt reported on September 13 that TRM Labs examined 198.9 million x402 settlements and, after filtering anomalous flows, attributed between 0.6% and 7.5% of the value to AI agents. Scheduled jobs, load tests and self-dealing leave the same blockchain records, and TRM’s test cannot separate an agent from a script.

Google’s AI contribution pilot answers the same question from the platform side. It pays when a page contributed while an answer was being generated. A link added afterwards does not qualify. Publishers get a monthly earnings figure in Search Console without the calculation behind it. A 402 trades that for a smaller, stranger market with a price and a receipt the site owner can check.

What to do

No site-wide pricing yet. The crawlers most sites care about cannot transact, so this month is reconnaissance, in rough order:

• See the offer yourself. curl -i https://paid.suganthan.com/research/agentic-seo/ returns HTTP/2 402 with a base64 payment-required header you can decode.
• Shortlist resources, not pages. Original datasets, research and working tools are the candidates. General explainers earn discovery, so leave them free.
• Set budgets before wallets. If your own agents will be buying pages, cap spend per call and per day, and check both before anything gets signed.
• Ask Google for the arithmetic. If pilot terms arrive, ask how the monthly figure was calculated before accepting them.

Deciding which resources carry a price is cheaper before the checkbox appears in a dashboard than after.

Watch out for

• User-agent gating is spoofable. A mode he built but left disabled charges only requests identifying as GPTBot, ClaudeBot or PerplexityBot, and he tested it with faked user-agent strings. A string proves nothing about who sent the request, which is why Cloudflare verifies crawler identity cryptographically instead.
• A reserved Cloudflare handle is not a wallet. Cloudflare’s documentation says a reserved handle cannot yet send, receive or hold funds. Account Wallets hold the money and the limits, while Virtual Wallets are what agents spend from.
• Testnet is one variable away from production. A NETWORK setting flips the Worker from base-sepolia to base and moves the flow to real USDC. Nothing in the published results has cost or earned anything.
• Cloudflare’s own program is gated on the buyer side. Pay Per Crawl requires bot operators to clear Web Bot Auth verification, Stripe onboarding and program approval, which is why the demo used the open protocol.