Verify the crawler, not its name: Web Bot Auth, IP checks, and the third gate.

Two posts ago I told people to check their server logs for AI crawlers getting a 403. That leaves the obvious next question: fine, you found the block — now how do you let the right crawlers through without letting everyone in?

This is that post: the model I've worked out for it, and the cracks I keep hitting along the way.

Upfront, so nobody mistakes this for a war story: I haven't shipped this on my own edge yet. It's research toward doing it properly. If you run this in production, the comments are where I learn something.

One question that's actually two

You want AI crawlers to reach your content — that's how you get cited in ChatGPT, Perplexity and Gemini. You don't want every scraper on the internet walking in behind them.

"Let the good ones in, keep the bad ones out" sounds like one job. It's two, and they pull against each other. Open the door wider for GPTBot and you've opened it for anything willing to type GPTBot into a header. Close it and you vanish from the answers you were trying to appear in.

The whole problem is identity: how do you know a request is really from the crawler it claims to be? Everything below is three increasingly honest answers to that.

Answer 1 — the name (and why it's worthless)

The lazy fix: allow anything whose User-Agent contains GPTBot.

A User-Agent is a string. Anyone can type it. A scraper labels itself GPTBot, sails through, and your gesture toward AI visibility is now an open door for exactly the traffic you were keeping out.

The name isn't identity. It's a claim. You need something the visitor can't fake.

Answer 2 — the origin (IP)

Better: check where the request actually came from. Done right, it's a two-step DNS dance, not a flat IP lookup:

1. Reverse-DNS the requesting IP → get a hostname
2. Confirm the hostname belongs to the vendor — .crawl.openai.com, .anthropic.com, .perplexity.ai, .googlebot.com
3. Forward-resolve that hostname → it must return the original IP

A scraper can claim a hostname. It can't make the vendor's DNS forward-resolve to its own IP. Both directions have to agree.

Some vendors also publish raw ranges to match against:

• OpenAI — gptbot.json, searchbot.json, chatgpt-user.json
• Perplexity — perplexitybot.json, perplexity-user.json
• Google — crawler ranges

Where origin breaks — three cracks, none small

Not everyone publishes. This one surprised me: Anthropic does not publish IP ranges for ClaudeBot. The IPs in their docs are for the Claude API, not the crawler. Their own guidance for identifying ClaudeBot is User-Agent plus reverse-DNS against *.anthropic.com. So for one of the three biggest crawlers, "just match the published range" isn't an option that exists.

Google won't separate training traffic by IP. Google-Extended — the control for Gemini and AI Overviews training — is a robots.txt token, not an IP set. There's no google-extended.json. You can allow or deny it in robots.txt, but you cannot distinguish it at the network layer. IP verification doesn't reach it at all.

Rotation. This is the sharpest crack of all. A stale allowlist and a 20-minute 403 window are both real costs, and "just poll the JSON more often" is a band-aid on a design that shouldn't have required polling.

Every published range is a snapshot of a moving target. The vendor rotates, or ships a new block late, and you choose between an allowlist that's already wrong and a window where legitimate crawlers eat a 403. You're not verifying identity anymore. You're babysitting CIDR blocks and hoping the JSON is current.

Answer 3 — the proof (Web Bot Auth)

This is where the trust model changes shape instead of just getting fussier.

Web Bot Auth — a Cloudflare-led IETF draft (draft-meunier-web-bot-auth-architecture, built on RFC 9421 HTTP Message Signatures) — has each crawler sign its requests with an Ed25519 key. The public key lives at a well-known directory on the vendor's own domain. You verify the signature. Identity is proven by cryptography, not inferred from where the packet came from.

The shift is the entire point: from "where did this connection originate" to "can you prove you hold the key." That survives rotation. It doesn't care which datacenter the request left. No CIDR to babysit, no JSON to poll.

It's real and moving fast:

• Integrated into Cloudflare's Verified Bots program, with a Signed Agents directory
• ChatGPT's agent was in the first signed cohort, 2025
• AWS WAF added Web Bot Auth support in late 2025, auto-allowing verified agents

Status, honestly: an active IETF draft, not a ratified standard. But with Cloudflare, OpenAI, Anthropic and AWS moving in lockstep, it's already the de-facto direction.

Where proof breaks

It only helps for crawlers that actually sign. And the scrapers you most want to block are exactly the ones who'll never adopt it. Signatures verify the honest. They do nothing to the dishonest.

So why not just wait for signatures to win?

Because signatures answer a narrower question than it looks. The reframing that finally made it click for me: signatures move the question from "who are you" to "what are you doing."

Sit with that. A valid signature proves identity, not intent. A legitimate crawler having a bad day and a stolen key behave identically at the signature layer — both present a valid signature. Verification hands you a name. It says nothing about whether the holder of that name is currently hammering your origin at 400 requests a second.

Which means there is always a floor under everything else: behaviour. Rate limits, request patterns, what a client reaches for once it's through the door. Verification gets you a name. Behaviour decides whether the name gets to keep acting like one.

The model: three gates, not one

Put the three answers in order and they stop competing. They stack.

1. Signed? Verify the signature → fast-path it. (Web Bot Auth — proof)
2. Not signed, but claims a known crawler? Reverse-DNS + forward-confirm, or a published range. (IP — origin)
3. Through either gate — now watch it. Rate limits and behaviour, always on, for everyone. (the floor)

The way to hold it in your head: additive, not a replacement. Once you see the gates it's obvious why. Signatures don't replace the IP layer — they shrink its blast radius. Every honest crawler that signs steps out of the pool you're forced to judge by circumstance. What's left for the IP-and-vibes gate is smaller and dirtier — a population that's mostly bad actors, which is a much easier one to be aggressive with. And behind both, behaviour never stops watching, because identity was never the same thing as intent.

Spotting a fake while the world catches up

Until signatures are universal, most of the work is catching User-Agent liars. The tells are consistent:

• Reverse-resolves to a general host (Hetzner, OVH, DigitalOcean) with no vendor match
• NXDOMAIN — no reverse-DNS at all
• Goes straight for /admin, /.env, /wp-login — real crawlers read content, not secrets
• Hundreds of requests a minute, no Crawl-Delay
• A typo in the name: GPT-Bot, or a version that doesn't exist (ClaudeBot/2.x)

None is proof alone. Together they're a scraper wearing a badge.

The rule I took away

The whole progression is one idea getting more honest with itself:

• The name asks the visitor to identify themselves. Trusts a string.
• The origin asks where they came from. Trusts an address that rotates or goes unpublished.
• The proof asks them to prove they hold a key. Trusts math — but only about who, never what.

Each gate moves trust closer to something the visitor can't fake. None of them is the finish line, because the finish line doesn't exist: even perfect identity tells you nothing about intent. You verify by proof where you can, by origin where you can't, and you hold the line with behaviour for everyone — because a name was never a promise of good behaviour.

I haven't built this yet. When I do, it's signatures first, IP as fallback, behaviour underneath — and I'll write up what broke, because something always does.

More about this

• Your HTML is fine. The CDN still blocks the bot.
• Your page ranks #1 on Google and is invisible to ChatGPT
• We were prerendering the spinner, not the page