---
title: "Verify the crawler, not its name: Web Bot Auth and IP checks"
description: "The name is a claim, IP rotates, Web Bot Auth proves identity, but only behaviour proves intent. A three-gate model for letting the right crawlers in."
url: https://seo7.es/en/blog/ai-seo-blog/verify-the-crawler-not-its-name
canonical: https://seo7.es/en/blog/ai-seo-blog/verify-the-crawler-not-its-name
lang: en
type: BlogPosting
---

# Verify the crawler, not its name: Web Bot Auth and IP checks

> The name is a claim, IP rotates, Web Bot Auth proves identity, but only behaviour proves intent. A three-gate model for letting the right crawlers in.

Page: https://seo7.es/en/blog/ai-seo-blog/verify-the-crawler-not-its-name

## Summary for AI

The name is a claim, IP rotates, Web Bot Auth proves identity: but only behaviour proves intent. A three-gate model for letting the right crawlers in. Two posts ago I told people to check their server logs for AI crawlers getting a 403. That leaves the obvious next question: fine, you found the block: now how do you let the right crawlers through without letting everyone in?

## Questions and answers

### Why is it dangerous to let bots in based on the GPTBot string in the User-Agent?

The User-Agent is a plain string anyone can write. A scraper signs itself as GPTBot and gets through the rule that was meant for AI visibility. The name is just a claim and does not prove identity.

### How does the two-step DNS check of a crawler work?

First, reverse-DNS on the request IP gives a hostname, and it must belong to the vendor: .crawl.openai.com, .anthropic.com, .perplexity.ai or .googlebot.com. Then a forward resolve of that hostname must return the original IP. If both sides match, the request is genuine.

### Where do OpenAI and Perplexity publish their crawler IPs?

OpenAI publishes gptbot.json, searchbot.json and chatgpt-user.json, and Perplexity publishes perplexitybot.json and perplexity-user.json. Google has its own crawler ranges. With these files you can check a request's IP against the official list.

### Does Anthropic publish ClaudeBot's IP ranges?

No. The IPs in Anthropic's documentation belong to the Claude API, and there are no ranges for the crawler. Anthropic advises identifying ClaudeBot by User-Agent plus reverse-DNS on *.anthropic.com. For one of the largest crawlers, checking against a published range isn't possible.

### Can Google-Extended be identified by IP?

No. Google-Extended controls Gemini training and AI Overviews through a robots.txt token; it has no separate set of IPs and no google-extended.json file. You can allow or disallow it in robots.txt, and at the network level it is indistinguishable from the rest of Google's traffic.

### How does IP rotation hurt a crawler allowlist?

Every published range is a snapshot of a moving target. The vendor changes blocks and the list updates late. The owner chooses between a stale allowlist and a window when legitimate crawlers get a 403. Polling the JSON often only masks the problem.

### What is Web Bot Auth?

An IETF draft led by Cloudflare, draft-meunier-web-bot-auth-architecture, built on RFC 9421 HTTP Message Signatures. The crawler signs every request with an Ed25519 key, the public key sits in a well-known directory on the vendor's domain, and the server verifies the signature.

### Why does a Web Bot Auth signature survive IP rotation?

The check asks whether the visitor holds the key and doesn't care which data center the request came from. So there's no need to track CIDR blocks or poll JSON with ranges: cryptography proves identity.

### Who already supports Web Bot Auth?

Signatures are built into Cloudflare's Verified Bots program with its Signed Agents directory. The ChatGPT agent joined the first signed cohort in 2025, and AWS WAF added support in late 2025 and lets verified agents through itself. The standard is still an active draft.

### Why don't signatures stop harmful scrapers?

Signatures only verify those who sign. The scrapers you most want to block will never adopt Web Bot Auth. For them, origin checks and behavior control remain.

### Why doesn't a valid signature guarantee good behavior?

A signature proves identity and says nothing about intent. A legitimate crawler on a bad day and a stolen key look the same: both present a valid signature. Only rate limits and request analysis show whether the visitor is overloading the server.

### What three gates are needed to let AI crawlers through?

First: a signed request takes the fast path after signature verification. Second: an unsigned one claiming to be a known crawler goes through reverse-DNS with forward confirmation or a range check. Third, for everyone: rate limits and behavior monitoring.

### What signs reveal a fake GPTBot?

The IP resolves to shared hosting such as Hetzner, OVH or DigitalOcean, or there is no reverse-DNS at all (NXDOMAIN). Requests hit /admin, /.env, /wp-login, hundreds per minute with no Crawl-Delay. The name has a typo: GPT-Bot or a nonexistent ClaudeBot/2.x version.

### Has the author implemented the verification scheme described?

Not yet: the article describes research, not a finished solution. The author's plan: signatures first, IPs as a fallback, behavior control underneath. After implementing it, he promises to tell what broke.

## Other languages

- RU: https://seo7.es/ru/blog/ai-seo-blog/verify-the-crawler-not-its-name
- ES: https://seo7.es/es/blog/ai-seo-blog/verify-the-crawler-not-its-name

## schema.org markup

```json
{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "name": "Verify the crawler, not its name: Web Bot Auth and IP checks",
  "description": "The name is a claim, IP rotates, Web Bot Auth proves identity, but only behaviour proves intent. A three-gate model for letting the right crawlers in.",
  "url": "https://seo7.es/en/blog/ai-seo-blog/verify-the-crawler-not-its-name",
  "mainEntityOfPage": "https://seo7.es/en/blog/ai-seo-blog/verify-the-crawler-not-its-name",
  "inLanguage": "en-GB",
  "headline": "Verify the crawler, not its name: Web Bot Auth and IP checks",
  "isPartOf": {
    "@type": "WebSite",
    "name": "SEO7.ES",
    "url": "https://seo7.es/"
  },
  "publisher": {
    "@type": "Organization",
    "name": "SEO7.ES",
    "url": "https://seo7.es/"
  }
}
```