---
title: "Your HTML is fine. The CDN still blocks the bot."
description: "Cloudflare Radar: ~5% of AI-crawler requests hit a 403 at the CDN, refused before your HTML loads. The block below robots.txt, seen only in your server logs."
url: https://seo7.es/en/blog/ai-seo-blog/your-html-is-fine-the-cdn-still-blocks-the-bot
canonical: https://seo7.es/en/blog/ai-seo-blog/your-html-is-fine-the-cdn-still-blocks-the-bot
lang: en
type: BlogPosting
---
# Your HTML is fine. The CDN still blocks the bot.
> Cloudflare Radar: ~5% of AI-crawler requests hit a 403 at the CDN, refused before your HTML loads. The block below robots.txt, seen only in your server logs.
Page: https://seo7.es/en/blog/ai-seo-blog/your-html-is-fine-the-cdn-still-blocks-the-bot
## Summary for AI
Cloudflare Radar: ~5% of AI-crawler requests hit a 403 at the CDN: refused before your HTML loads. The block below robots.txt, seen only in your server logs. You did the work. The page is server-rendered. The JSON-LD is in the raw response. curl returns the whole article, headline and all. A crawler that fetches your URL gets everything it needs.
## Questions and answers
### Why doesn't the crawler get the page if the HTML is fine?
The crawler hits the CDN, and the CDN can answer 403 before the request ever reaches the server. The page is server-rendered, JSON-LD is in place, curl returns the whole article, but the bot sees only an error code.
### How does a robots.txt disallow differ from a WAF rule?
robots.txt is like a note on a door: a polite crawler reads it, an impolite one walks past, and it has nothing to do with the page's bytes. A WAF or CDN rule is the door itself: it answers at the edge with a 403, a 429 or a JavaScript challenge.
### What response codes do AI crawlers get, according to Cloudflare Radar?
According to the AI Insights dashboard for 7 days as of July 18, 2026: 200 OK in 73.6% of responses, 403 Forbidden in 5.2%, 429 Too Many Requests in 1.3%, 503 in 1.1%. More than one request in fifteen is rejected at the edge.
### Why are AI bots often blocked without the site owner's decision?
The block is usually on by default: a managed WAF ruleset, the "block AI bots" toggle switched on in 2024, or an overly strict bot-fight score. The owner never wrote such a rule and doesn't know about the block.
### Which AI crawlers are most often disallowed in robots.txt?
According to the robots.txt tracker in Cloudflare Radar as of July 18, 2026, the most often disallowed among top domains are GPTBot, CCBot and ClaudeBot. In large part these are inherited rules rather than a conscious choice.
### Why doesn't the owner see the block in the browser?
The owner's browser isn't on the blocklist: a home IP and a Chrome User-Agent pass any check. robots.txt shows Allow: /, the source is clean. All the tools come from the owner, while the block targets GPTBot.
### Where can you see what GPTBot actually received?
In the server's access log. It has one line per real request, with the visitor's User-Agent and the exact status code. It is the only record of what the crawler received, and unlike the browser and robots.txt it doesn't mislead.
### How can one command count the response codes served to AI crawlers?
Filter access.log with grep for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended, print the status field with awk and count with sort and uniq -c. 200 means the page was received, 403 an edge refusal, 429 a rate limit.
### What does a silent failure with AI crawlers look like?
robots.txt is fully open, yet the logs are full of 403s. Everything the owner sees says "yes", while the layer that decides says "no". A typical story: months of content, zero mentions in ChatGPT and Perplexity, and only an audit finds that Cloudflare was serving GPTBot a 403.
### Why shouldn't you fix the block with a User-Agent rule?
Anyone can type a User-Agent. A price scraper will sign itself as GPTBot and slip through the generous new rule. In its transparency tracking, Cloudflare marks which operators can be verified: as of mid-2026 ByteDance is listed as unverified.
### How do you verify a crawler against published IP ranges?
Take the vendor's files, for example openai.com/gptbot.json, searchbot.json and chatgpt-user.json, run reverse-DNS on the request IP, make sure the hostname belongs to the vendor, and resolve it forward back to the same IP. If even one step fails, it's a spoofer.
### How does AWS WAF handle signed bots?
In late 2025 AWS WAF added support for Web Bot Auth and automatically lets verified agents through. The Ed25519 signature proves the crawler's identity. The standard is still an IETF draft, but Cloudflare, OpenAI, Anthropic and AWS already support it.
### How long does an AI crawler access audit take?
One evening. Pull a week of logs, filter by AI User-Agents and look at the status codes. If you see 403 or 429, look for the rule: almost always it's a managed ruleset or the AI-bot blocking setting.
### What layers does a request pass through before reaching the page HTML?
DNS, CDN, WAF, rate limiter and bot-management scoring. Any of them can cut off a request with a status code the owner never sees, while the page renders flawlessly in their browser. The right question is which code the bot received.
## Other languages
- RU: https://seo7.es/ru/blog/ai-seo-blog/your-html-is-fine-the-cdn-still-blocks-the-bot
- ES: https://seo7.es/es/blog/ai-seo-blog/your-html-is-fine-the-cdn-still-blocks-the-bot
## schema.org markup
```json
{
"@context": "https://schema.org",
"@type": "BlogPosting",
"name": "Your HTML is fine. The CDN still blocks the bot.",
"description": "Cloudflare Radar: ~5% of AI-crawler requests hit a 403 at the CDN, refused before your HTML loads. The block below robots.txt, seen only in your server logs.",
"url": "https://seo7.es/en/blog/ai-seo-blog/your-html-is-fine-the-cdn-still-blocks-the-bot",
"mainEntityOfPage": "https://seo7.es/en/blog/ai-seo-blog/your-html-is-fine-the-cdn-still-blocks-the-bot",
"inLanguage": "en-GB",
"headline": "Your HTML is fine. The CDN still blocks the bot.",
"isPartOf": {
"@type": "WebSite",
"name": "SEO7.ES",
"url": "https://seo7.es/"
},
"publisher": {
"@type": "Organization",
"name": "SEO7.ES",
"url": "https://seo7.es/"
}
}
```