GPTBot, ClaudeBot, Bytespider, and 47+ crawlers are harvesting your content at industrial scale. Block them from DNS to WAF to reverse proxy using our 17,410+ classified AI-tool domains.
Traditional bot management assumed crawlers identify themselves honestly and respect your rules. AI agents broke that assumption.
| Traditional Search Crawlers | AI Agents & Crawlers | |
|---|---|---|
| Identity | Honest user-agent strings | Often disguised or rotated |
| Robots.txt | Universally respected | Inconsistently followed |
| Value Exchange | Referral traffic & visibility | One-sided extraction |
| Rate Limiting | Self-throttled, crawl-delay | Aggressive, often ignores limits |
| Purpose | Search indexing | Model training, RAG, answer synthesis |
| IP Source | Known datacenter ranges | Datacenter + residential proxies |
Autonomous AI assistants pose direct operational risks. They browse the web on behalf of users, fill out forms, exhaust API quotas, and create phantom sessions that consume server resources.
GPTBot, ClaudeBot, Google-Extended, CCBot, and Bytespider scrape content for LLM training datasets. They crawl aggressively and harvest enormous volumes of text, code, and structured data.
PerplexityBot, YouBot, and Cohere-ai fetch pages on-demand to synthesize AI-generated answers. They consume your content while actively diverting your traffic.
AI assistants that use headless browsers to browse, interact with apps, and execute tasks. They simulate human behavior from diverse IP ranges, creating unpredictable load patterns.
New AI crawlers appear weekly and change user-agent strings frequently. A config that was comprehensive in January may miss a dozen new crawlers by June.
# ============================================================ # AI Crawler Blocking — robots.txt # Comprehensive block of known AI training, RAG, and # content-synthesis crawlers. Update monthly from the # AI Tools Blocklist crawler registry. # ============================================================ # --- OpenAI crawlers --- User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OAI-SearchBot Disallow: / # --- Anthropic crawlers --- User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / # --- Google AI crawlers (separate from Googlebot search) --- User-agent: Google-Extended Disallow: / # --- Common Crawl (used by many AI training pipelines) --- User-agent: CCBot Disallow: / # --- ByteDance / TikTok --- User-agent: Bytespider Disallow: / # --- Meta / Facebook AI --- User-agent: FacebookBot Disallow: / User-agent: Meta-ExternalAgent Disallow: / # --- Apple AI training --- User-agent: Applebot-Extended Disallow: / # --- AI search / answer engines --- User-agent: PerplexityBot Disallow: / User-agent: YouBot Disallow: / User-agent: Cohere-ai Disallow: / # --- Other known AI crawlers --- User-agent: Diffbot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Timpibot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: img2telegrambot Disallow: / User-agent: Scrapy Disallow: / # --- Preserve search engine access --- User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: * Allow: /
Googlebot = search indexing (allow). Google-Extended = AI training (block).Applebot = Siri & search (allow). Applebot-Extended = AI training (block). Blocking the wrong identifier removes your site from search results.The X-Robots-Tag header extends robots.txt controls to API responses and dynamic content. Several AI companies now respect X-Robots-Tag: noai and X-Robots-Tag: noimageai directives.
Robots.txt relies on crawler cooperation. Academic crawlers, startup MVPs, and crawlers from weak-IP jurisdictions routinely ignore it entirely.
AI agents that browse the web in real time don't consult robots.txt because they don't consider themselves "crawlers."
To enforce blocking rather than merely requesting it, implement server-side controls at the reverse proxy, WAF, or application layer. These inspect every incoming request and reject AI agent traffic regardless of robots.txt compliance.
The crawl wave is accelerating — and a layered blocking stack measurably turns it away at the edge.
Reverse proxy rules actively inspect every request and reject AI agent matches before they reach your application. User-agent string matching adds less than a microsecond of latency per request.
Uses a hash table for O(1) lookup at any traffic volume. Add new signatures by appending a line and reloading — no restart needed. Automate updates via cron from the AI Tools Blocklist feed.
Uses a compound regex in RewriteCond. Functionally identical but less performant for 30+ patterns since regex evaluates sequentially. Consider mod_security for large lists.
# ============================================================ # Nginx — AI Agent Blocking Configuration # Add to nginx.conf or include in site-specific server block. # Uses map for O(1) lookup performance at any request volume. # ============================================================ # Define AI bot detection map (place in http {} context) map $http_user_agent $is_ai_bot { default 0; # Training data crawlers ~*GPTBot 1; ~*ChatGPT-User 1; ~*OAI-SearchBot 1; ~*ClaudeBot 1; ~*anthropic-ai 1; ~*Google-Extended 1; ~*CCBot 1; ~*Bytespider 1; ~*FacebookBot 1; ~*Meta-ExternalAgent 1; ~*Applebot-Extended 1; # RAG and answer engine crawlers ~*PerplexityBot 1; ~*YouBot 1; ~*Cohere-ai 1; # Additional AI scrapers ~*Diffbot 1; ~*Omgilibot 1; ~*Timpibot 1; ~*PetalBot 1; ~*ImagesiftBot 1; ~*Scrapy 1; ~*img2dataset 1; ~*Kangaroo\sBot 1; ~*ISSCyberRiskCrawler 1; ~*Sidetrade\sindexer 1; } server { listen 443 ssl http2; server_name example.com; # Block AI bots before any processing if ($is_ai_bot) { return 403; } # Log blocked AI bot requests separately for analysis access_log /var/log/nginx/ai_bots_blocked.log combined if=$is_ai_bot; # Rate limiting for unidentified bots limit_req_zone $binary_remote_addr zone=bot_limit:10m rate=10r/s; location / { limit_req zone=bot_limit burst=20 nodelay; proxy_pass http://backend; } # Additional protection for sensitive endpoints location /api/ { limit_req zone=bot_limit burst=5 nodelay; # Require authentication for API access if ($http_authorization = "") { return 401; } proxy_pass http://api_backend; } } # ============================================================ # Apache .htaccess equivalent # ============================================================ # RewriteEngine On # RewriteCond %{HTTP_USER_AGENT} (GPTBot|ChatGPT-User|OAI-SearchBot # |ClaudeBot|anthropic-ai|Google-Extended|CCBot|Bytespider # |FacebookBot|Meta-ExternalAgent|Applebot-Extended # |PerplexityBot|YouBot|Cohere-ai|Diffbot|Omgilibot # |Timpibot|PetalBot|Scrapy) [NC] # RewriteRule .* - [F,L]
Sophisticated AI agents disguise their user-agent strings to evade pattern matching. A production-grade strategy must layer user-agent filtering with behavioral detection that identifies AI traffic patterns regardless of declared identity.
JA3/JA4 TLS fingerprints reveal the client's true identity regardless of user-agent. Crawlers built on Python requests, Go net/http, or Node.js produce distinctive handshake signatures that differ from real browsers.
AI crawlers show distinctive patterns: sequential page fetches, systematic URL traversal, no CSS/JS/image requests, missing referrer headers, and unnaturally consistent timing.
Executes on every request with full access to headers, URL, and TLS metadata. Supports arbitrary blocking logic combining user-agent matching, JA3 fingerprints, and behavioral heuristics.
// ============================================================ // Cloudflare Worker — AI Agent Detection & Blocking // Deploy as a route-level Worker on your zone. // Combines user-agent, TLS fingerprint, and behavioral signals. // ============================================================ const AI_BOT_PATTERNS = [ /GPTBot/i, /ChatGPT-User/i, /OAI-SearchBot/i, /ClaudeBot/i, /anthropic-ai/i, /Google-Extended/i, /CCBot/i, /Bytespider/i, /FacebookBot/i, /Meta-ExternalAgent/i, /Applebot-Extended/i, /PerplexityBot/i, /YouBot/i, /Cohere-ai/i, /Diffbot/i, /Omgilibot/i, /Timpibot/i, /PetalBot/i, /Scrapy/i, /img2dataset/i, ]; // Known AI crawler TLS fingerprints (JA3 hashes) const AI_JA3_HASHES = new Set([ "b32309a26951912be7dba376398abc3b", // Python requests "3b5074b1b5d032e5620f69f9f700ff0e", // Go net/http "e7d705a3286e19ea42f587b344ee6865", // Node.js fetch "cd08e31494f9531f0ab8fd18ff2af55c", // aiohttp ]); async function handleRequest(request) { const ua = request.headers.get("user-agent") || ""; const cf = request.cf || {}; let score = 0; let signals = []; // Signal 1: User-agent match (highest confidence) if (AI_BOT_PATTERNS.some(p => p.test(ua))) { score += 100; signals.push("ua_match"); } // Signal 2: Empty or missing user-agent if (!ua || ua.length < 10) { score += 40; signals.push("empty_ua"); } // Signal 3: TLS fingerprint match if (cf.tlsClientAuth && AI_JA3_HASHES.has(cf.ja3Hash)) { score += 60; signals.push("ja3_match"); } // Signal 4: Missing typical browser headers if (!request.headers.get("accept-language") && !request.headers.get("sec-fetch-mode")) { score += 30; signals.push("missing_browser_headers"); } // Signal 5: Datacenter ASN (non-residential IP) if (cf.asOrganization && /(amazon|google|microsoft|digital.?ocean|hetzner|ovh)/i .test(cf.asOrganization)) { score += 20; signals.push("datacenter_ip"); } // Block if score exceeds threshold if (score >= 70) { return new Response( "Access denied: automated AI agent traffic is not permitted.", { status: 403, headers: { "X-Block-Reason": signals.join(","), "X-Bot-Score": String(score), "Cache-Control": "no-store", }, } ); } return fetch(request); } addEventListener("fetch", (event) => { event.respondWith(handleRequest(event.request)); });
| Signal | Score | What It Catches |
|---|---|---|
| User-agent match | +100 | Known AI crawlers that self-identify |
| JA3 TLS fingerprint | +60 | Python/Go/Node clients posing as browsers |
| Empty user-agent | +40 | Lazy scrapers with no identification |
| Missing browser headers | +30 | Non-browser clients (no accept-language/sec-fetch) |
| Datacenter ASN | +20 | AWS/GCP/Azure/Hetzner/OVH infrastructure |
A known crawler (100) always triggers. A disguised agent from a datacenter (20) with no browser headers (30) and a Python TLS fingerprint (60) scores 110 and is blocked even while claiming to be Chrome.
The X-Block-Reason and X-Bot-Score headers are for debugging only. In production, log signals server-side via Cloudflare Logpush — returning them to clients leaks detection logic.
Most enterprise WAFs offer AI bot rule groups, but they cover only the top 10-15 crawlers. Custom rules sourced from the AI Tools Blocklist provide significantly broader protection.
Create rules with (http.user_agent contains "GPTBot") or ... expressions. WAF evaluates before Workers for an additional enforcement layer. Use the Cloudflare API to auto-update from the Blocklist feed.
Deploy custom rule groups with User-Agent string-match conditions. Combine with AWS WAF Bot Control for behavioral detection. Associate with CloudFront, ALBs, and API Gateway for unified enforcement.
Web pages are not the only target. AI agents increasingly interact with APIs — documentation endpoints, data APIs, and form submission endpoints. APIs don't serve robots.txt, so protection combines authentication, rate limiting, and pattern analysis.
Block DNS resolution for requests from known AI company IP ranges. The crawler can't even establish a connection — your site becomes invisible to their infrastructure.
Deploying blocking rules is the beginning, not the end. New crawlers launch weekly, existing ones change signatures, and evasion techniques grow more sophisticated — a sustainable program monitors effectiveness, detects bypasses, and measures performance impact.
Parse access logs daily for AI user-agents that bypassed rules. Cross-reference against the Blocklist crawler registry for new agents.
Automate weekly updates to robots.txt, Nginx maps, and WAF rules. Validate syntax before deploying. Auto-rollback if errors spike.
Track CPU, bandwidth, and latency before vs. after. Most orgs see 15-30% request volume reduction and faster response times.
Audit blocking logs weekly for false positives. Maintain an allowlist for partner integrations and monitoring services.
The multi-layer stack (robots.txt + reverse proxy + edge WAF + behavioral detection) consistently blocks over 98% of identifiable AI agent traffic.
Advanced evasion techniques: residential proxy rotation, full browser emulation with JS execution, and human-behavioral mimicry. Requires ML-based bot management platforms layered on top.
AI agent blocking is a continuously-evolving practice, not a one-time deployment. The AI Tools Blocklist's daily updates keep your domain intelligence current. Automate the pipeline from blocklist updates to rule deployment to monitoring for a self-maintaining defense posture.
Get started with our comprehensive AI crawler user-agent registry and IP range dataset. The free sample includes the top AI crawlers — upgrade to the full 17,410+ domain feed for complete protection with daily updates.
Tell us about your infrastructure and what AI agents you need to block — we will recommend the right blocking configuration for your stack.