# Pay-Per-Crawl and the New Bot Web

> AI Web Crawlers: GPTBot, ClaudeBot, PerplexityBot, and Pay‑Per‑Crawl As of 2026, AI‑driven web crawlers such as GPTBot, ClaudeBot, and PerplexityBot now account for a…

- **Canonical:** https://agentflare.org/research/pay-per-crawl-and-the-new-bot-web.html
- **Updated:** 2026-06-23
- **Category:** research
- **Full structured data:** `/research/pay-per-crawl-and-the-new-bot-web.data.json` — $0.02 via AISA HTTP 402 (https://esa.aisa.one/api/v1/access/verify; agents set crawlerAutoPrice=true)

## Key data

- **Topic:** discovery
- **Sources:** 10
- **Updated:** 2026-06-23

# AI Web Crawlers: GPTBot, ClaudeBot, PerplexityBot, and Pay‑Per‑Crawl

_…full analysis and the complete dataset are available to agents for $0.02 — fetch `/research/pay-per-crawl-and-the-new-bot-web.data.json` (HTTP 402)._

## Sources

1. [https://www.tencentcloud.com/techpedia/143900](https://www.tencentcloud.com/techpedia/143900)
2. [https://evolveamz.com/ai-crawler-list-2026-ecommerce/](https://evolveamz.com/ai-crawler-list-2026-ecommerce/)
3. [https://nohacks.co/blog/ai-user-agents-landscape-2026](https://nohacks.co/blog/ai-user-agents-landscape-2026)
4. [https://www.digitalapplied.com/blog/ai-crawler-bot-traffic-statistics-2026-data-reference](https://www.digitalapplied.com/blog/ai-crawler-bot-traffic-statistics-2026-data-reference)
5. [https://www.tryaivo.com/blog/ai-crawler-cheat-sheet-2025-which-bots-should-you-allow](https://www.tryaivo.com/blog/ai-crawler-cheat-sheet-2025-which-bots-should-you-allow)
6. [https://www.oncrawl.com/ai/what-ai-bots-really-doing-your-site/](https://www.oncrawl.com/ai/what-ai-bots-really-doing-your-site/)
7. [https://blog.cloudflare.com/crawlers-click-ai-bots-training/](https://blog.cloudflare.com/crawlers-click-ai-bots-training/)
8. [https://www.digitalapplied.com/blog/ai-crawler-access-control-2026-robots-llms-txt-decision-matrix](https://www.digitalapplied.com/blog/ai-crawler-access-control-2026-robots-llms-txt-decision-matrix)
9. [https://www.humansecurity.com/learn/blog/crawlers-list-known-bots-guide/](https://www.humansecurity.com/learn/blog/crawlers-list-known-bots-guide/)
10. [Introducing pay per crawl: Enabling content owners to charge AI ...](https://blog.cloudflare.com/introducing-pay-per-crawl)

## Related

- [Generative Engine Optimization (GEO): A Primer](https://agentflare.org/research/generative-engine-optimization-geo-a-primer.html)
- [llms.txt: The Standard for AI-Readable Sites](https://agentflare.org/research/llmstxt-the-standard-for-ai-readable-sites.html)
- [HTTP 402 & x402: How AI Agents Pay for Content](https://agentflare.org/research/http-402-x402-how-ai-agents-pay-for-content.html)
- [The AI Agent Economy in 2026](https://agentflare.org/research/the-ai-agent-economy-in-2026.html)
- [Model Context Protocol (MCP) Explained](https://agentflare.org/research/model-context-protocol-mcp-explained.html)
- [Stablecoins as Rails for Autonomous Agents](https://agentflare.org/research/stablecoins-as-rails-for-autonomous-agents.html)

---
_Part of AgentFlare, an agent-native data network powered by AISA. https://aisa.one/docs_