# Budgeting a Paid Crawl: Discover, Survey, Then Buy

> Crawl a whole site on a fixed budget by reading the free discovery files first, then using free summaries to choose, and buying only the .data.json or…

- **Canonical:** https://agentflare.org/guides/budgeting-a-paid-crawl-discover-survey-then-buy.html
- **Updated:** 2026-10-30
- **Category:** guides

## Key data

- **Track:** For agent developers
- **Level:** Intermediate
- **Access:** Free / open
- **Updated:** 2026-10-30

Crawl a whole site on a fixed budget by reading the free discovery files first, then using free summaries to choose, and buying only the .data.json or .full.md files worth paying for. Stop cleanly when the balance runs out.

## Sources

1. [Header name: X-Crawler-Token](https://esa.aisa.one/cdn/open-verify.html)
2. [AISA Agent Integration Guide — API reference](https://esa.aisa.one/agent/docs.html)
3. [AISA agent skill (SKILL.md)](https://esa.aisa.one/agent/skill.md)
4. [AgentFlare pricing.json (directory price map)](https://agentflare.org/pricing.json)
5. [AgentFlare robots.txt](https://agentflare.org/robots.txt)

## Related

- [Getting Started with HTTP 402 Pay-Per-Crawl](https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html)
- [Crawler Tokens & Auto-Pay: A How-To](https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html)
- [Handling 402 and 403 Responses: An Error Reference for Agents](https://agentflare.org/guides/handling-402-and-403-responses-an-error-reference-for-agents.html)
- [Building a Pay-Per-Crawl Client for Your Agent](https://agentflare.org/guides/building-a-pay-per-crawl-client-for-your-agent.html)
- [Pricing Content by Directory: Best Practices](https://agentflare.org/guides/pricing-content-by-directory-best-practices.html)
- [robots.txt, llms.txt & Sitemaps for AI Crawlers](https://agentflare.org/guides/robotstxt-llmstxt-sitemaps-for-ai-crawlers.html)

---
_Part of AgentFlare, an agent-native data network powered by AISA. https://aisa.one/docs_