Budgeting a Paid Crawl: Discover, Survey, Then Buy

This guide is for agent developers who need to crawl an entire AISA-protected site without overspending. You will fetch the free discovery files, read the directory price map, skip blocked directories before sending a request, use free summaries to pick what to buy, and track spending against your real balance. Your loop will stop on insufficient_balance and pass a checkout link to your user instead of trying to fund itself. The examples use agentflare.org's published price map.

How the flow works

Before you start

• A registered AISA crawler account with a crawler token
• curl and Python 3 with the requests library
• A budget in USD that you set before the crawl starts

Step by step

1. Fetch the free discovery files

   Start with the files that cost nothing: robots.txt, llms.txt, sitemap.xml and pricing.json. Together they give you the site map, the section descriptions and the per-directory prices before you spend anything. Send your crawler token in the X-Crawler-Token header, which replaces the earlier name X-AISA-Crawler-Token, and your honest User-Agent on every request. Denials never charge, and these files belong to the free tier.

   for f in robots.txt llms.txt sitemap.xml pricing.json; do
     curl -s https://agentflare.org/$f -o $f \
       -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \
       -A 'MyBot/1.0 (+https://example.com/bot)'
   done

   S1 · esa.aisa.oneS2 · esa.aisa.oneS3 · esa.aisa.oneS5 · agentflare.org

2. Read the price map and mark blocked paths

   Parse pricing.json for three things: directory_pricing_usd for the cost of each directory, rules with mode allow for open data, and rules with mode block for directories you must not request. robots.txt allows every crawler here, so it will not show you what is blocked. The block rules in pricing.json do. Build your skip list from them before the crawl starts.

   import requests
   p = requests.get('https://agentflare.org/pricing.json').json()
   prices = p['directory_pricing_usd']
   blocked = [r['urlPattern'] for r in p['rules'] if r['mode'] == 'block']
   free = [r['urlPattern'] for r in p['rules'] if r['mode'] == 'allow']
   print('prices', prices)
   print('skip these', blocked)
   print('open to agents', free)

   S2 · esa.aisa.oneS3 · esa.aisa.oneS4 · agentflare.orgS5 · agentflare.org

3. Survey with free summaries before buying

   Every article has a free .md summary and a free .html page. Use them only to decide what is worth buying, and take the real article slugs from sitemap.xml. Only the structured deep files, .data.json and .full.md, are priced. If a summary shows the article is not relevant to your task, drop its deep file from the purchase list and spend nothing on it.

   curl -s https://PUBLISHER-SITE/research/report.md \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \
     -A 'MyBot/1.0 (+https://example.com/bot)'

   S2 · esa.aisa.oneS3 · esa.aisa.oneS4 · agentflare.org

4. Buy selectively and track a running total

   Fetch each chosen deep file once, because fetching twice pays twice. Read the real balance from GET /api/agent/balance before each request, and derive spend as the starting balance minus the balance now. Stop at your budget. On a 402 with error insufficient_balance, stop and report crawler.shortfall. On a 403 access_blocked, skip that URL and move on, because payment does not unlock it.

   import requests
   H = {'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN', 'User-Agent': 'MyBot/1.0 (+https://example.com/bot)'}
   BASE = 'https://cdn.aisa.one'
   BUDGET = 0.10
   def bal():
       return requests.get(BASE + '/api/agent/balance', headers=H).json()['crawler']['balance']
   start = bal()
   for u in ['https://PUBLISHER-SITE/research/report.data.json']:
       if start - bal() >= BUDGET:
           print('budget reached'); break
       r = requests.get(u, headers=H)
       if r.status_code == 200:
           print('fetched', u)
       elif r.status_code == 402 and r.json().get('error') == 'insufficient_balance':
           print('out of balance, shortfall', r.json()['crawler']['shortfall']); break
       elif r.status_code == 403:
           print('blocked, skipping', u)

   S2 · esa.aisa.oneS3 · esa.aisa.one

5. Hand the checkout link to your user

   When the loop stops on insufficient_balance, do not fund yourself. Completing a Stripe payment needs a human, so create a checkout session with your session token and show the checkout_url to your user. The minimum top-up is 0.50 USD. After they pay, fetch the real balance again and resume. Never call POST /api/agent/balance/add, which exists only for human operators seeding test accounts.

   curl -s -X POST https://cdn.aisa.one/api/agent/topup/create-session \
     -H 'Content-Type: application/json' \
     -H 'Authorization: Bearer SESSION_TOKEN' \
     -d '{ "amount": 5.00 }'
   # returns checkout_url: show it to your user and wait

   S2 · esa.aisa.oneS3 · esa.aisa.one

agentflare.org directory prices and modes

Common pitfalls

• Each granted fetch of a priced resource deducts its price once. Fetch it twice and you pay twice, so keep a list of what you have already bought.
• A 403 access_blocked is the publisher's policy, so retrying or topping up will not help. Skip that URL and carry on with purchasable directories.
• Never send the crawler token in Authorization: Bearer. That header is reserved for the gateway layer. Use X-Crawler-Token only.
• Do not quote a remembered balance. Call GET /api/agent/balance before reporting any figure, because a stored value goes stale the moment you spend.

Key takeaways

• Discovery files and free summaries cost nothing, so read pricing.json and the .md teasers before buying any deep file.
• Check the real balance from GET /api/agent/balance, and stop the loop on insufficient_balance or when your budget is reached.
• Skip 403 access_blocked URLs and block-mode directories, and have a human pay through the checkout_url when funds run out.

FAQ

How this page is made. Written from 5 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.