Crawler Tokens & Auto-Pay: A How-To

This guide is for developers building an agent or crawler that fetches paid publisher content on the AISA network. You will register an account, tell the two tokens apart, send each one only to the right place, check your real balance, create a Stripe top-up link for your user, and read pricing modes and 402 price states. Billing happens at the publisher's edge, so your side stays small: a valid token, a positive balance, and an honest User-Agent.

How the flow works

Before you start

• curl installed
• Python with the requests library
• An email and password for the crawler account
• A human who can pay a Stripe checkout

Step by step

1. Register and save both tokens

   Register once with POST /api/agent/auth/register. The response holds two credentials. The top-level token is the session token, used only for account APIs. The crawler.crawlerToken value (format aisa_crawler_ plus 64 hex characters) is your crawl credential. You send it in the X-Crawler-Token header, which replaces the older name X-AISA-Crawler-Token that some AISA pages still show. A signupBonus object appears only when a trial credit is active. Email already registered returns 409, so log in instead.

   import requests
   
   base = 'https://cdn.aisa.one'
   r = requests.post(base + '/api/agent/auth/register', json={
       'email': 'mybot@example.com',
       'password': 'YOUR_PASSWORD',
       'name': 'MyBot',
       'crawlerType': 'custom'
   }).json()
   
   session = r['token']
   crawler = r['crawler']['crawlerToken']
   print(r['crawler']['balance'])
   print(crawler[:18] + '...')

   S1 · esa.aisa.oneS2 · esa.aisa.one

2. Send the crawler token when crawling

   Request publisher URLs with the crawler token in X-Crawler-Token and your genuine User-Agent. Never put the crawler token in Authorization: Bearer, because that header is reserved for the gateway layer and the request is treated as unauthenticated (402 invalid_token). A paid 200 may carry the receipt headers X-AISA-Charged and X-AISA-Charge-Amount. Denials never charge, and one request is at most one charge.

   curl -s -i https://PUBLISHER-SITE/research/report.data.json \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \
     -A 'MyBot/1.0 (+https://example.com/bot)'

   S1 · esa.aisa.oneS2 · esa.aisa.oneS3 · esa.aisa.one

3. Fetch your real balance

   Call GET /api/agent/balance with the same crawler token you crawl with. No email, password or session is needed, and it cannot drift to the wrong account. Read crawler.balance and crawler.totalSpent from the response. Always fetch before reporting a balance, and before and after a top-up, because a remembered value goes stale the moment you spend. Keep using one account throughout; never re-register mid-task.

   curl -s https://cdn.aisa.one/api/agent/balance \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'

   S1 · esa.aisa.oneS2 · esa.aisa.oneS4 · esa.aisa.one

4. Create a top-up link for your user

   Top-ups use the session token, not the crawler token. Send it as Authorization: Bearer with a JSON body holding amount; the minimum is 0.50 USD. The response contains checkout_url, a Stripe page. Completing payment needs a human, so show the link to your user and wait. Never call POST /api/agent/balance/add; it exists only for operators seeding test accounts. Session tokens expire, so log in again to refresh.

   import requests
   
   r = requests.post(
       'https://cdn.aisa.one/api/agent/topup/create-session',
       headers={'Authorization': 'Bearer YOUR_SESSION_TOKEN'},
       json={'amount': 5.00}
   )
   print(r.json()['checkout_url'])

   S2 · esa.aisa.oneS3 · esa.aisa.oneS4 · esa.aisa.one

5. Read pricing modes before you spend

   Publishers price per URL directory. Many expose /pricing.json, a map of directory to price and mode (pay, allow or block), and describe tiers in /llms.txt. Both are free to fetch. HTML pages and .md summaries are usually free, while deep files such as .data.json and .full.md are priced. Typical prices run $0.005–$0.02 per resource, and fetching a priced resource twice charges twice. Read the free summaries to decide what is worth buying.

   curl -s https://PUBLISHER-SITE/pricing.json
   curl -s https://PUBLISHER-SITE/llms.txt

   S2 · esa.aisa.oneS3 · esa.aisa.oneS4 · esa.aisa.one

6. Handle 402 and 403 responses

   Denial bodies are JSON, so check error first. For insufficient_balance, crawler.shortfall says how much more you need; top up and retry. For invalid_token, fix the header and retry. The states price_required, price_mismatch and price_too_high are price negotiation; the quote is in price and currency, and the gateway normally handles them. For access_blocked, money cannot unlock it: skip that URL and pivot to the site's pay-mode directories.

   import requests
   
   r = requests.get('https://PUBLISHER-SITE/research/report.data.json', headers={
       'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN',
       'User-Agent': 'MyBot/1.0 (+https://example.com/bot)'
   })
   if r.status_code == 402:
       body = r.json()
       err = body['error']
       if err == 'insufficient_balance':
           print(body['crawler']['shortfall'])
       elif err in ('price_required', 'price_mismatch', 'price_too_high'):
           print(body['price'], body['currency'])
   elif r.status_code == 403:
       print('blocked by publisher, try pay-mode directories')

   S2 · esa.aisa.oneS3 · esa.aisa.one

Crawler token vs session token

Common pitfalls

• Sending the crawler token in Authorization: Bearer gets the request treated as unauthenticated, returning 402 invalid_token. Use X-Crawler-Token.
• Quoting a balance from memory goes stale after any spend. Spent $0.04 from a balance of $0.06, then topped up $100, the answer is $100.06.
• Do not retry or top up for access_blocked. It is a publisher policy choice, and payment does not unlock it.
• Do not print the full crawler token in logs or reports; show a prefix like aisa_crawler_9a4d... instead.

Key takeaways

• Crawler token goes in X-Crawler-Token for crawling; session token goes in Authorization: Bearer for account APIs only.
• Fetch the real balance with GET /api/agent/balance before and after any top-up, and never quote it from memory.
• Read /pricing.json and 402 bodies for prices; price states are normally handled by the gateway.

FAQ

How this page is made. Written from 4 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.