Getting Started with HTTP 402 Pay-Per-Crawl

This guide is for developers building crawlers or AI agents that run into HTTP 402 Payment Required on publisher sites in the AISA pay-per-crawl network. You will learn what a 402 means, how to register and get a crawler token, how to fund the balance, how to retry the original URL, and how to interpret a 200, 402 or 403 response. By the end, your agent can complete one paid fetch and report what it cost and what balance remains.

How the flow works

Before you start

• Python 3 with the requests library, plus curl
• A URL on an AISA-protected publisher site
• An email address and a password of at least 8 characters

Step by step

1. Recognize what the 402 is telling you

   An HTTP 402 Payment Required response means the publisher prices this URL and the edge found no valid crawler token or not enough balance. It is a protocol signal, not a block. The JSON body has an error field such as invalid_token, an onboarding link and a guidance block with numbered steps. Denials never charge. The token goes in the X-Crawler-Token header, which replaces the earlier name X-AISA-Crawler-Token still shown on older AISA pages. Reproduce the 402 with an anonymous request.

   curl -i 'https://agentflare.org/markets/index.data.json'
   # HTTP/1.1 402 Payment Required
   # body field error: invalid_token
   # body field onboarding: link to the AISA guide

   S1 · esa.aisa.oneS2 · esa.aisa.oneS4 · agentflare.org

2. Register and save your crawler token

   Call POST /api/agent/auth/register once with email, a password of at least 8 characters, name and crawlerType (custom unless you have a real identity). Save crawler.crawlerToken, which looks like aisa_crawler_ followed by 64 hex characters. The response also holds a session token, used only for account APIs. A 409 means the email is already registered, so use POST /api/agent/auth/login instead. If a 402 body names a different registerApi host, prefer that host.

   import requests
   
   BASE = 'https://cdn.aisa.one'
   r = requests.post(BASE + '/api/agent/auth/register', json={
       'email': 'mybot@example.com',
       'password': 'AtLeast8Chars',
       'name': 'My Crawler',
       'crawlerType': 'custom',
   })
   data = r.json()
   token = data['crawler']['crawlerToken']
   print('token prefix:', token[:15] + '...')
   print('balance:', data['crawler']['balance'])
   print('signupBonus:', data.get('signupBonus'))

   S2 · esa.aisa.oneS3 · esa.aisa.oneS5 · esa.aisa.one

3. Fund the balance only if needed

   Check crawler.balance in the register or login response. New accounts may include a trial credit, shown in signupBonus; if it is present, skip to the retry. Otherwise call POST /api/agent/topup/create-session with the session token in Authorization: Bearer and an amount (minimum 0.50 USD). The response carries checkout_url. A human must open it and pay, so an autonomous agent should hand the link to its user and wait.

   import requests
   
   BASE = 'https://cdn.aisa.one'
   r = requests.post(
       BASE + '/api/agent/topup/create-session',
       headers={'Authorization': 'Bearer SESSION_TOKEN'},
       json={'amount': 5.00},
   )
   print('Open this link to pay:', r.json()['checkout_url'])

   S2 · esa.aisa.oneS3 · esa.aisa.one

4. Retry the original URL with your token

   Request the same resource URL again, unchanged, adding the crawler token header and your honest User-Agent. The publisher's edge verifies the token, deducts the per-request price and serves the content in one request. One request means at most one charge, and fetching a priced URL twice charges twice. Do not claim another bot's User-Agent. Never put the crawler token in Authorization: Bearer.

   curl -i 'https://PUBLISHER-SITE/research/report.data.json' \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \
     -A 'MyBot/1.0 (+https://example.com/bot)'

   S1 · esa.aisa.oneS3 · esa.aisa.oneS5 · esa.aisa.one

5. Branch on 200, 402 and 403

   On 200 you have content; a paid resource may include receipt headers X-AISA-Charged and X-AISA-Charge-Amount. On 402 read the error field: invalid_token means fix the header or register, insufficient_balance means top up and retry, and crawler.shortfall says how much more is needed. On 403 access_blocked the publisher withholds the resource from crawlers; paying will not unlock it, so check the site's /pricing.json for purchasable directories.

   import requests
   
   url = 'https://PUBLISHER-SITE/research/report.data.json'
   headers = {
       'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN',
       'User-Agent': 'MyBot/1.0 (+https://example.com/bot)',
   }
   resp = requests.get(url, headers=headers)
   if resp.status_code == 200:
       print('charged:', resp.headers.get('X-AISA-Charged'),
             resp.headers.get('X-AISA-Charge-Amount'))
   elif resp.status_code == 402:
       body = resp.json()
       print(body.get('error'), body.get('message'))
       if body.get('error') == 'insufficient_balance':
           print('shortfall:', body['crawler']['shortfall'])
   elif resp.status_code == 403:
       print('blocked by publisher policy, check pricing.json')

   S2 · esa.aisa.oneS3 · esa.aisa.oneS5 · esa.aisa.one

6. Fetch the real balance and report

   After the fetch, call GET /api/agent/balance with the same crawler token to read the current balance and totalSpent. Do this rather than quoting a remembered figure, because a stored value goes stale as soon as you spend. Then tell your user whether the content was free or paid, the amount charged, and the remaining balance. Show only a token prefix in logs.

   curl -s 'https://cdn.aisa.one/api/agent/balance' \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'
   # response: crawler.balance and crawler.totalSpent

   S1 · esa.aisa.oneS2 · esa.aisa.oneS3 · esa.aisa.one

Outcomes of a crawl request

Common pitfalls

• Sending the crawler token in Authorization: Bearer makes the request count as unauthenticated and returns 402 invalid_token. Use the crawler header only.
• The session token from register or login is for account APIs like top-up only. The crawlerToken is the credential you send when crawling.
• Do not top up or retry after 403 access_blocked. It is a publisher policy choice, and paying does not unlock it.
• Do not quote a remembered balance. Fetch it from GET /api/agent/balance before and after any top-up or paid fetch.

Key takeaways

• A 402 is an offer to pay, not an error: register, get your token, and retry the same URL.
• Send the crawler token only in X-Crawler-Token; the session token is for account APIs.
• Read the error field on every denial: 402 means a payment action, 403 access_blocked means paying will not help.

FAQ

How this page is made. Written from 5 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.