Building a Pay-Per-Crawl Client for Your Agent

This guide is for developers who want their agent to fetch paid publisher content on the AISA network without hand-written special cases. You will build a small client with four parts: register-or-login, a crawl call that sends the token header and your real User-Agent, a receipt reader for the charge headers, and a denial handler that branches on error codes. Python and Node.js versions are shown because the pattern is the same in any stack. Billing happens at the publisher's edge, so your client only sends the token and reads what comes back.

How the flow works

Before you start

• An email and password for the crawler account
• An HTTP client library in your agent's language
• A public page describing your bot, for the User-Agent

Step by step

1. Register or log in for a token

   Post to /api/agent/auth/register first. If the email already exists the API answers 409, so fall back to /api/agent/auth/login. Both responses carry crawler.crawlerToken, which you crawl with, and a session token under token, which is only for account APIs. You will send the crawler token in the X-Crawler-Token header, which replaces the earlier name X-AISA-Crawler-Token. Store it securely and never log the full value.

   import requests
   
   BASE = 'https://cdn.aisa.one'
   UA = 'MyBot/1.0 (+https://example.com/bot)'
   
   def get_account(email, password):
       body = {'email': email, 'password': password,
               'name': 'MyBot', 'crawlerType': 'custom'}
       r = requests.post(BASE + '/api/agent/auth/register', json=body)
       if r.status_code == 409:
           r = requests.post(BASE + '/api/agent/auth/login',
                             json={'email': email, 'password': password})
       r.raise_for_status()
       return r.json()
   
   data = get_account('mybot@example.com', 'MySecurePass123')
   token = data['crawler']['crawlerToken']
   session = data['token']
   bonus = data.get('signupBonus')

   S1 · esa.aisa.oneS2 · esa.aisa.oneS3 · esa.aisa.one

2. Send the crawl request with honest identity

   Request the publisher URL with two headers: X-Crawler-Token and your genuine User-Agent. The gateway passes your original User-Agent through for identification and audit, so do not claim another bot's identity. Never put the crawler token in Authorization: Bearer, because that header is reserved for the gateway layer and the request is treated as unauthenticated. Priced resources answer 402 if the token header is missing.

   def crawl(url, token):
       return requests.get(url, headers={
           'X-Crawler-Token': token,
           'User-Agent': UA,
       })
   
   resp = crawl('https://PUBLISHER-SITE/research/report.data.json', token)
   print(resp.status_code)

   S1 · esa.aisa.oneS2 · esa.aisa.one

3. Read the charge receipt headers

   On a paid 200, gateways may attach X-AISA-Charged and X-AISA-Charge-Amount. Treat them as optional: a free-tier 200 costs nothing and may carry neither header. Log what you read next to the URL so your user can see what each fetch cost. One request from you is at most one charge, and fetching the same priced file twice pays twice, so cache what you buy.

   def receipt(resp):
       charged = resp.headers.get('X-AISA-Charged') == 'true'
       amount = resp.headers.get('X-AISA-Charge-Amount')
       return {'charged': charged, 'amount': amount}
   
   if resp.status_code == 200:
       print(receipt(resp))

   S2 · esa.aisa.oneS3 · esa.aisa.one

4. Branch on denial error codes

   A 402 or 403 body is self-describing JSON, so check error first. invalid_token means register or fix the header. insufficient_balance carries crawler.shortfall, which you report to your user. access_blocked is the publisher's policy choice: do not retry or top up, skip that URL and move to purchasable directories listed in /pricing.json. Price states put the quote in price and currency. Denials never charge.

   def handle(resp):
       if resp.status_code == 200:
           return {'action': 'use content', 'receipt': receipt(resp)}
       denial = resp.json()
       err = denial.get('error')
       if err == 'invalid_token':
           return {'action': 'register or fix the token header',
                   'guidance': denial.get('guidance')}
       if err == 'insufficient_balance':
           return {'action': 'ask the user to top up',
                   'shortfall': denial['crawler']['shortfall']}
       if err == 'access_blocked':
           return {'action': 'skip this URL', 'next': '/pricing.json'}
       return {'action': 'read the quote',
               'price': denial.get('price'),
               'currency': denial.get('currency')}

   S2 · esa.aisa.oneS4 · agentflare.org

5. Mirror the same client in Node.js

   The pattern carries over unchanged: register, fall back to login on 409, crawl with the token header and your User-Agent, then return either content with receipt headers or the denial error and guidance. Keep the return shape identical across languages so the layer above your client, whether an agent planner or a job queue, never needs to know which runtime made the call.

   const BASE = 'https://cdn.aisa.one';
   const UA = 'MyBot/1.0 (+https://example.com/bot)';
   
   const post = (path, body) => fetch(BASE + path, {
     method: 'POST',
     headers: { 'Content-Type': 'application/json' },
     body: JSON.stringify(body),
   });
   
   async function getToken(email, password) {
     let r = await post('/api/agent/auth/register',
       { email, password, name: 'MyBot', crawlerType: 'custom' });
     if (r.status === 409) r = await post('/api/agent/auth/login', { email, password });
     return (await r.json()).crawler.crawlerToken;
   }
   
   async function crawl(url, token) {
     const r = await fetch(url, {
       headers: { 'X-Crawler-Token': token, 'User-Agent': UA },
     });
     if (r.status === 200) {
       return { ok: true, body: await r.text(),
         charged: r.headers.get('X-AISA-Charged'),
         amount: r.headers.get('X-AISA-Charge-Amount') };
     }
     const d = await r.json();
     return { ok: false, status: r.status, error: d.error, guidance: d.guidance };
   }

   S1 · esa.aisa.oneS2 · esa.aisa.one

6. Fetch the balance and hand off top-ups

   Always fetch the balance with GET /api/agent/balance using the same crawler token; never quote it from memory. When you hit insufficient_balance, create a Stripe checkout session with the session token and give the checkout_url to your user. The minimum amount is 0.50 USD. An agent cannot complete the payment itself, so wait for the human, then retry the same URL.

   def balance(token):
       r = requests.get(BASE + '/api/agent/balance',
                        headers={'X-Crawler-Token': token})
       return r.json()['crawler']['balance']
   
   def checkout_url(session, amount):
       r = requests.post(BASE + '/api/agent/topup/create-session',
                         json={'amount': amount},
                         headers={'Authorization': 'Bearer ' + session})
       return r.json()['checkout_url']
   
   print(balance(token))
   # on insufficient_balance: show checkout_url(session, 5.00) to the user

   S2 · esa.aisa.oneS3 · esa.aisa.one

Outcomes and what the client does

Common pitfalls

• Sending the crawler token in Authorization: Bearer gets the request treated as unauthenticated and answered with 402 invalid_token. Use X-Crawler-Token only.
• Older AISA documentation pages still show a longer header name. Send X-Crawler-Token as the Open Verify specification says.
• Never call POST /api/agent/balance/add to fund yourself. It exists only for a human operator pre-seeding a test account; real top-ups go through Stripe checkout.
• Do not retry or top up after 403 access_blocked. Payment does not unlock it, so skip that URL and continue with other content.

Key takeaways

• Your whole crawl contract is the X-Crawler-Token header plus an honest User-Agent; the edge handles authorization and billing.
• Branch on the error field of denial bodies: invalid_token, insufficient_balance and access_blocked each need a different action.
• Fetch the real balance with the crawler token before reporting it, and hand Stripe checkout links to a human.

FAQ

How this page is made. Written from 4 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.