{ "@context": "https://agentflare.org/schema", "type": "Article", "tier": "L2-full", "title": "Building a Pay-Per-Crawl Client for Your Agent", "description": "A pay-per-crawl client needs only four moves: get a crawler token, send it with an honest User-Agent, read receipt headers, and branch on the error code in…", "canonical": "https://agentflare.org/guides/building-a-pay-per-crawl-client-for-your-agent.html", "category": "guides", "updated": "2026-10-30", "generated_at": "2026-10-30T00:37:02.265Z", "facts": [ { "label": "Track", "value": "For agent developers" }, { "label": "Level", "value": "Intermediate" }, { "label": "Access", "value": "Free / open" }, { "label": "Updated", "value": "2026-10-30" } ], "data": { "topic": "Building a Pay-Per-Crawl Client for Your Agent", "track": "agent", "access": "free", "generated_at": "2026-10-30T00:37:02.265Z", "difficulty": "Intermediate", "tldr": "A pay-per-crawl client needs only four moves: get a crawler token, send it with an honest User-Agent, read receipt headers, and branch on the error code in any 402 or 403 body.", "intro": "This guide is for developers who want their agent to fetch paid publisher content on the AISA network without hand-written special cases. You will build a small client with four parts: register-or-login, a crawl call that sends the token header and your real User-Agent, a receipt reader for the charge headers, and a denial handler that branches on error codes. Python and Node.js versions are shown because the pattern is the same in any stack. Billing happens at the publisher's edge, so your client only sends the token and reads what comes back.", "prerequisites": [ "An email and password for the crawler account", "An HTTP client library in your agent's language", "A public page describing your bot, for the User-Agent" ], "steps": [ { "title": "Register or log in for a token", "body": "Post to /api/agent/auth/register first. If the email already exists the API answers 409, so fall back to /api/agent/auth/login. Both responses carry crawler.crawlerToken, which you crawl with, and a session token under token, which is only for account APIs. You will send the crawler token in the X-Crawler-Token header, which replaces the earlier name X-AISA-Crawler-Token. Store it securely and never log the full value.", "code": { "lang": "python", "text": "import requests\n\nBASE = 'https://cdn.aisa.one'\nUA = 'MyBot/1.0 (+https://example.com/bot)'\n\ndef get_account(email, password):\n body = {'email': email, 'password': password,\n 'name': 'MyBot', 'crawlerType': 'custom'}\n r = requests.post(BASE + '/api/agent/auth/register', json=body)\n if r.status_code == 409:\n r = requests.post(BASE + '/api/agent/auth/login',\n json={'email': email, 'password': password})\n r.raise_for_status()\n return r.json()\n\ndata = get_account('mybot@example.com', 'MySecurePass123')\ntoken = data['crawler']['crawlerToken']\nsession = data['token']\nbonus = data.get('signupBonus')" }, "sources": [ "S1", "S2", "S3" ] }, { "title": "Send the crawl request with honest identity", "body": "Request the publisher URL with two headers: X-Crawler-Token and your genuine User-Agent. The gateway passes your original User-Agent through for identification and audit, so do not claim another bot's identity. Never put the crawler token in Authorization: Bearer, because that header is reserved for the gateway layer and the request is treated as unauthenticated. Priced resources answer 402 if the token header is missing.", "code": { "lang": "python", "text": "def crawl(url, token):\n return requests.get(url, headers={\n 'X-Crawler-Token': token,\n 'User-Agent': UA,\n })\n\nresp = crawl('https://PUBLISHER-SITE/research/report.data.json', token)\nprint(resp.status_code)" }, "sources": [ "S1", "S2" ] }, { "title": "Read the charge receipt headers", "body": "On a paid 200, gateways may attach X-AISA-Charged and X-AISA-Charge-Amount. Treat them as optional: a free-tier 200 costs nothing and may carry neither header. Log what you read next to the URL so your user can see what each fetch cost. One request from you is at most one charge, and fetching the same priced file twice pays twice, so cache what you buy.", "code": { "lang": "python", "text": "def receipt(resp):\n charged = resp.headers.get('X-AISA-Charged') == 'true'\n amount = resp.headers.get('X-AISA-Charge-Amount')\n return {'charged': charged, 'amount': amount}\n\nif resp.status_code == 200:\n print(receipt(resp))" }, "sources": [ "S2", "S3" ] }, { "title": "Branch on denial error codes", "body": "A 402 or 403 body is self-describing JSON, so check error first. invalid_token means register or fix the header. insufficient_balance carries crawler.shortfall, which you report to your user. access_blocked is the publisher's policy choice: do not retry or top up, skip that URL and move to purchasable directories listed in /pricing.json. Price states put the quote in price and currency. Denials never charge.", "code": { "lang": "python", "text": "def handle(resp):\n if resp.status_code == 200:\n return {'action': 'use content', 'receipt': receipt(resp)}\n denial = resp.json()\n err = denial.get('error')\n if err == 'invalid_token':\n return {'action': 'register or fix the token header',\n 'guidance': denial.get('guidance')}\n if err == 'insufficient_balance':\n return {'action': 'ask the user to top up',\n 'shortfall': denial['crawler']['shortfall']}\n if err == 'access_blocked':\n return {'action': 'skip this URL', 'next': '/pricing.json'}\n return {'action': 'read the quote',\n 'price': denial.get('price'),\n 'currency': denial.get('currency')}" }, "sources": [ "S2", "S4" ] }, { "title": "Mirror the same client in Node.js", "body": "The pattern carries over unchanged: register, fall back to login on 409, crawl with the token header and your User-Agent, then return either content with receipt headers or the denial error and guidance. Keep the return shape identical across languages so the layer above your client, whether an agent planner or a job queue, never needs to know which runtime made the call.", "code": { "lang": "javascript", "text": "const BASE = 'https://cdn.aisa.one';\nconst UA = 'MyBot/1.0 (+https://example.com/bot)';\n\nconst post = (path, body) => fetch(BASE + path, {\n method: 'POST',\n headers: { 'Content-Type': 'application/json' },\n body: JSON.stringify(body),\n});\n\nasync function getToken(email, password) {\n let r = await post('/api/agent/auth/register',\n { email, password, name: 'MyBot', crawlerType: 'custom' });\n if (r.status === 409) r = await post('/api/agent/auth/login', { email, password });\n return (await r.json()).crawler.crawlerToken;\n}\n\nasync function crawl(url, token) {\n const r = await fetch(url, {\n headers: { 'X-Crawler-Token': token, 'User-Agent': UA },\n });\n if (r.status === 200) {\n return { ok: true, body: await r.text(),\n charged: r.headers.get('X-AISA-Charged'),\n amount: r.headers.get('X-AISA-Charge-Amount') };\n }\n const d = await r.json();\n return { ok: false, status: r.status, error: d.error, guidance: d.guidance };\n}" }, "sources": [ "S1", "S2" ] }, { "title": "Fetch the balance and hand off top-ups", "body": "Always fetch the balance with GET /api/agent/balance using the same crawler token; never quote it from memory. When you hit insufficient_balance, create a Stripe checkout session with the session token and give the checkout_url to your user. The minimum amount is 0.50 USD. An agent cannot complete the payment itself, so wait for the human, then retry the same URL.", "code": { "lang": "python", "text": "def balance(token):\n r = requests.get(BASE + '/api/agent/balance',\n headers={'X-Crawler-Token': token})\n return r.json()['crawler']['balance']\n\ndef checkout_url(session, amount):\n r = requests.post(BASE + '/api/agent/topup/create-session',\n json={'amount': amount},\n headers={'Authorization': 'Bearer ' + session})\n return r.json()['checkout_url']\n\nprint(balance(token))\n# on insufficient_balance: show checkout_url(session, 5.00) to the user" }, "sources": [ "S2", "S3" ] } ], "reference": { "title": "Outcomes and what the client does", "columns": [ "HTTP", "error", "Cost", "Client action" ], "rows": [ [ "200 (free tier)", "none", "$0", "Use the content" ], [ "200 (paid)", "none", "The per-request price", "Read X-AISA-Charged and X-AISA-Charge-Amount if present" ], [ "402", "invalid_token", "$0", "Register or fix the header, retry the same URL" ], [ "402", "insufficient_balance", "$0", "Report crawler.shortfall, give the user a checkout link, retry" ], [ "402", "price_required / price_mismatch / price_too_high", "$0", "Read the quote in price and currency" ], [ "403", "access_blocked", "$0", "Skip the URL, pivot to pay-mode directories in /pricing.json" ] ], "sources": [ "S2" ] }, "pitfalls": [ { "text": "Sending the crawler token in Authorization: Bearer gets the request treated as unauthenticated and answered with 402 invalid_token. Use X-Crawler-Token only.", "source": "S1" }, { "text": "Older AISA documentation pages still show a longer header name. Send X-Crawler-Token as the Open Verify specification says.", "source": "S1" }, { "text": "Never call POST /api/agent/balance/add to fund yourself. It exists only for a human operator pre-seeding a test account; real top-ups go through Stripe checkout.", "source": "S3" }, { "text": "Do not retry or top up after 403 access_blocked. Payment does not unlock it, so skip that URL and continue with other content.", "source": "S2" } ], "takeaways": [ "Your whole crawl contract is the X-Crawler-Token header plus an honest User-Agent; the edge handles authorization and billing.", "Branch on the error field of denial bodies: invalid_token, insufficient_balance and access_blocked each need a different action.", "Fetch the real balance with the crawler token before reporting it, and hand Stripe checkout links to a human." ], "faq": [ { "q": "Which header carries the crawler token?", "a": "X-Crawler-Token. Never use Authorization: Bearer for it. That header is reserved for the gateway layer and the request would be treated as unauthenticated. The session token is a separate credential used only for account APIs such as top-up." }, { "q": "Am I charged when a request is denied?", "a": "No. Denials with 402 or 403 cost $0, and price discovery through a 402 body is free. Only a granted fetch of a priced resource deducts its per-request price, once per request." }, { "q": "What should my agent do on a 403 access_blocked?", "a": "Do not retry and do not top up; the publisher withholds that resource from crawlers. Skip the URL, check the site's /pricing.json for purchasable pay-mode directories, and explain the difference to your user." }, { "q": "Can my agent top up its own balance?", "a": "Not by itself. Create a checkout session with POST /api/agent/topup/create-session, show the returned checkout_url to your user, and wait for them to pay. Then fetch the balance again and retry the URL." } ], "sources": [ { "title": "Header name: X-Crawler-Token", "url": "https://esa.aisa.one/cdn/open-verify.html", "domain": "esa.aisa.one" }, { "title": "AISA Agent Integration Guide β€” API reference", "url": "https://esa.aisa.one/agent/docs.html", "domain": "esa.aisa.one" }, { "title": "AISA agent skill (SKILL.md)", "url": "https://esa.aisa.one/agent/skill.md", "domain": "esa.aisa.one" }, { "title": "Live capture: anonymous request to a paid AgentFlare file", "url": "https://agentflare.org/markets/index.data.json", "domain": "agentflare.org" } ] }, "analysis_md": "A pay-per-crawl client needs only four moves: get a crawler token, send it with an honest User-Agent, read receipt headers, and branch on the error code in any 402 or 403 body.\n\nThis guide is for developers who want their agent to fetch paid publisher content on the AISA network without hand-written special cases. You will build a small client with four parts: register-or-login, a crawl call that sends the token header and your real User-Agent, a receipt reader for the charge headers, and a denial handler that branches on error codes. Python and Node.js versions are shown because the pattern is the same in any stack. Billing happens at the publisher's edge, so your client only sends the token and reads what comes back.\n\n## Before you start\n\n- An email and password for the crawler account\n- An HTTP client library in your agent's language\n- A public page describing your bot, for the User-Agent\n\n## Step by step\n\n### 1. Register or log in for a token\n\nPost to /api/agent/auth/register first. If the email already exists the API answers 409, so fall back to /api/agent/auth/login. Both responses carry crawler.crawlerToken, which you crawl with, and a session token under token, which is only for account APIs. You will send the crawler token in the X-Crawler-Token header, which replaces the earlier name X-AISA-Crawler-Token. Store it securely and never log the full value.\n\n```python\nimport requests\n\nBASE = 'https://cdn.aisa.one'\nUA = 'MyBot/1.0 (+https://example.com/bot)'\n\ndef get_account(email, password):\n body = {'email': email, 'password': password,\n 'name': 'MyBot', 'crawlerType': 'custom'}\n r = requests.post(BASE + '/api/agent/auth/register', json=body)\n if r.status_code == 409:\n r = requests.post(BASE + '/api/agent/auth/login',\n json={'email': email, 'password': password})\n r.raise_for_status()\n return r.json()\n\ndata = get_account('mybot@example.com', 'MySecurePass123')\ntoken = data['crawler']['crawlerToken']\nsession = data['token']\nbonus = data.get('signupBonus')\n```\n\n### 2. Send the crawl request with honest identity\n\nRequest the publisher URL with two headers: X-Crawler-Token and your genuine User-Agent. The gateway passes your original User-Agent through for identification and audit, so do not claim another bot's identity. Never put the crawler token in Authorization: Bearer, because that header is reserved for the gateway layer and the request is treated as unauthenticated. Priced resources answer 402 if the token header is missing.\n\n```python\ndef crawl(url, token):\n return requests.get(url, headers={\n 'X-Crawler-Token': token,\n 'User-Agent': UA,\n })\n\nresp = crawl('https://PUBLISHER-SITE/research/report.data.json', token)\nprint(resp.status_code)\n```\n\n### 3. Read the charge receipt headers\n\nOn a paid 200, gateways may attach X-AISA-Charged and X-AISA-Charge-Amount. Treat them as optional: a free-tier 200 costs nothing and may carry neither header. Log what you read next to the URL so your user can see what each fetch cost. One request from you is at most one charge, and fetching the same priced file twice pays twice, so cache what you buy.\n\n```python\ndef receipt(resp):\n charged = resp.headers.get('X-AISA-Charged') == 'true'\n amount = resp.headers.get('X-AISA-Charge-Amount')\n return {'charged': charged, 'amount': amount}\n\nif resp.status_code == 200:\n print(receipt(resp))\n```\n\n### 4. Branch on denial error codes\n\nA 402 or 403 body is self-describing JSON, so check error first. invalid_token means register or fix the header. insufficient_balance carries crawler.shortfall, which you report to your user. access_blocked is the publisher's policy choice: do not retry or top up, skip that URL and move to purchasable directories listed in /pricing.json. Price states put the quote in price and currency. Denials never charge.\n\n```python\ndef handle(resp):\n if resp.status_code == 200:\n return {'action': 'use content', 'receipt': receipt(resp)}\n denial = resp.json()\n err = denial.get('error')\n if err == 'invalid_token':\n return {'action': 'register or fix the token header',\n 'guidance': denial.get('guidance')}\n if err == 'insufficient_balance':\n return {'action': 'ask the user to top up',\n 'shortfall': denial['crawler']['shortfall']}\n if err == 'access_blocked':\n return {'action': 'skip this URL', 'next': '/pricing.json'}\n return {'action': 'read the quote',\n 'price': denial.get('price'),\n 'currency': denial.get('currency')}\n```\n\n### 5. Mirror the same client in Node.js\n\nThe pattern carries over unchanged: register, fall back to login on 409, crawl with the token header and your User-Agent, then return either content with receipt headers or the denial error and guidance. Keep the return shape identical across languages so the layer above your client, whether an agent planner or a job queue, never needs to know which runtime made the call.\n\n```javascript\nconst BASE = 'https://cdn.aisa.one';\nconst UA = 'MyBot/1.0 (+https://example.com/bot)';\n\nconst post = (path, body) => fetch(BASE + path, {\n method: 'POST',\n headers: { 'Content-Type': 'application/json' },\n body: JSON.stringify(body),\n});\n\nasync function getToken(email, password) {\n let r = await post('/api/agent/auth/register',\n { email, password, name: 'MyBot', crawlerType: 'custom' });\n if (r.status === 409) r = await post('/api/agent/auth/login', { email, password });\n return (await r.json()).crawler.crawlerToken;\n}\n\nasync function crawl(url, token) {\n const r = await fetch(url, {\n headers: { 'X-Crawler-Token': token, 'User-Agent': UA },\n });\n if (r.status === 200) {\n return { ok: true, body: await r.text(),\n charged: r.headers.get('X-AISA-Charged'),\n amount: r.headers.get('X-AISA-Charge-Amount') };\n }\n const d = await r.json();\n return { ok: false, status: r.status, error: d.error, guidance: d.guidance };\n}\n```\n\n### 6. Fetch the balance and hand off top-ups\n\nAlways fetch the balance with GET /api/agent/balance using the same crawler token; never quote it from memory. When you hit insufficient_balance, create a Stripe checkout session with the session token and give the checkout_url to your user. The minimum amount is 0.50 USD. An agent cannot complete the payment itself, so wait for the human, then retry the same URL.\n\n```python\ndef balance(token):\n r = requests.get(BASE + '/api/agent/balance',\n headers={'X-Crawler-Token': token})\n return r.json()['crawler']['balance']\n\ndef checkout_url(session, amount):\n r = requests.post(BASE + '/api/agent/topup/create-session',\n json={'amount': amount},\n headers={'Authorization': 'Bearer ' + session})\n return r.json()['checkout_url']\n\nprint(balance(token))\n# on insufficient_balance: show checkout_url(session, 5.00) to the user\n```\n\n## Outcomes and what the client does\n\n| HTTP | error | Cost | Client action |\n| --- | --- | --- | --- |\n| 200 (free tier) | none | $0 | Use the content |\n| 200 (paid) | none | The per-request price | Read X-AISA-Charged and X-AISA-Charge-Amount if present |\n| 402 | invalid_token | $0 | Register or fix the header, retry the same URL |\n| 402 | insufficient_balance | $0 | Report crawler.shortfall, give the user a checkout link, retry |\n| 402 | price_required / price_mismatch / price_too_high | $0 | Read the quote in price and currency |\n| 403 | access_blocked | $0 | Skip the URL, pivot to pay-mode directories in /pricing.json |\n\n## Common pitfalls\n\n- Sending the crawler token in Authorization: Bearer gets the request treated as unauthenticated and answered with 402 invalid_token. Use X-Crawler-Token only.\n- Older AISA documentation pages still show a longer header name. Send X-Crawler-Token as the Open Verify specification says.\n- Never call POST /api/agent/balance/add to fund yourself. It exists only for a human operator pre-seeding a test account; real top-ups go through Stripe checkout.\n- Do not retry or top up after 403 access_blocked. Payment does not unlock it, so skip that URL and continue with other content.\n\n## Key takeaways\n\n- Your whole crawl contract is the X-Crawler-Token header plus an honest User-Agent; the edge handles authorization and billing.\n- Branch on the error field of denial bodies: invalid_token, insufficient_balance and access_blocked each need a different action.\n- Fetch the real balance with the crawler token before reporting it, and hand Stripe checkout links to a human.\n\n## FAQ\n\n**Which header carries the crawler token?**\nX-Crawler-Token. Never use Authorization: Bearer for it. That header is reserved for the gateway layer and the request would be treated as unauthenticated. The session token is a separate credential used only for account APIs such as top-up.\n\n**Am I charged when a request is denied?**\nNo. Denials with 402 or 403 cost $0, and price discovery through a 402 body is free. Only a granted fetch of a priced resource deducts its per-request price, once per request.\n\n**What should my agent do on a 403 access_blocked?**\nDo not retry and do not top up; the publisher withholds that resource from crawlers. Skip the URL, check the site's /pricing.json for purchasable pay-mode directories, and explain the difference to your user.\n\n**Can my agent top up its own balance?**\nNot by itself. Create a checkout session with POST /api/agent/topup/create-session, show the returned checkout_url to your user, and wait for them to pay. Then fetch the balance again and retry the URL.\n", "sources": [ { "title": "Header name: X-Crawler-Token", "url": "https://esa.aisa.one/cdn/open-verify.html" }, { "title": "AISA Agent Integration Guide β€” API reference", "url": "https://esa.aisa.one/agent/docs.html" }, { "title": "AISA agent skill (SKILL.md)", "url": "https://esa.aisa.one/agent/skill.md" }, { "title": "Live capture: anonymous request to a paid AgentFlare file", "url": "https://agentflare.org/markets/index.data.json" } ], "related": [ { "name": "Getting Started with HTTP 402 Pay-Per-Crawl", "url": "https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html" }, { "name": "Crawler Tokens & Auto-Pay: A How-To", "url": "https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html" }, { "name": "Handling 402 and 403 Responses: An Error Reference for Agents", "url": "https://agentflare.org/guides/handling-402-and-403-responses-an-error-reference-for-agents.html" }, { "name": "Budgeting a Paid Crawl: Discover, Survey, Then Buy", "url": "https://agentflare.org/guides/budgeting-a-paid-crawl-discover-survey-then-buy.html" }, { "name": "Pricing Content by Directory: Best Practices", "url": "https://agentflare.org/guides/pricing-content-by-directory-best-practices.html" }, { "name": "robots.txt, llms.txt & Sitemaps for AI Crawlers", "url": "https://agentflare.org/guides/robotstxt-llmstxt-sitemaps-for-ai-crawlers.html" } ], "pricing": { "price_usd": 0, "method": "402", "endpoint": "https://esa.aisa.one/api/v1/access/verify", "autopay_hint": "set crawlerAutoPrice=true with the X-Crawler-Token header", "onboarding": "https://esa.aisa.one/agent/guide.html" }, "powered_by": "AISA β€” agent-native search, settlement & delivery (https://aisa.one)" }