Reading .data.json: A Field Guide for Agents

This guide is for developers and agent builders who want to pull structured AgentFlare data instead of scraping HTML. You will learn which files are paid, how to send a crawler token, and how to read the envelope fields that every .data.json carries. You will also see how the vertical-specific data block differs between Markets files (index, dated, per-ticker) and Newswire files (digest index and cluster files). By the end you can fetch a file, parse the envelope, and branch on the shape of data without guessing field names.

How the flow works

Before you start

• A registered crawler account and its crawler token
• curl and Python 3 with the requests library
• Basic familiarity with JSON and HTTP status codes

Step by step

1. Check which files are paid

   AgentFlare splits content into two tiers. L1_free covers index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt. L2_paid covers every *.data.json and *.full.md file. Each file is metered per request by directory: markets is 0.01 USD, news is 0.005 USD. Read pricing.json once to confirm the rule for the directory you plan to crawl before you automate anything.

   curl -s https://agentflare.org/pricing.json | jq '.directory_pricing_usd, .tiers'

   S4 · agentflare.org

2. Request a paid file anonymously

   Fetch a Markets file with no token to see the gate. The server answers HTTP 402 Payment Required with Cache-Control: no-cache, no-store and a JSON body whose error is invalid_token. The body also carries an onboarding URL, https://esa.aisa.one/agent/guide.html, which is where you register a crawler account and get a token. Treat this 402 as a signal to fix your token, not to retry blindly.

   curl -i https://agentflare.org/markets/index.data.json
   # Expect: HTTP/1.1 402 Payment Required
   # Body error field: invalid_token

   S5 · agentflare.org

3. Fetch with the crawler token

   Send your token in the X-Crawler-Token header, which replaces the earlier name X-AISA-Crawler-Token that older AISA pages still show. Never put the token in Authorization: Bearer, because that header is reserved for the gateway layer. The same request shape works for every file pattern: /markets/index.data.json, /markets/YYYY-MM-DD.data.json, /markets/<slug>.data.json and the /news/ equivalents.

   curl -s \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \
     https://agentflare.org/markets/xom.data.json \
     | jq 'keys'

   S1 · esa.aisa.oneS2 · agentflare.org

4. Parse the shared envelope

   Every file starts with the same top-level fields. The strings are @context, type, tier, title, description, canonical, category, updated and generated_at, plus powered_by. Then come facts (objects with label and value strings), analysis_md, sources (title and url), related (name and url) and a pricing object. Read these first, then hand data to a vertical-specific parser.

   import requests
   
   URL = 'https://agentflare.org/markets/xom.data.json'
   HEADERS = {'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'}
   doc = requests.get(URL, headers=HEADERS).json()
   
   for key in ['type', 'tier', 'title', 'canonical', 'category', 'updated', 'generated_at']:
       print(key, doc[key])
   for fact in doc['facts']:
       print(fact['label'], fact['value'])
   print(doc['pricing']['price_usd'], doc['pricing']['endpoint'])

   S2 · agentflare.org

5. Read the Markets data block

   In Markets, the shape of data depends on the file. The index and dated files hold count, advancers, decliners, sectors and a stocks array with ticker, price_usd, day_change_pct, return_6m_pct, market_cap_usd, pe and sector. A per-ticker file such as xom.data.json instead holds ticker, name, price_usd, day_change_pct, returns_pct (m1, m3, m6), a long metrics object and a company object.

   import requests
   
   HEADERS = {'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'}
   base = 'https://agentflare.org/markets/'
   
   idx = requests.get(base + 'index.data.json', headers=HEADERS).json()['data']
   print(idx['count'], idx['advancers'], idx['decliners'])
   for s in idx['stocks']:
       print(s['ticker'], s['price_usd'], s['day_change_pct'], s['sector'])
   
   xom = requests.get(base + 'xom.data.json', headers=HEADERS).json()['data']
   print(xom['returns_pct']['m6'], xom['metrics']['price_to_earnings_ratio'])
   print(xom['company']['name'], xom['company']['sector'])

   S2 · agentflare.org

6. Read the Newswire data block

   Newswire files use the same envelope with a different data block. The news index holds date and a digests array (date, cluster, headline, url, stories). Cluster files such as YYYY-MM-DD-ai.data.json and YYYY-MM-DD-crypto.data.json hold date, cluster, headline, dek, freshness_hours, stories, watch, takeaways, faq, count and items. Only the crypto file shows typed market and prediction_markets entries.

   import requests
   
   HEADERS = {'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'}
   url = 'https://agentflare.org/news/index.data.json'
   doc = requests.get(url, headers=HEADERS).json()
   
   print(doc['data']['date'])
   for d in doc['data']['digests']:
       print(d['cluster'], d['headline'], d['stories'], d['url'])

   S3 · agentflare.org

Envelope fields shared by every .data.json

Common pitfalls

• Sending the token as Authorization: Bearer is wrong; that header is reserved for the gateway layer. Use X-Crawler-Token only.
• An HTTP 402 with error invalid_token means the token matches no registered crawler. Check the value or register through the onboarding URL.
• Do not reuse one data parser across files. Markets index has data.stocks, while a ticker file has data.metrics and data.company instead.
• Some data arrays are untyped in the news files, for example market and policy_rates in the ai file. Check for items before reading fields.

Key takeaways

• Every .data.json shares one envelope; only the data block changes by vertical and file type.
• Send the token in X-Crawler-Token, never in Authorization: Bearer; the old name X-AISA-Crawler-Token is superseded.
• A 402 response with error invalid_token means fix or register your crawler token before retrying.

FAQ

How this page is made. Written from 5 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.