Using AgentFlare Newswire and Markets Data

This guide is for developers and agent builders who want to pull AgentFlare's structured newswire and markets datasets into their own code. You will learn what each .data.json file contains field by field, what each request costs, how the dated archive URLs are named, and how to find valid dates from the newswire index. You will finish with a runnable curl command and a short Python script that fetches a daily AI digest, reads its stories, and then pulls the daily markets snapshot and a single ticker file.

How the flow works

Before you start

• A crawler token registered with AISA
• curl and Python 3 installed
• Basic familiarity with JSON and HTTP headers

Step by step

1. Check which files are paid

   AgentFlare serves index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt for free. The complete datasets, *.data.json and *.full.md, are the paid L2 tier, metered per request. The price map is published at pricing.json: newswire under /news/ is 0.005 USD per request, and markets under /markets/ is 0.01 USD per request. Read the map before you start so your budget matches your crawl size.

   curl -s https://agentflare.org/pricing.json

   S4 · agentflare.orgS5 · agentflare.org

2. List available dates from the index

   Start with the newswire index file. Its data.digests array has one entry per digest, each with date, cluster, headline, url and stories, so you can pick a valid date without guessing. Send your crawler identity in the X-Crawler-Token header, which replaces the earlier name X-AISA-Crawler-Token. Do not put the token in Authorization: Bearer, because that header is reserved for the gateway layer.

   curl -s https://agentflare.org/news/index.data.json \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'

   S1 · esa.aisa.oneS2 · agentflare.org

3. Fetch a dated newswire file

   Dated files follow the pattern /news/YYYY-MM-DD-CLUSTER.data.json, where the cluster is ai, crypto, earnings, macro or markets. Replace YYYY-MM-DD with a date from the index. Each file carries facts, a data object, analysis_md, sources, related and a pricing object with price_usd, method, endpoint, autopay_hint and onboarding. Per-item files use /news/SLUG.data.json.

   curl -i https://agentflare.org/news/YYYY-MM-DD-ai.data.json \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'

   S1 · esa.aisa.oneS2 · agentflare.org

4. Read the stories in Python

   Inside data you will find date, cluster, headline, dek, generated_at and freshness_hours, then stories, watch, takeaways, faq, count and items. Each story has title, summary, why, sources and entities. Each item has title, url, source, date, tickers and iso. The crypto file also fills data.market with symbol, name, last, asOf, d1, d7 and series, plus prediction_markets.

   import json
   import urllib.request
   
   url = 'https://agentflare.org/news/YYYY-MM-DD-ai.data.json'
   req = urllib.request.Request(
       url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'}
   )
   with urllib.request.urlopen(req) as resp:
       doc = json.load(resp)
   
   data = doc['data']
   print(data['headline'])
   for story in data['stories']:
       print(story['title'], story['entities'])
   print(doc['pricing']['price_usd'])

   S1 · esa.aisa.oneS2 · agentflare.org

5. Pull markets snapshots and tickers

   The daily snapshot lives at /markets/YYYY-MM-DD.data.json. Its data holds count, advancers, decliners, sectors (label, value, n) and stocks (ticker, price_usd, day_change_pct, return_6m_pct, market_cap_usd, pe, sector). Single companies use /markets/SLUG.data.json, such as xom, with returns_pct (m1, m3, m6), a long metrics object and company details including cik, sector and industry.

   curl -s https://agentflare.org/markets/YYYY-MM-DD.data.json \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'
   
   curl -s https://agentflare.org/markets/xom.data.json \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'

   S1 · esa.aisa.oneS3 · agentflare.orgS4 · agentflare.org

Newswire and markets files

Common pitfalls

• Sending the crawler token in Authorization: Bearer is wrong; that header is reserved for the gateway layer. Use X-Crawler-Token.
• The placeholder YYYY-MM-DD in URL patterns must be replaced with a real date, which you can read from data.digests[].date in the newswire index.
• In the ai file, data.market, data.policy_rates and data.prediction_markets are empty arrays, while the crypto file fills market and prediction_markets. Do not assume every cluster has the same populated fields.
• Markets files cost 0.01 USD per request, twice the 0.005 USD newswire price, so budget a ticker-by-ticker crawl accordingly.

Key takeaways

• Newswire .data.json files cost 0.005 USD per request; markets .data.json files cost 0.01 USD per request.
• Find valid dates in data.digests of /news/index.data.json, then build YYYY-MM-DD-cluster URLs.
• Always authenticate with the X-Crawler-Token header, never Authorization: Bearer.

FAQ

How this page is made. Written from 5 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.