{
  "@context": "https://agentflare.org/schema",
  "type": "Article",
  "tier": "L2-full",
  "title": "Using AgentFlare Newswire and Markets Data",
  "description": "AgentFlare newswire files cost 0.005 USD per request and markets files cost 0.01 USD per request, both as .data.json. This guide lists the fields in each…",
  "canonical": "https://agentflare.org/guides/using-agentflare-newswire-and-markets-data.html",
  "category": "guides",
  "updated": "2026-10-27",
  "generated_at": "2026-10-27T00:37:01.404Z",
  "facts": [
    {
      "label": "Track",
      "value": "Using AgentFlare data"
    },
    {
      "label": "Level",
      "value": "Beginner"
    },
    {
      "label": "Access",
      "value": "Free / open"
    },
    {
      "label": "Updated",
      "value": "2026-10-27"
    }
  ],
  "data": {
    "topic": "Using AgentFlare Newswire and Markets Data",
    "track": "data",
    "access": "free",
    "generated_at": "2026-10-27T00:37:01.404Z",
    "difficulty": "Beginner",
    "tldr": "AgentFlare newswire files cost 0.005 USD per request and markets files cost 0.01 USD per request, both as .data.json. This guide lists the fields in each file, the dated URL patterns, and a working fetch example.",
    "intro": "This guide is for developers and agent builders who want to pull AgentFlare's structured newswire and markets datasets into their own code. You will learn what each .data.json file contains field by field, what each request costs, how the dated archive URLs are named, and how to find valid dates from the newswire index. You will finish with a runnable curl command and a short Python script that fetches a daily AI digest, reads its stories, and then pulls the daily markets snapshot and a single ticker file.",
    "prerequisites": [
      "A crawler token registered with AISA",
      "curl and Python 3 installed",
      "Basic familiarity with JSON and HTTP headers"
    ],
    "steps": [
      {
        "title": "Check which files are paid",
        "body": "AgentFlare serves index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt for free. The complete datasets, *.data.json and *.full.md, are the paid L2 tier, metered per request. The price map is published at pricing.json: newswire under /news/ is 0.005 USD per request, and markets under /markets/ is 0.01 USD per request. Read the map before you start so your budget matches your crawl size.",
        "code": {
          "lang": "bash",
          "text": "curl -s https://agentflare.org/pricing.json"
        },
        "sources": [
          "S4",
          "S5"
        ]
      },
      {
        "title": "List available dates from the index",
        "body": "Start with the newswire index file. Its data.digests array has one entry per digest, each with date, cluster, headline, url and stories, so you can pick a valid date without guessing. Send your crawler identity in the X-Crawler-Token header, which replaces the earlier name X-AISA-Crawler-Token. Do not put the token in Authorization: Bearer, because that header is reserved for the gateway layer.",
        "code": {
          "lang": "bash",
          "text": "curl -s https://agentflare.org/news/index.data.json \\\n  -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'"
        },
        "sources": [
          "S1",
          "S2"
        ]
      },
      {
        "title": "Fetch a dated newswire file",
        "body": "Dated files follow the pattern /news/YYYY-MM-DD-CLUSTER.data.json, where the cluster is ai, crypto, earnings, macro or markets. Replace YYYY-MM-DD with a date from the index. Each file carries facts, a data object, analysis_md, sources, related and a pricing object with price_usd, method, endpoint, autopay_hint and onboarding. Per-item files use /news/SLUG.data.json.",
        "code": {
          "lang": "bash",
          "text": "curl -i https://agentflare.org/news/YYYY-MM-DD-ai.data.json \\\n  -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'"
        },
        "sources": [
          "S1",
          "S2"
        ]
      },
      {
        "title": "Read the stories in Python",
        "body": "Inside data you will find date, cluster, headline, dek, generated_at and freshness_hours, then stories, watch, takeaways, faq, count and items. Each story has title, summary, why, sources and entities. Each item has title, url, source, date, tickers and iso. The crypto file also fills data.market with symbol, name, last, asOf, d1, d7 and series, plus prediction_markets.",
        "code": {
          "lang": "python",
          "text": "import json\nimport urllib.request\n\nurl = 'https://agentflare.org/news/YYYY-MM-DD-ai.data.json'\nreq = urllib.request.Request(\n    url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'}\n)\nwith urllib.request.urlopen(req) as resp:\n    doc = json.load(resp)\n\ndata = doc['data']\nprint(data['headline'])\nfor story in data['stories']:\n    print(story['title'], story['entities'])\nprint(doc['pricing']['price_usd'])"
        },
        "sources": [
          "S1",
          "S2"
        ]
      },
      {
        "title": "Pull markets snapshots and tickers",
        "body": "The daily snapshot lives at /markets/YYYY-MM-DD.data.json. Its data holds count, advancers, decliners, sectors (label, value, n) and stocks (ticker, price_usd, day_change_pct, return_6m_pct, market_cap_usd, pe, sector). Single companies use /markets/SLUG.data.json, such as xom, with returns_pct (m1, m3, m6), a long metrics object and company details including cik, sector and industry.",
        "code": {
          "lang": "bash",
          "text": "curl -s https://agentflare.org/markets/YYYY-MM-DD.data.json \\\n  -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'\n\ncurl -s https://agentflare.org/markets/xom.data.json \\\n  -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'"
        },
        "sources": [
          "S1",
          "S3",
          "S4"
        ]
      }
    ],
    "reference": {
      "title": "Newswire and markets files",
      "columns": [
        "File pattern",
        "Price per request",
        "Key data fields"
      ],
      "rows": [
        [
          "/news/index.data.json",
          "0.005 USD",
          "data.date, data.digests[] (date, cluster, headline, url, stories)"
        ],
        [
          "/news/YYYY-MM-DD-ai.data.json",
          "0.005 USD",
          "data.headline, data.dek, data.stories[], data.items[], data.watch[], data.faq[]"
        ],
        [
          "/news/YYYY-MM-DD-crypto.data.json",
          "0.005 USD",
          "Same as ai, plus data.market[] (symbol, last, d1, d7, series) and data.prediction_markets[]"
        ],
        [
          "/news/SLUG.data.json",
          "0.005 USD",
          "One file per item"
        ],
        [
          "/markets/index.data.json",
          "0.01 USD",
          "data.count, data.advancers, data.decliners, data.sectors[], data.stocks[]"
        ],
        [
          "/markets/YYYY-MM-DD.data.json",
          "0.01 USD",
          "Same fields as the markets index, for one date"
        ],
        [
          "/markets/xom.data.json",
          "0.01 USD",
          "data.returns_pct (m1, m3, m6), data.metrics, data.company"
        ]
      ],
      "sources": [
        "S2",
        "S3",
        "S4"
      ]
    },
    "pitfalls": [
      {
        "text": "Sending the crawler token in Authorization: Bearer is wrong; that header is reserved for the gateway layer. Use X-Crawler-Token.",
        "source": "S1"
      },
      {
        "text": "The placeholder YYYY-MM-DD in URL patterns must be replaced with a real date, which you can read from data.digests[].date in the newswire index.",
        "source": "S2"
      },
      {
        "text": "In the ai file, data.market, data.policy_rates and data.prediction_markets are empty arrays, while the crypto file fills market and prediction_markets. Do not assume every cluster has the same populated fields.",
        "source": "S2"
      },
      {
        "text": "Markets files cost 0.01 USD per request, twice the 0.005 USD newswire price, so budget a ticker-by-ticker crawl accordingly.",
        "source": "S4"
      }
    ],
    "takeaways": [
      "Newswire .data.json files cost 0.005 USD per request; markets .data.json files cost 0.01 USD per request.",
      "Find valid dates in data.digests of /news/index.data.json, then build YYYY-MM-DD-cluster URLs.",
      "Always authenticate with the X-Crawler-Token header, never Authorization: Bearer."
    ],
    "faq": [
      {
        "q": "How much does an AgentFlare newswire request cost?",
        "a": "Files under /news/ ending in .data.json or .full.md are priced at 0.005 USD per request, billed per-request. Markets files under /markets/ are 0.01 USD per request. Index pages, .html articles and .md teasers stay free."
      },
      {
        "q": "Which header carries my crawler token?",
        "a": "Use X-Crawler-Token, as defined in the Open Verify specification. It replaces the older name X-AISA-Crawler-Token still shown on some AISA pages. Never send the token in Authorization: Bearer."
      },
      {
        "q": "What clusters exist for dated newswire files?",
        "a": "Dated newswire files are named YYYY-MM-DD-ai, YYYY-MM-DD-crypto, YYYY-MM-DD-earnings, YYYY-MM-DD-macro and YYYY-MM-DD-markets, each ending in .data.json. Per-item files use a slug instead of a date."
      },
      {
        "q": "What does a markets ticker file include?",
        "a": "A file such as /markets/xom.data.json has price_usd, day_change_pct, returns_pct (m1, m3, m6), a metrics object with valuation, margin, growth and liquidity ratios, and company details: ticker, name, cik, sector and industry."
      }
    ],
    "sources": [
      {
        "title": "Header name: X-Crawler-Token",
        "url": "https://esa.aisa.one/cdn/open-verify.html",
        "domain": "esa.aisa.one"
      },
      {
        "title": "AgentFlare Newswire .data.json field reference",
        "url": "https://agentflare.org/news/index.html",
        "domain": "agentflare.org"
      },
      {
        "title": "AgentFlare Markets .data.json field reference",
        "url": "https://agentflare.org/markets/index.html",
        "domain": "agentflare.org"
      },
      {
        "title": "AgentFlare pricing.json (directory price map)",
        "url": "https://agentflare.org/pricing.json",
        "domain": "agentflare.org"
      },
      {
        "title": "AgentFlare llms.txt (header and structure)",
        "url": "https://agentflare.org/llms.txt",
        "domain": "agentflare.org"
      }
    ]
  },
  "analysis_md": "AgentFlare newswire files cost 0.005 USD per request and markets files cost 0.01 USD per request, both as .data.json. This guide lists the fields in each file, the dated URL patterns, and a working fetch example.\n\nThis guide is for developers and agent builders who want to pull AgentFlare's structured newswire and markets datasets into their own code. You will learn what each .data.json file contains field by field, what each request costs, how the dated archive URLs are named, and how to find valid dates from the newswire index. You will finish with a runnable curl command and a short Python script that fetches a daily AI digest, reads its stories, and then pulls the daily markets snapshot and a single ticker file.\n\n## Before you start\n\n- A crawler token registered with AISA\n- curl and Python 3 installed\n- Basic familiarity with JSON and HTTP headers\n\n## Step by step\n\n### 1. Check which files are paid\n\nAgentFlare serves index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt for free. The complete datasets, *.data.json and *.full.md, are the paid L2 tier, metered per request. The price map is published at pricing.json: newswire under /news/ is 0.005 USD per request, and markets under /markets/ is 0.01 USD per request. Read the map before you start so your budget matches your crawl size.\n\n```bash\ncurl -s https://agentflare.org/pricing.json\n```\n\n### 2. List available dates from the index\n\nStart with the newswire index file. Its data.digests array has one entry per digest, each with date, cluster, headline, url and stories, so you can pick a valid date without guessing. Send your crawler identity in the X-Crawler-Token header, which replaces the earlier name X-AISA-Crawler-Token. Do not put the token in Authorization: Bearer, because that header is reserved for the gateway layer.\n\n```bash\ncurl -s https://agentflare.org/news/index.data.json \\\n  -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'\n```\n\n### 3. Fetch a dated newswire file\n\nDated files follow the pattern /news/YYYY-MM-DD-CLUSTER.data.json, where the cluster is ai, crypto, earnings, macro or markets. Replace YYYY-MM-DD with a date from the index. Each file carries facts, a data object, analysis_md, sources, related and a pricing object with price_usd, method, endpoint, autopay_hint and onboarding. Per-item files use /news/SLUG.data.json.\n\n```bash\ncurl -i https://agentflare.org/news/YYYY-MM-DD-ai.data.json \\\n  -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'\n```\n\n### 4. Read the stories in Python\n\nInside data you will find date, cluster, headline, dek, generated_at and freshness_hours, then stories, watch, takeaways, faq, count and items. Each story has title, summary, why, sources and entities. Each item has title, url, source, date, tickers and iso. The crypto file also fills data.market with symbol, name, last, asOf, d1, d7 and series, plus prediction_markets.\n\n```python\nimport json\nimport urllib.request\n\nurl = 'https://agentflare.org/news/YYYY-MM-DD-ai.data.json'\nreq = urllib.request.Request(\n    url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'}\n)\nwith urllib.request.urlopen(req) as resp:\n    doc = json.load(resp)\n\ndata = doc['data']\nprint(data['headline'])\nfor story in data['stories']:\n    print(story['title'], story['entities'])\nprint(doc['pricing']['price_usd'])\n```\n\n### 5. Pull markets snapshots and tickers\n\nThe daily snapshot lives at /markets/YYYY-MM-DD.data.json. Its data holds count, advancers, decliners, sectors (label, value, n) and stocks (ticker, price_usd, day_change_pct, return_6m_pct, market_cap_usd, pe, sector). Single companies use /markets/SLUG.data.json, such as xom, with returns_pct (m1, m3, m6), a long metrics object and company details including cik, sector and industry.\n\n```bash\ncurl -s https://agentflare.org/markets/YYYY-MM-DD.data.json \\\n  -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'\n\ncurl -s https://agentflare.org/markets/xom.data.json \\\n  -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'\n```\n\n## Newswire and markets files\n\n| File pattern | Price per request | Key data fields |\n| --- | --- | --- |\n| /news/index.data.json | 0.005 USD | data.date, data.digests[] (date, cluster, headline, url, stories) |\n| /news/YYYY-MM-DD-ai.data.json | 0.005 USD | data.headline, data.dek, data.stories[], data.items[], data.watch[], data.faq[] |\n| /news/YYYY-MM-DD-crypto.data.json | 0.005 USD | Same as ai, plus data.market[] (symbol, last, d1, d7, series) and data.prediction_markets[] |\n| /news/SLUG.data.json | 0.005 USD | One file per item |\n| /markets/index.data.json | 0.01 USD | data.count, data.advancers, data.decliners, data.sectors[], data.stocks[] |\n| /markets/YYYY-MM-DD.data.json | 0.01 USD | Same fields as the markets index, for one date |\n| /markets/xom.data.json | 0.01 USD | data.returns_pct (m1, m3, m6), data.metrics, data.company |\n\n## Common pitfalls\n\n- Sending the crawler token in Authorization: Bearer is wrong; that header is reserved for the gateway layer. Use X-Crawler-Token.\n- The placeholder YYYY-MM-DD in URL patterns must be replaced with a real date, which you can read from data.digests[].date in the newswire index.\n- In the ai file, data.market, data.policy_rates and data.prediction_markets are empty arrays, while the crypto file fills market and prediction_markets. Do not assume every cluster has the same populated fields.\n- Markets files cost 0.01 USD per request, twice the 0.005 USD newswire price, so budget a ticker-by-ticker crawl accordingly.\n\n## Key takeaways\n\n- Newswire .data.json files cost 0.005 USD per request; markets .data.json files cost 0.01 USD per request.\n- Find valid dates in data.digests of /news/index.data.json, then build YYYY-MM-DD-cluster URLs.\n- Always authenticate with the X-Crawler-Token header, never Authorization: Bearer.\n\n## FAQ\n\n**How much does an AgentFlare newswire request cost?**\nFiles under /news/ ending in .data.json or .full.md are priced at 0.005 USD per request, billed per-request. Markets files under /markets/ are 0.01 USD per request. Index pages, .html articles and .md teasers stay free.\n\n**Which header carries my crawler token?**\nUse X-Crawler-Token, as defined in the Open Verify specification. It replaces the older name X-AISA-Crawler-Token still shown on some AISA pages. Never send the token in Authorization: Bearer.\n\n**What clusters exist for dated newswire files?**\nDated newswire files are named YYYY-MM-DD-ai, YYYY-MM-DD-crypto, YYYY-MM-DD-earnings, YYYY-MM-DD-macro and YYYY-MM-DD-markets, each ending in .data.json. Per-item files use a slug instead of a date.\n\n**What does a markets ticker file include?**\nA file such as /markets/xom.data.json has price_usd, day_change_pct, returns_pct (m1, m3, m6), a metrics object with valuation, margin, growth and liquidity ratios, and company details: ticker, name, cik, sector and industry.\n",
  "sources": [
    {
      "title": "Header name: X-Crawler-Token",
      "url": "https://esa.aisa.one/cdn/open-verify.html"
    },
    {
      "title": "AgentFlare Newswire .data.json field reference",
      "url": "https://agentflare.org/news/index.html"
    },
    {
      "title": "AgentFlare Markets .data.json field reference",
      "url": "https://agentflare.org/markets/index.html"
    },
    {
      "title": "AgentFlare pricing.json (directory price map)",
      "url": "https://agentflare.org/pricing.json"
    },
    {
      "title": "AgentFlare llms.txt (header and structure)",
      "url": "https://agentflare.org/llms.txt"
    }
  ],
  "related": [
    {
      "name": "Reading .data.json: A Field Guide for Agents",
      "url": "https://agentflare.org/guides/reading-datajson-a-field-guide-for-agents.html"
    },
    {
      "name": "Using AgentFlare Social, Predictions and Crypto Data",
      "url": "https://agentflare.org/guides/using-agentflare-social-predictions-and-crypto-data.html"
    },
    {
      "name": "Using AgentFlare Research and Scholar Data",
      "url": "https://agentflare.org/guides/using-agentflare-research-and-scholar-data.html"
    },
    {
      "name": "Getting Started with HTTP 402 Pay-Per-Crawl",
      "url": "https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html"
    },
    {
      "name": "Crawler Tokens & Auto-Pay: A How-To",
      "url": "https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html"
    },
    {
      "name": "Handling 402 and 403 Responses: An Error Reference for Agents",
      "url": "https://agentflare.org/guides/handling-402-and-403-responses-an-error-reference-for-agents.html"
    }
  ],
  "pricing": {
    "price_usd": 0,
    "method": "402",
    "endpoint": "https://esa.aisa.one/api/v1/access/verify",
    "autopay_hint": "set crawlerAutoPrice=true with the X-Crawler-Token header",
    "onboarding": "https://esa.aisa.one/agent/guide.html"
  },
  "powered_by": "AISA — agent-native search, settlement & delivery (https://aisa.one)"
}