Warconomy can now expand using free, legitimate public datasets and APIs — but carefully. This page records, for each candidate source, whether to ingest it now, watch it, use it detect-only (discovery, not facts), or reject it, and why. The rules: no paid APIs, no committed API keys, and no fragile live dependencies in the default build. Anything that needs the network is run on-demand and committed as a static snapshot, kept out of the default build and validation.
- No paid APIs and no committed API keys or secrets.
- No runtime services or background workers; the default build makes no external network calls.
- Network-dependent ingests are on-demand and produce a committed static snapshot, kept out of default validation.
- Noisy/news-derived sources are for topic/report discovery only — never auto-published as facts.
- Sensitive casualty/displacement sources (UNHCR, IOM, UCDP, WHO) are citation-only — linked as estimates, never republished as Warconomy figures.
At a glance
- 21 candidates reviewed.
- 6 ingest-now · 6 watch · 2 detect-only · 5 citation-only · 2 reject.
- Live integrations: conflict economies and human-capital demographics (World Bank, on-demand snapshots, key-free).
- Sensitive sources (UNHCR, IOM, UCDP, WHO) are citation-only: linked as estimates, never republished as Warconomy figures.
Candidates
| Source | Coverage | Access | Cadence | Decision |
|---|---|---|---|---|
| World Bank Indicators API World Bank | Country macro indicators (GDP, inflation, military expenditure % GDP, trade) | free keyfree api | Annual (mostly) | Ingest now |
| World Bank Commodity Markets (Pink Sheet) World Bank | Monthly commodity price levels (energy, metals, food, fertilizer) | free download | Monthly | Ingest now |
| FAO Food Price Index FAO | Global food price index + sub-indices | free download | Monthly | Ingest now |
| EIA Open Data API U.S. Energy Information Administration | Oil/gas prices, production, stocks, chokepoint flows | free keyed api | Daily–monthly | Watch |
| FRED (Federal Reserve Economic Data) Federal Reserve Bank of St. Louis | Macro and price series (incl. some commodity/energy) | free keyed api | Varies | Watch |
| IMF PortWatch International Monetary Fund | Port and chokepoint trade disruption indicators | free download | Frequent | Watch |
| USGS Mineral Commodity Summaries U.S. Geological Survey | Production, reserves and trade for critical minerals (cobalt, copper, uranium, etc.) | free download | Annual | Citation-only |
| World Bank demographic & labour indicators World Bank | Population, ages 15–64, labour force, fertility, net migration, GDP, health spend | free keyfree api | Annual | Ingest now |
| Population age structure & dependency (UN WPP via World Bank) World Bank / UN DESA | Age bands (0–14, 15–64, 65+), working-age & dependency ratios for conflict economies | free keyfree api | Annual | Ingest now |
| UNHCR Refugee Data Finder UNHCR (UN Refugee Agency) | Refugees, asylum-seekers, returnees by country/year (+ an API) | free keyfree api | Annual + updates | Citation-only |
| IOM Displacement Tracking Matrix International Organization for Migration (IOM) | Internal displacement (IDP) tracking by country/crisis | free keyfree api | Frequent | Citation-only |
| UN DESA World Population Prospects UN DESA Population Division | Population estimates and long-run projections by country | free download | Periodic revisions | Watch |
| WHO Global Health Observatory World Health Organization (WHO) | Health indicators incl. disability, mental health, health systems | free keyfree api | Periodic | Citation-only |
| ILOSTAT (ILO labour statistics) International Labour Organization (ILO) | Labour force, participation, employment, youth employment | free keyfree api | Annual + updates | Watch |
| UCDP (Uppsala Conflict Data Program) Uppsala University | Organized-violence events and datasets | free keyfree api | Annual + candidate monthly | Citation-only |
| ReliefWeb API UN OCHA | Humanitarian situation reports, disasters, country pages | free keyfree api | Continuous | Detect-only |
| GDELT GDELT Project | Global news event/tone index | free keyfree api | Continuous (15-min) | Detect-only |
| UN Comtrade United Nations | Bilateral trade flows by product | free keyed api | Annual/monthly | Watch |
| ACLED ACLED | Armed conflict event data | free keyed api | Weekly | Reject |
| SIPRI / NATO defense expenditure SIPRI / NATO | Military expenditure (world, regional, national) | free download | Annual | Ingest now |
| OECD / IEA datasets OECD / IEA | Energy balances, prices, macro | free keyed api | Varies | Reject |
Why each decision
- World Bank Indicators API — Ingest now: Key-free, stable, official. Integrated as an on-demand snapshot powering /conflict-economies. No key, no default-build network. (terms risk: Low — open data, permissive terms with attribution.)
- World Bank Commodity Markets (Pink Sheet) — Ingest now: Already integrated (npm run commodities:ingest:worldbank); committed snapshot. (terms risk: Low — open data with attribution.)
- FAO Food Price Index — Ingest now: Already integrated (npm run commodities:ingest); committed snapshot. (terms risk: Low — FAO open terms with attribution.)
- EIA Open Data API — Watch: Optional-key future integration (see the optional-key framework below). Never commit a key; the default build must not require one. EIA's published tables are already cited without the API. (terms risk: Low — US government, public domain; but requires a free API key.)
- FRED (Federal Reserve Economic Data) — Watch: Optional-key future integration (see framework below). Keep the key uncommitted and out of the default build; some CSV endpoints work key-free on demand. (terms risk: Low — free, but API needs a key.)
- IMF PortWatch — Watch: Already cited as a source. Linking/discovery is safe now; numeric ingest would need a reviewed snapshot. (terms risk: Low–medium — check redistribution terms; cite/link rather than republish.)
- USGS Mineral Commodity Summaries — Citation-only: USGS pages block automated requests (HTTP 403) and data is annual PDF / ZIP-CSV per commodity — not a clean key-free API. Used citation-only: the /minerals page links USGS per-commodity pages and publishes no figures without a reviewed source packet. Do not scrape or hand-transcribe. (terms risk: Low — US government, public domain.)
- World Bank demographic & labour indicators — Ingest now: Integrated as an on-demand snapshot powering /human-capital (npm run worldbank:humancapital). Macro/demographic context only — NOT casualty data. (terms risk: Low — open data with attribution.)
- Population age structure & dependency (UN WPP via World Bank) — Ingest now: Integrated via npm run population:ingest (committed snapshot). UN DESA's Data Portal API is reachable but heavy/less stable for build-and-forget, so the same UN-WPP estimates are taken from the key-free World Bank API. Demographic estimates only — NOT casualty data. (terms risk: Low — UN WPP estimates republished by World Bank open data, with attribution.)
- UNHCR Refugee Data Finder — Citation-only: Cite/link rather than republish. Displacement counts are estimates that change over time; do not present as Warconomy figures or as certainties. (terms risk: Low — UN open data; sensitive subject, cite carefully and as estimates.)
- IOM Displacement Tracking Matrix — Citation-only: Link for displacement context; estimates only, not republished as Warconomy figures. (terms risk: Low–medium — check redistribution; sensitive subject.)
- UN DESA World Population Prospects — Watch: Strong candidate for a future reviewed snapshot of population/age-structure; overlaps World Bank for current values. (terms risk: Low — UN open data with attribution.)
- WHO Global Health Observatory — Citation-only: Cite for health/disability context; do not derive clinical or prevalence claims. (terms risk: Low — WHO open data; sensitive health topics, cite carefully.)
- ILOSTAT (ILO labour statistics) — Watch: Candidate for a future labour-force snapshot; World Bank covers the headline labour indicators for now. (terms risk: Low — open data with attribution.)
- UCDP (Uppsala Conflict Data Program) — Citation-only: Open academic data, usable. But fatality figures are sensitive and estimated — cite/link as ranges with caveats; never republish as Warconomy casualty counts or as certainties. (terms risk: Low — academic open data with citation.)
- ReliefWeb API — Detect-only: Use for report discovery/linking only; do NOT derive numeric claims. API endpoint shape changes — keep on-demand. (A tested endpoint returned 410, so treat as unstable.) (terms risk: Low — UN open content; use for discovery/linking, not numeric claims.)
- GDELT — Detect-only: Event/topic discovery only. Too noisy to cite as a fact source; never auto-publish. (terms risk: Medium — derived from news; noisy; attribution/quality vary.)
- UN Comtrade — Watch: Free tier is rate-limited; suitable for a small reviewed snapshot later, not default build. (terms risk: Medium — free tier has rate/volume limits; premium for bulk.)
- ACLED — Reject: Licensing/redistribution terms make it a poor fit for a static, openly-republished site. Prefer UCDP for open conflict context. (terms risk: Higher — registration + license terms restrict redistribution.)
- SIPRI / NATO defense expenditure — Ingest now: Already used as cited sources (annual). Manual transcription of headline values, not bulk redistribution. (terms risk: Low–medium — cite values; SIPRI terms restrict bulk redistribution.)
- OECD / IEA datasets — Reject: Most useful IEA series are paywalled/restricted; free EIA and World Bank cover our needs without that risk. (terms risk: Medium–high — much IEA data is paywalled or redistribution-restricted.)
Optional-key APIs (EIA, FRED)
EIA and FRED expose useful data behind a free API key. They are classified as OPTIONAL-KEY FUTURE INTEGRATIONS: useful later, not blocking now, and never a default-build dependency.
- The key lives only in a local .env (or shell env); it is never committed.
- The ingest command reads the key from the environment and FAILS GRACEFULLY with a clear message if it is missing.
- Only the generated snapshot is committed — after review — never the key.
- A test asserts no API key/secret is committed anywhere in the repo.
- The default build and validation never require the key or a network call.
Status: not started — no EIA/FRED key handling exists yet; recorded as a future option only.
Related
Machine-readable: /methodology/data-sources/data.json. See also critical minerals & conflict, dataset versioning vs editorial overlay, data needs (backlog), and the source registry.