Free data sources roadmap

At a glance

• 27 candidates reviewed.
• 6 ingest-now · 8 watch · 3 detect-only · 8 citation-only · 2 reject.
• Live integrations: conflict economies and human-capital demographics (World Bank, on-demand snapshots, key-free).
• Sensitive sources (UNHCR, IOM, UCDP, WHO) are citation-only: linked as estimates, never republished as Warconomy figures.

Candidates

Why each decision

• World Bank Indicators API — Ingest now: Key-free, stable, official. Powers /conflict-economies (npm run worldbank:macro), the /human-capital demographic context (npm run worldbank:humancapital), and the /war-and-gdp historical comparison of GDP growth around conflict onset (npm run wars:gdp). No key, no default-build network. (terms risk: Low — open data, permissive terms with attribution.)
• World Bank Commodity Markets (Pink Sheet) — Ingest now: Integrated (npm run commodities:ingest:worldbank); committed snapshot expanded to 19 monthly series, adding 10 metals (copper, aluminum, nickel, zinc, lead, tin, iron ore, gold, silver, platinum) surfaced as indexed comparison views on /commodities and a price section on /minerals. Cobalt, lithium and uranium are not in the Pink Sheet. (terms risk: Low — open data with attribution.)
• FAO Food Price Index — Ingest now: Already integrated (npm run commodities:ingest); committed snapshot. (terms risk: Low — FAO open terms with attribution.)
• EIA Open Data API — Watch: Optional-key future integration (see the optional-key framework below). Never commit a key; the default build must not require one. EIA's published tables are already cited without the API. (terms risk: Low — US government, public domain; but requires a free API key.)
• EIA weekly retail gasoline & diesel prices — Watch: Optional-key future integration only. A committed snapshot of a single weekly average would go stale fast and read as a current claim, so pump price stays user-entered; EIA's gasoline page is cited as where to find today's number. Never commit a key; the default build stays network-free. (terms risk: Low — US government, public domain; the API needs a free key.)
• EIA Short-Term Energy Outlook (STEO) — Citation-only: Citation-only by design: Warconomy makes no forecasts, so the STEO would only ever appear as 'EIA forecast (as of {vintage})', visually separated and source-cited — never ingested as Warconomy's own number. Cited from the scenario tools as 'what an official forecaster expects' context. (terms risk: Low — US government, public domain; the forecasts are clearly the EIA's.)
• FRED (Federal Reserve Economic Data) — Watch: Optional-key future integration (see framework below). Keep the key uncommitted and out of the default build; some CSV endpoints work key-free on demand. (terms risk: Low — free, but API needs a key.)
• IMF PortWatch — Watch: Already cited as a source. Linking/discovery is safe now; numeric ingest would need a reviewed snapshot. (terms risk: Low–medium — check redistribution terms; cite/link rather than republish.)
• ReliefWeb API (UN OCHA) — Watch: MANUAL UNLOCK: v1 is decommissioned; v2 returns 403 for unapproved appnames. To unlock, a maintainer registers a free appname at https://apidoc.reliefweb.int/parameters#appname (state the non-commercial humanitarian use; it is not a secret and needs no env var). Once approved, an ingest can fetch recent report metadata (title, source, date, country, url) for the conflict topics and write a committed snapshot read by /current — link and summarize conservatively, never convert reports into numeric observations, never store full bodies. Until then ReliefWeb is cited as a 'watch on' source only; nothing about the default build depends on it. (terms risk: Low — humanitarian open data, but the v2 API now requires a pre-approved appname (free, manual registration).)
• GDELT DOC 2.0 — Detect-only: Verified reachable and keyless, but noisy (mixed-quality multi-language outlets) AND aggressively rate-limited. An on-demand attention probe exists (npm run attention:check) that stores coverage VOLUME only — never article text — but in testing only ~2 of 8 topics returned data per run even at 6.5s spacing. DEFERRED from any public or prominent operator surface for unreliability; attention ≠ truth ≠ impact. The /current watchlist stands on its own without it. Re-evaluate if GDELT loosens throttling. (terms risk: Medium — aggregates mixed-quality, multi-language outlets; not an authoritative source.)
• USGS Mineral Commodity Summaries — Citation-only: USGS pages block automated requests (HTTP 403) and data is annual PDF / ZIP-CSV per commodity — not a clean key-free API. Used citation-only: the /minerals page links USGS per-commodity pages and publishes no figures without a reviewed source packet. Do not scrape or hand-transcribe. (terms risk: Low — US government, public domain.)
• World Bank demographic & labour indicators — Ingest now: Integrated as an on-demand snapshot powering /human-capital (npm run worldbank:humancapital). Macro/demographic context only — NOT casualty data. (terms risk: Low — open data with attribution.)
• Population age structure & dependency (UN WPP via World Bank) — Ingest now: Integrated via npm run population:ingest (committed snapshot). UN DESA's Data Portal API is reachable but heavy/less stable for build-and-forget, so the same UN-WPP estimates are taken from the key-free World Bank API. Demographic estimates only — NOT casualty data. (terms risk: Low — UN WPP estimates republished by World Bank open data, with attribution.)
• UNHCR Refugee Data Finder — Citation-only: Cite/link rather than republish. Displacement counts are estimates that change over time; do not present as Warconomy figures or as certainties. (terms risk: Low — UN open data; sensitive subject, cite carefully and as estimates.)
• IOM Displacement Tracking Matrix — Citation-only: Link for displacement context; estimates only, not republished as Warconomy figures. (terms risk: Low–medium — check redistribution; sensitive subject.)
• UN DESA World Population Prospects — Watch: Strong candidate for a future reviewed snapshot of population/age-structure; overlaps World Bank for current values. (terms risk: Low — UN open data with attribution.)
• WHO Global Health Observatory — Citation-only: Cite for health/disability context; do not derive clinical or prevalence claims. (terms risk: Low — WHO open data; sensitive health topics, cite carefully.)
• ILOSTAT (ILO labour statistics) — Watch: Candidate for a future labour-force snapshot; World Bank covers the headline labour indicators for now. (terms risk: Low — open data with attribution.)
• UCDP (Uppsala Conflict Data Program) — Citation-only: Open academic data, usable. But fatality figures are sensitive and estimated — cite/link as ranges with caveats; never republish as Warconomy casualty counts or as certainties. (terms risk: Low — academic open data with citation.)
• ReliefWeb API — Detect-only: Use for report discovery/linking only; do NOT derive numeric claims. API endpoint shape changes — keep on-demand. (A tested endpoint returned 410, so treat as unstable.) (terms risk: Low — UN open content; use for discovery/linking, not numeric claims.)
• GDELT — Detect-only: Event/topic discovery only. Too noisy to cite as a fact source; never auto-publish. (terms risk: Medium — derived from news; noisy; attribution/quality vary.)
• UN Comtrade — Watch: Free tier is rate-limited; suitable for a small reviewed snapshot later, not default build. (terms risk: Medium — free tier has rate/volume limits; premium for bulk.)
• ACLED — Reject: Licensing/redistribution terms make it a poor fit for a static, openly-republished site. Prefer UCDP for open conflict context. (terms risk: Higher — registration + license terms restrict redistribution.)
• SIPRI / NATO defense expenditure — Ingest now: Already used as cited sources (annual). Manual transcription of headline values, not bulk redistribution. (terms risk: Low–medium — cite values; SIPRI terms restrict bulk redistribution.)
• OECD / IEA datasets — Reject: Most useful IEA series are paywalled/restricted; free EIA and World Bank cover our needs without that risk. (terms risk: Medium–high — much IEA data is paywalled or redistribution-restricted.)
• IMF World Economic Outlook (WEO) database — Citation-only: Citation-only by design: the WEO mixes observed history with IMF projections, and Warconomy makes no forecasts. If ever surfaced it would appear only as 'IMF projection (as of {vintage})', visually separated from observed series and source-cited — never ingested as Warconomy data. The network-free build is unchanged. (terms risk: Low — freely downloadable with attribution; the projection columns are clearly the IMF's.)
• UNCTAD trade & maritime statistics — Citation-only: Already linked as a watch/citation source on the shipping pages and the simulator. Citation-only: pointed to for context with no figure republished as Warconomy's own. No ingestion. (terms risk: Low — UN open data and publications with attribution.)

Optional-key APIs (EIA, FRED)

EIA and FRED expose useful data behind a free API key. They are classified as OPTIONAL-KEY FUTURE INTEGRATIONS: useful later, not blocking now, and never a default-build dependency.

• The key lives only in a local .env (or shell env); it is never committed.
• The ingest command reads the key from the environment and FAILS GRACEFULLY with a clear message if it is missing.
• Only the generated snapshot is committed — after review — never the key.
• A test asserts no API key/secret is committed anywhere in the repo.
• The default build and validation never require the key or a network call.

Status: not started — no EIA/FRED key handling exists yet; recorded as a future option only.

Related

Machine-readable: /methodology/data-sources/data.json. See also critical minerals & conflict, dataset versioning vs editorial overlay, data needs (backlog), and the source registry.