How to Scrape World Bank Data: GDP & Dev Indicators in 2026

The World Bank Open Data API is the most comprehensive free source of global economic and development statistics — GDP, inflation, population, poverty, life expectancy, CO2 emissions, and 16,000+ more indicators across 200+ countries, some series running back to 1960. It’s fully open and keyless. The catch is the API’s ergonomics: the response is a quirky two-element JSON array (metadata in element zero, data in element one), it paginates in fixed batches, and pulling a multi-country time series means stitching pages together and flattening nested country/indicator objects. This guide covers how the API works and how to pull clean, flat, ready-to-analyze tables without writing that pagination-and-flattening glue.

What’s worth extracting

The scraper runs in three modes, each with its own flat row schema.

indicatorData (the main mode) — a time series for one indicator across many countries. One row per country per year:

• country / countryCode / countryIso3 — full name, ISO2 (e.g. US), ISO3 (e.g. USA).
• indicator / indicatorCode — human-readable name (e.g. GDP (current US$)) and the World Bank code (NY.GDP.MKTP.CD).
• date — the year or period (e.g. 2023).
• value — the numeric value as a string; null when the observation is unavailable.
• unit — unit of measurement if provided.
• observationStatus — status code (E = estimate, P = provisional, etc.).

indicators mode — the full catalog of 16,000+ indicator codes: id, name, sourceNote (the full description of what’s measured), topics (tag list), unit. Run this first to discover what’s available.

countries mode — metadata for all 295 World Bank country/territory records: id (ISO3), iso2Code, name, region, incomeLevel (High / Upper-middle / Lower-middle / Low), capitalCity, and latitude/longitude of the capital.

How the World Bank API works

The actor connects directly to the official REST API at https://api.worldbank.org/v2, which is keyless — no registration, no tokens, no monthly quota. It uses got-scraping for HTTP with retry logic and pushes normalized rows straight into a dataset. No login, no browser, pure JSON.

For indicatorData, you supply a semicolon-separated list of ISO2 country codes (or all), an indicator code, and a year range. The actor paginates in 1,000-row batches using the ?page=N parameter until every matching observation is collected. That pagination is the part that’s tedious to hand-roll: a broad pull can span dozens of pages, and you have to detect the last page from the metadata element rather than a simple count.

A realistic input — GDP for five countries across two decades:

{
  "mode": "indicatorData",
  "countries": "US;CN;IN;DE;GB",
  "indicator": "NY.GDP.MKTP.CD",
  "dateFrom": 2000,
  "dateTo": 2023
}

Common indicator codes you’ll reach for:

• GDP (current US$) — NY.GDP.MKTP.CD
• GDP per capita (current US$) — NY.GDP.PCAP.CD
• Population, total — SP.POP.TOTL
• Inflation, consumer prices (%) — FP.CPI.TOTL.ZG
• Unemployment (% of labor force) — SL.UEM.TOTL.ZS
• Life expectancy at birth (years) — SP.DYN.LE00.IN
• CO2 emissions (metric tons per capita) — EN.ATM.CO2E.PC
• Internet users (% of population) — IT.NET.USER.ZS

To discover others, run indicators mode with a small maxResults first:

{
  "mode": "indicators",
  "maxResults": 500
}

And for a mapping dashboard, countries mode gives you region, income level and capital coordinates in one call:

{
  "mode": "countries"
}

▶ Run the World Bank Data Scraper on Apify — GDP, inflation, population and 16,000+ indicators across 200+ countries, fully paginated into flat rows. Keyless, no login. Export to CSV, JSON or Excel.

The access reality

This is one of the cleaner data sources on the open web, and the friction is minimal by design:

• No key, no auth, no quota. The API is fully open and explicitly built for programmatic access. There’s nothing to register or rotate.
• Datacenter proxy is sufficient. The World Bank API does not block datacenter IPs, so no residential proxy is needed — the actor defaults to Apify’s datacenter proxy.
• The real cost is pagination, not access. The engineering the actor absorbs is the 1,000-row batching and the flattening of the two-element response array, not any anti-bot dance.
• Speed. A 5-country by 24-year pull (about 120 rows) finishes in under 10 seconds; a full all-countries run for one indicator (6,000+ rows) takes roughly 30–90 seconds depending on pagination.

Update cadence is worth knowing for scheduling: most indicators refresh annually, with the main bulk release in late spring (April–June); some series update more often. Quarterly or monthly scheduled runs keep a dataset fresh without over-fetching.

Example output

A single indicatorData row:

{
  "country": "United States",
  "countryCode": "US",
  "countryIso3": "USA",
  "indicator": "GDP (current US$)",
  "indicatorCode": "NY.GDP.MKTP.CD",
  "date": "2023",
  "value": "27811517000000",
  "unit": "",
  "observationStatus": ""
}

That value — 27811517000000 — is raw US dollars, not millions or billions. It’s ~$27.8 trillion; divide by 1e9 for billions in your spreadsheet.

Use cases

• Comparative growth analysis — pull GDP per capita for all G20 countries from 1990 to 2023 and load it into Google Sheets for a growth chart.
• Monetary-policy research — pull 20 years of inflation (FP.CPI.TOTL.ZG) for emerging markets to study policy impacts.
• Mapping dashboards — build a country metadata table (region, income level, capital coordinates) from countries mode.
• BI automation — schedule quarterly downloads of poverty and unemployment data into Tableau or Power BI.
• Indicator discovery — run indicators mode and filter by topic to find codes covering CO2 emissions, renewable energy or gender equality.
• Academic datasets — build long, flat panels for theses, papers and quantitative coursework; the row-per-country-per-year format maps straight into a pandas DataFrame.

Build it yourself vs. use the managed actor

You can call the World Bank API directly — it’s public. What the managed actor absorbs:

• The two-element response quirk. Every response is [metadata, data]; you have to read the total-page count from element zero and the observations from element one.
• Pagination. The 1,000-row ?page=N batching, with last-page detection, across pulls that can span dozens of pages.
• Flattening. Turning nested country and indicator objects into a flat row-per-observation table.
• Retry logic for transient HTTP errors on long paginated runs.

Pricing is pay-per-result, per compute unit — large runs stay cheap and there’s no monthly quota. Set maxResults: 50 for a quick sanity check before a full all-countries pull, which can produce 6,000–9,000 rows for one indicator over a 30-year range.

Common pitfalls

Specific gotchas for World Bank data:

• Null values are legitimate gaps, not errors. World Bank data has real holes — not every country reports every indicator every year. A value of null means the observation wasn’t collected, not that the scraper failed. Don’t treat nulls as a bug or drop rows blindly; the gap itself is information.
• Use ISO2 codes, not names or ISO3, in countries. The countries input takes 2-letter codes (US, DE, IN). Country names or ISO3 codes won’t match. (The output carries all three forms, but the input is ISO2.)
• One indicator per run. The current version fetches a single indicator per indicatorData run. For multiple indicators, run once per code and merge the datasets by country + year downstream.
• Not all series go back equally far. Many indicators lack data before 1990. If a wide dateFrom returns mostly nulls, narrow the range to where the series actually starts.
• value is a string. All numeric values ship as strings. Cast before computing, and remember GDP figures are full dollar amounts (divide by 1e9 for billions).
• Chain the modes to find codes. Guessing indicator codes is error-prone. Run indicators mode (or browse data.worldbank.org) to get the exact id before a big indicatorData pull — a wrong code returns nothing.

Scheduling and downstream analysis

Because World Bank indicators refresh on a slow annual cadence — mostly in the April-to-June window — you don’t need aggressive scheduling. A monthly or quarterly scheduled run keeps a dataset current without hammering the API for data that hasn’t changed. Point the run at Apify’s scheduler and let it refresh your economic panel automatically.

The flat row-per-observation format is deliberately warehouse- and DataFrame-friendly. Each indicatorData row is one country, one indicator, one year, one value — which maps directly onto a pandas DataFrame via pd.read_json() or pd.read_csv() against the dataset’s API endpoint, or straight into a BI tool like Tableau or Power BI. For multi-indicator analysis, run the actor once per indicator code and merge the resulting datasets on countryCode plus date; the consistent key columns make that join trivial. The Apify integrations round it out: export to Google Sheets for instant pivot tables, POST results to your own ETL endpoint via a webhook, or push into Airtable, Notion or a database through Zapier or Make when a run finishes.

Licensing note

World Bank Open Data is published under the Creative Commons Attribution 4.0 International license (CC BY 4.0) — commercial and non-commercial use are both permitted, with attribution. That’s a genuinely permissive license for a dataset this comprehensive, and it’s why this source is a staple for data journalism and commercial analytics alike. Verify coverage and quality for any high-stakes decision, as you would with any public dataset.

Wrapping up

The World Bank API is free and keyless — the friction is its two-element response shape, the fixed-batch pagination, and flattening nested objects into a clean panel. If you need GDP, inflation, population or any of 16,000+ indicators as a flat, warehouse-ready table refreshed on a schedule, a managed actor hands you exactly that with the pagination and flattening already solved.

▶ Open the World Bank Data Scraper on Apify — three modes (time series, indicator directory, country metadata), thousands of flat rows per run, CC BY 4.0 data. Keyless, no login. Export to CSV, JSON or Excel. Start on Apify’s free monthly credit.