How to Scrape Wikipedia Category Article Lists in 2026

Wikipedia’s category system is a vast, human-curated taxonomy — every article is filed under one or more Category: pages, and those pages are effectively ready-made topic lists maintained by thousands of editors. If you want every article about machine learning or a multilingual corpus of the same subject, that taxonomy is exactly what you need. The problem is getting it out cleanly. The official MediaWiki API works, but it’s paginated, rate-limited, and genuinely awkward for walking a category tree; the raw category HTML pages are consistent but tedious to parse and split across multiple pages for large categories. This guide covers how Wikipedia category pages are structured, how to extract full article lists in bulk across languages, and what the output looks like.

What’s worth extracting

Each row is one article found in a category, with these fields:

• Article identity — articleTitle (human-readable title), articleUrl (full canonical URL), pageId (Wikipedia’s internal, stable page ID — your durable key).
• Category context — categoryName (the category the article was found in) and categoryUrl (the source category page).
• Optional text — summary (the article’s first paragraph, populated only when scrapeSummaries is on) — clean, ready-to-use text for embeddings and NLP.
• Metadata — type (always "article"), language (the edition code: en, de, fr, …), and scrapedAt (ISO 8601 timestamp).

For a topic map you need title, URL and page ID. For an NLP or RAG dataset, the summary field is the payload — enable it and each row carries usable first-paragraph text.

How Wikipedia category pages work

The actor parses Wikipedia’s public HTML directly — no MediaWiki API token, no browser, no DOM guesswork. Category pages have a clean, consistent structure across editions, which makes direct HTTP extraction reliable and fast.

You target categories one of two ways. Pass bare category names and let the actor build the correct per-language URL:

{
  "categoryNames": ["Machine_learning", "Artificial_intelligence", "Large_language_models"],
  "language": "en",
  "scrapeSummaries": false,
  "maxArticles": 500,
  "proxyConfiguration": { "useApifyProxy": true }
}

Or pass full category URLs when you already have them:

{
  "categoryUrls": [
    { "url": "https://en.wikipedia.org/wiki/Category:Deep_learning" },
    { "url": "https://en.wikipedia.org/wiki/Category:Neural_network_architectures" }
  ],
  "scrapeSummaries": true,
  "maxArticles": 300
}

Two behaviors matter. First, automatic pagination: category pages typically list 50–200 articles per page, and large categories span multiple pages — the actor walks all of them so you don’t have to chase “next page” links. Second, one-level subcategory discovery: it follows subcategory links one level deep, walking a bit further into the human-curated tree. To go deeper than one level, feed the discovered subcategory URLs back into a second run.

To find a category in the first place: open any Wikipedia article, scroll to the Categories bar at the bottom, and click one — the URL reads .../wiki/Category:Some_Topic. Copy the full URL into categoryUrls, or just the Some_Topic part into categoryNames. Underscores replace spaces.

▶ Run the Wikipedia Category Scraper on Apify — point it at a category name or URL and get every listed article with title, URL and page ID. No API key, no login. Export to CSV or JSON.

The access reality

Wikipedia is one of the friendliest sources on the web to extract from — it’s a nonprofit that publishes its content under a free license and rarely blocks well-behaved clients. The constraints here are about scope and language, not defenses:

• Ten language editions, and only ten. The actor supports English (en), German (de), French (fr), Spanish (es), Italian (it), Japanese (ja), Chinese (zh), Russian (ru), Portuguese (pt) and Arabic (ar). To build a multilingual corpus, run the same subject across editions — but note that category names differ per language (English Machine_learning is French Apprentissage_automatique), so you supply the localized category name for each edition.
• Subcategory crawl is one level deep. By design, the actor follows subcategory links a single level from each seeded category. This is a deliberate volume control — deep, unbounded category recursion on Wikipedia can explode into tens of thousands of pages. For deeper trees, chain runs: take the subcategory URLs from run one, feed them into run two.
• maxArticles caps the run at 2,000. The total article cap across all categories is 1–2,000 (default 300). You pay only for what you collect, so set it to the volume you actually need.
• Summaries cost extra requests. With scrapeSummaries: false (the default), the actor only reads category pages — fast. With it true, it additionally fetches each article page to pull the first paragraph, which is slower but gives you the text payload for NLP work.

Because Wikipedia is proxy-friendly, Apify Proxy with datacenter groups works out of the box — no residential IPs needed.

Example output

One article row with a summary (from a scrapeSummaries: true run):

{
  "type": "article",
  "articleTitle": "Convolutional neural network",
  "articleUrl": "https://en.wikipedia.org/wiki/Convolutional_neural_network",
  "categoryName": "Deep learning",
  "categoryUrl": "https://en.wikipedia.org/wiki/Category:Deep_learning",
  "summary": "A convolutional neural network (CNN) is a regularized type of feed-forward neural network that learns features by itself via filter optimization...",
  "pageId": "40409788",
  "language": "en",
  "scrapedAt": "2026-07-06T12:00:00Z"
}

Without summaries, the summary field is absent and the run is faster — ideal when you just want the article list and page IDs.

Who uses this

• Data engineers and ML teams building training corpora — a category is a labeled topic bucket, and scrapeSummaries turns it into ready-to-embed text.
• SEO and content-marketing agencies mapping topic clusters and content silos; every category is a pre-built list of subtopics to structure topical authority around.
• Academic and NLP researchers collecting clean first-paragraph text by topic for classification, embeddings and retrieval-augmented generation.
• Knowledge-management and taxonomy teams converting Wikipedia’s curated category tree into structured taxonomies and entity-relationship graphs.
• Data journalists assembling structured lists of companies, technologies, historical figures or scientific concepts by subject.
• Anyone building a directory who needs a clean, structured slice of Wikipedia scoped to one topic.

Cost and effort: build vs. managed

You can hit the MediaWiki API or parse category HTML yourself — Wikipedia doesn’t fight you. But the tedium adds up: handling multi-page categories, following subcategory links exactly one level (and not accidentally recursing into thousands of pages), building the correct per-language URLs, optionally fetching each article for its first paragraph, and normalizing everything into the flat article record. The MediaWiki API’s pagination and rate limits make the “clean” path more painful than it sounds, which is why direct HTML parsing is often the pragmatic choice.

The managed actor is pay-per-result — a fraction of a cent per article row plus a trivial start fee. A few hundred articles complete in minutes without summaries; enabling summaries costs more per row but delivers the text you’d otherwise scrape separately. With no anti-bot or residential-proxy cost, the bill is essentially per-row, so building topic datasets and multilingual corpora stays inexpensive.

Common pitfalls

Wikipedia-category-specific things to know:

• Category names are localized. The same topic has a different category name in each language edition. To build a parallel corpus, look up the native category name per language rather than reusing the English one.
• Subcategory depth is one level — plan for chaining. If a topic’s articles live in deeply nested subcategories, a single run won’t reach them. Take the subcategory URLs from the first run and feed them into a second.
• pageId is your stable key, not the title. Article titles get renamed; the pageId is Wikipedia’s internal, stable identifier. Join and dedupe on it.
• Summaries roughly double the work per article. Turning on scrapeSummaries fetches each article page individually. Leave it off for pure list-building; turn it on only when you need the text.
• Underscores, not spaces, in names. Category and article slugs use underscores (Machine_learning). Pass names in that form.
• Attribution is on you. Wikipedia content is CC BY-SA. The actor collects only public page data, but if you republish, you’re responsible for proper attribution and complying with Wikipedia’s terms.
• A category is a list, not the article body. The default output is references (title, URL, page ID) — the article lists, not full article text. If you need text, scrapeSummaries gets the first paragraph; full-article extraction is a different job.

Wrapping up

Wikipedia’s category system is a free, human-curated taxonomy that maps directly onto the topic lists and training corpora people want to build. The access is open and unhostile — the work is in the extraction: paginating large categories, walking one level of subcategories without exploding, handling ten localized editions, and optionally pulling first-paragraph text. If you need one category once, the HTML is parseable by hand. If you need clean, structured article lists — or an NLP-ready text dataset — across topics and languages, a managed actor delivers it as a spreadsheet for a fraction of a cent per row.

▶ Open the Wikipedia Category Scraper on Apify — one run yields every article in a category across 10 language editions, with titles, URLs, page IDs and optional first-paragraph summaries, exportable to CSV, Excel or JSON. No API key, no login. Start on Apify’s free monthly credit.