How to Scrape Zenodo: Research Datasets, Papers & Software

Zenodo is CERN’s open-access research repository — datasets, publications, software, images, and more, each with a citable DOI. It’s a goldmine for literature reviews, ML training-data sourcing, and citation tracking, but pulling a topic’s worth of records through the web interface means clicking page after page. The REST API is the right tool, and it’s fully public, but you still have to handle its cursor-based pagination and strip the HTML out of every description before the data is usable. This guide covers how the Zenodo API is shaped and how to extract clean records at scale.

What’s worth extracting

The scraper returns 16 fields per record, covering identity, content, and impact:

• Identity — id (Zenodo internal record ID), doi (the Digital Object Identifier, e.g. 10.5281/zenodo.1234567), and recordUrl (canonical HTML page).
• Bibliographic — title, creators (author names, semicolon-delimited), publicationDate (ISO 8601), and version.
• Content — description (plain text, HTML stripped), keywords (comma-delimited), and resourceType (dataset, publication, software, image, video, lesson, or other).
• Access & licensing — license (identifier like cc-by-4.0 or mit) and accessRight (open, embargoed, restricted, or closed).
• Files — downloadUrl, a direct URL to the primary file or archive.
• Impact — views and downloads (total counts), and communities (comma-delimited community slugs).

Two of these do a lot of work. accessRight tells you whether the downloadUrl will actually give you a file — filter for open if you need the underlying data, not just the metadata. And resourceType lets you separate datasets from papers from software, which is the difference between building a training-data catalog and a literature review.

How the Zenodo REST API works

The scraper connects to Zenodo’s public REST API at https://zenodo.org/api/records and supports three modes:

• records — keyword search across the full index. Popular topics span 210,000+ records.
• recordDetail — bulk fetch specific records by ID, for when you already have the IDs.
• communityRecords — search scoped to a single Zenodo community (curated collections like biodiversity or astronomy).

Pagination is the mechanical detail that matters. Zenodo returns 25 records per page and links to the next page via a cursor in links.next; there’s no simple page-number offset. The actor chains that cursor automatically, following links.next page after page until it hits your maxResults (up to a ceiling of 10,000) or runs out of records. That’s what turns a 25-record page into a few-thousand-record dataset in one run.

A keyword search with a type filter is the most common configuration:

{
  "mode": "records",
  "query": "machine learning dataset",
  "resourceType": "dataset",
  "maxResults": 500
}

If you already know the record IDs, fetch them directly:

{
  "mode": "recordDetail",
  "ids": ["3715409", "10580891", "7025386"]
}

And community mode scopes the search to a curated collection, with an optional query to filter within it:

{
  "mode": "communityRecords",
  "community": "biodiversity",
  "query": "species occurrence",
  "maxResults": 300
}

The query field supports Zenodo’s own query syntax — combine terms with AND, OR, NOT, quote phrases, and filter on fields like creators.name:Hansen.

▶ Run the Zenodo Scraper on Apify — search by keyword, fetch by record ID, or scrape a whole community, and get DOI, authors, license, download stats and file URLs per record. No API key, no login. Export to CSV, JSON or Excel.

The access reality

Zenodo’s API is built for programmatic access, so there’s no anti-bot wall — but a few real constraints shape a run.

• Keyless, but soft rate-limited. No credentials are needed, yet Zenodo applies soft rate limits and returns HTTP 429 under heavy traffic. The actor handles this automatically with exponential back-off, so a large paginated pull slows down rather than failing.
• 10,000 records is the per-run ceiling. maxResults caps at 10,000. For broader coverage than that, split the work across multiple queries — different keywords, or date-range slices via query syntax.
• Cursor pagination, not offsets. Because Zenodo pages via links.next rather than a page number, you can’t jump to “page 50.” The actor walks the cursor sequentially, which is why a 1,000-record pull takes a few minutes — it’s chaining ~40 pages of 25.
• Relevance-ranked by default. Results come back sorted by bestmatch (relevance), not date. If you need the newest uploads chronologically, that’s a sort=mostrecent query parameter you’d apply against the API directly.

There’s also a coverage nuance for community mode: if a community search returns zero results, the slug may not accept the communities filter — the documented fallback is to use records mode with a topic query instead. You can verify a slug at zenodo.org/communities.

Example output

One representative record — a research dataset with full metadata and a direct file URL:

{
  "id": "10580891",
  "doi": "10.5281/zenodo.10580891",
  "title": "Microbiome Diversity Dataset — Amazon Rainforest Soil Samples 2022",
  "creators": "Silva, Maria; Dos Santos, Pedro; Oliveira, Carla",
  "description": "16S rRNA amplicon sequencing data from 847 soil samples collected across 12 Amazon biomes. Includes OTU tables, taxonomy assignments, and environmental metadata.",
  "publicationDate": "2024-02-01",
  "resourceType": "dataset",
  "keywords": "microbiome, amazon, soil, 16S rRNA, biodiversity",
  "license": "cc-by-4.0",
  "accessRight": "open",
  "downloadUrl": "https://zenodo.org/api/records/10580891/files/sequences.tar.gz/content",
  "views": 3241,
  "downloads": 1892,
  "communities": "biodiversity, open-science",
  "recordUrl": "https://zenodo.org/records/10580891",
  "version": "1.0"
}

Note the description is clean plain text — the raw record stores HTML, and the actor strips it so the field drops straight into a dataframe or a search index without markup noise.

Typical use cases

• Literature reviews & meta-analyses — pull all records on a topic across disciplines, with DOIs and citation metadata, to seed a systematic review.
• ML training-data sourcing — search resourceType: dataset for open datasets, filter on accessRight: open, and collect downloadUrls for a training pipeline.
• Research-software cataloguing — build a database of open-source research software with license and downloads, filtering on resourceType: software.
• Citation & impact tracking — cross-reference a reading list of DOIs against Zenodo metadata for views, downloads, and version history.
• Community monitoring — run communityRecords weekly on a curated collection (biodiversity, neuroscience, NLP) to catch new uploads.
• Contributor analysis — export creators from a keyword search to identify the top authors in a research niche.

Cost / effort math

Calling the Zenodo API directly is free, so what you’d be building is the handling around it: chaining the links.next cursor correctly (the failure mode is silently stopping at page one), stripping HTML from every description, joining the nested creators and communities arrays into clean strings, and backing off gracefully on the 429s a big pull will trigger. None of it is hard, but it’s the difference between “I got 25 records” and “I got the 2,000 I actually wanted.”

The managed actor runs pay-per-result — a fraction of a cent per record — which fits research workflows where you pull a few hundred to a few thousand records occasionally rather than continuously. A 500-record search costs almost nothing, and you’re billed on the records returned, not the minutes spent paging. The cursor chaining, HTML stripping, and back-off are already solved.

Common pitfalls

• accessRight gates the download. A downloadUrl on an embargoed, restricted, or closed record won’t hand you the file. Filter for accessRight: open before you queue downloads, or you’ll collect URLs that 403.
• downloadUrl is the primary file only. Records with multiple files return one URL here; for the complete file list you’d hit Zenodo’s files API for that record separately.
• Optional fields are often empty. keywords, version, communities, and license are uploader-optional — many records omit them. Treat empty as “not provided,” not an error.
• Default sort is relevance, not recency. bestmatch ordering means the newest records aren’t first. For chronological results, apply sort=mostrecent via the API directly.
• Community slugs are exact and can be finicky. A wrong or non-filterable slug returns zero rows. Verify at zenodo.org/communities and fall back to records mode with a topic query if a community search comes back empty.
• Use query syntax to slice past the 10,000 ceiling. For a topic bigger than one run, split by date range in the query field, e.g. publication_date:[2023-01-01 TO 2024-01-01], and run each slice separately.
• Cite by DOI. Zenodo metadata is open, but derived work should attribute records by their DOI — that’s both good practice and part of Zenodo’s terms.

Wrapping up

Zenodo’s REST API is public and keyless, so the real work is mechanical: chaining the links.next cursor to get past the first 25 records, stripping HTML from descriptions, and respecting the soft rate limits with back-off. Filter on resourceType and accessRight to get exactly the records — and the downloadable files — you need, and slice big topics by date to work around the 10,000-per-run ceiling. If you’d rather not maintain the pagination and cleanup logic, the managed actor already handles it.

▶ Open the Zenodo Scraper on Apify — one run turns a keyword, ID list, or community into hundreds of clean records: DOI, authors, license, downloads, and direct file URLs. No API key, no login, export to CSV / JSON / Excel. Start on Apify’s free monthly credit.