How to Scrape Crossref: DOIs, Citations & Journal Data

Crossref is the plumbing under most of scholarly publishing — the world’s largest DOI registry, covering 150+ million works from 50,000+ publishers. Its REST API is public and keyless, which sounds like the problem is solved. It isn’t. If you want a clean bibliometric dataset — a thousand recent papers on a topic, every article in a journal, citation counts across a field — you still have to write cursor-pagination logic to get past the 10,000-row offset ceiling, join fiddly nested date and author arrays, and route through the polite pool so you don’t get throttled. This guide covers how the Crossref API is shaped, the three ways to query it, and how to pull structured scholarly rows without any of that plumbing.

What’s worth extracting

The actor flattens each Crossref record into 17 structured fields. Grouped by purpose:

• Identity — doi (the Digital Object Identifier), title, url (the canonical https://doi.org/... link), type (journal-article, book-chapter, proceedings-article, etc.).
• Authorship and dates — authors (semicolon-separated “Given Family” names), publishedDate (date parts joined).
• Venue — containerTitle (the journal or conference name), publisher, issn, volume, issue, page.
• Bibliometrics — citationCount (Crossref’s is-referenced-by-count, i.e. how many other works cite this one) and referencesCount (how many references are in this work’s own bibliography).
• Classification and rights — subject (comma-separated subject classifications), license (a license URL, often Creative Commons), funders (semicolon-separated funder names).

The two count fields are the ones people underestimate. citationCount gives you impact without touching a paid citation index, and referencesCount lets you build a rough citation graph from a single query.

How the source actually works

Crossref exposes a REST API at api.crossref.org. The actor supports three modes that map onto the API’s three main access patterns:

• works — full-text keyword search across the entire index, with Crossref’s filter syntax and cursor-based deep pagination.
• journalWorks — every publication from a specific journal, addressed by ISSN.
• workByDoi — resolve a list of specific DOIs to full metadata.

A works-mode search looks like this:

{
  "mode": "works",
  "query": "deep learning",
  "filter": "from-pub-date:2023-01-01,type:journal-article",
  "maxResults": 300,
  "mailto": "[email protected]"
}

The filter string is where Crossref’s power lives. It accepts comma-separated flags: from-pub-date, until-pub-date, type (journal-article, book-chapter, etc.), has-full-text, has-orcid, has-references, has-funder, has-license, member (a publisher’s member ID), and more. So from-pub-date:2022,has-orcid:true,has-full-text:true narrows to recent, ORCID-tagged, full-text-available work in one expression.

To pull a whole journal, switch modes and pass its ISSN:

{
  "mode": "journalWorks",
  "issn": "1476-4687",
  "filter": "from-pub-date:2023-01-01",
  "maxResults": 200,
  "mailto": "[email protected]"
}

And to resolve DOIs you already hold:

{
  "mode": "workByDoi",
  "dois": ["10.1038/s41586-021-03819-2", "10.1126/science.abm9891"],
  "mailto": "[email protected]"
}

The polite pool and pagination

Two Crossref mechanics matter more than anything else, and the actor handles both.

The polite pool. Crossref runs tiered service. Anonymous requests share a best-effort pool that gets throttled under load; requests that identify themselves with a mailto parameter join the polite pool, which offers higher, more predictable throughput. The mailto field is technically optional, but the README is emphatic: always supply it. It dramatically increases rate limits and reliability, and it’s the difference between a smooth 5,000-row run and a throttled crawl.

Cursor pagination. Crossref’s classic offset paging stops working past 10,000 results — request row 10,001 and you get an error, not data. The way around it is cursor-based pagination (the API’s cursor=* deep-paging mode), where each page returns a token for the next. The actor uses cursors automatically, so a broad query can retrieve far beyond 10,000 records — hundreds of thousands in principle — without you writing any of that loop.

▶ Open the Crossref Scholarly Scraper on Apify — search by keyword, pull a full journal by ISSN, or resolve a DOI list; 17 fields per work including citation count. Keyless, cursor pagination past 10,000 rows. Export to CSV, JSON or Excel.

Example output

A single work:

{
  "doi": "10.1038/s41586-021-03819-2",
  "title": "Highly accurate protein structure prediction with AlphaFold",
  "authors": "John Jumper; Richard Evans; Alexander Pritzel",
  "publishedDate": "2021-7-15",
  "containerTitle": "Nature",
  "publisher": "Springer Nature",
  "type": "journal-article",
  "citationCount": "23847",
  "referencesCount": "58",
  "subject": "Multidisciplinary",
  "url": "https://doi.org/10.1038/s41586-021-03819-2",
  "issn": "0028-0836, 1476-4687",
  "volume": "596",
  "issue": "7873",
  "page": "583-589",
  "license": "https://creativecommons.org/licenses/by/4.0",
  "funders": "Wellcome Trust; Google DeepMind"
}

Typical use cases

• Systematic reviews and meta-analyses — download 1,000+ recent papers on a topic (“CRISPR gene editing”) as a clean bibliometric dataset instead of hand-copying from a search UI.
• Journal auditing — pull every article from a journal by ISSN with a date filter and track its publication volume and citation growth over time.
• Grant and funder reporting — resolve a spreadsheet of 500 DOIs to full author and funder metadata for a report.
• Funding and publisher analysis — use the filter API to surface the top funders and publishers in a research area.
• Citation-graph construction — extract citationCount and referencesCount across a topic to build the skeleton of a citation network.
• ML training corpora — collect scholarly metadata and text pointers by subject to train or fine-tune models.

Filter recipes that actually narrow the noise

Most disappointing Crossref pulls come from a query that’s too broad and a filter that’s empty. The filter string is doing the heavy lifting, and a handful of combinations cover most real needs. All of these go in the filter field alongside your query:

• Recent journal articles only — from-pub-date:2023-01-01,type:journal-article. Drops preprints, datasets and book chapters, and cuts everything older than your window. This is the default you should reach for on any topic search.
• A precise time window — from-pub-date:2022-01-01,until-pub-date:2022-12-31. Bounds the search to a single year, which is how you build year-over-year publication-volume charts for a field.
• Open-access literature — has-license:true. Returns only works that deposited a license, which is a reasonable proxy for open access. Combine with a date filter for recent OA reviews.
• Author-disambiguated records — has-orcid:true. Keeps only works with at least one ORCID-tagged author, which is invaluable when you need to attribute papers to specific researchers without name-collision headaches.
• Funding analysis — has-funder:true. Restricts to works that deposited funder metadata, so your funders column is actually populated across the dataset rather than mostly null.
• A single publisher’s output — member:<member-id>. Crossref assigns each publisher a numeric member ID; filtering on it pulls one publisher’s works, useful for competitive or dominance analysis.

Filters are comma-separated and additive, so a serious query stacks several: from-pub-date:2022,type:journal-article,has-orcid:true,has-full-text:true. On the query text itself, be specific — “CRISPR therapeutic delivery 2023” returns tighter, more relevant results than a bare “CRISPR” that pulls in decades of tangential work. And when you already have exact DOIs from another dataset, skip search entirely and use workByDoi mode: it returns the most complete metadata per record because it addresses each work directly instead of ranking it out of a full-text search.

For AI-agent workflows, the actor is also exposed as an MCP tool, so a Claude or GPT agent can call it directly — “fetch the 200 most-cited journal articles on transformer attention published after 2021, then summarize the top funders” — and get structured JSON back to reason over, with no key management on the agent’s side.

Cost and effort math

The from-scratch version isn’t conceptually hard, but it’s a pile of small correctness traps: cursor pagination that must carry the token forward on every page, the polite-pool mailto header, joining Crossref’s date-parts arrays into a real date, flattening author objects into names, and gracefully handling the many records that omit license, funders or issn. Get any of those wrong and your dataset is subtly broken.

The managed actor is pay-per-result — a fraction of a cent per work returned plus a tiny per-run start fee — and Crossref itself charges nothing for API access. A 5,000-row bibliometric pull costs a few dollars and completes in a couple of minutes. Because Crossref is an open API that doesn’t block datacenter IPs, there’s no residential-proxy bill; a datacenter proxy is plenty.

Common pitfalls

Specific to Crossref metadata:

• Many fields are legitimately null. Metadata completeness depends on what each publisher deposits. license and funders are present in only roughly 30–60% of records; abstract, issn, volume and page are hit-or-miss. Nulls are expected, not errors — design your downstream schema to tolerate them.
• The 10,000-offset wall is real if you build your own. Classic offset paging dies at 10,000. The actor’s cursor paging steps past it, but if you ever query the API directly, plan for cursors from the start.
• citationCount and referencesCount are different things. citationCount (from is-referenced-by-count) is inbound — how many works cite this paper. referencesCount is outbound — how many references this paper lists. Don’t conflate them in a graph.
• Dates arrive as joined parts, not ISO. publishedDate can look like 2021-7-15 (unpadded month/day). Normalize it before sorting or bucketing by date.
• ISSN format matters in journalWorks mode. Use the hyphenated form (1476-4687), not the bare digits, or you’ll get zero results.
• Coverage skews to journal articles after 2000. Crossref indexes books, preprints and datasets too, but the deepest, most complete metadata is for post-2000 journal articles. Older or non-article works have thinner records.

Wrapping up

Crossref is one of the best open data sources in academia, and the price of admission is a handful of API mechanics — cursors, the polite pool, and the nested-metadata cleanup — that are easy to get almost right. If you’re pulling a one-off dataset for a review or a report, the managed actor lets you skip straight to clean rows in a spreadsheet. If you’re doing it on a schedule, wire the dataset to a webhook and let it refresh.

▶ Run the Crossref Scholarly Scraper on Apify — one run returns hundreds to thousands of works with DOIs, citation counts, funders and licenses across the 150M-record index. No API key. Start on Apify’s free monthly credit.