How to Scrape arXiv Papers, Abstracts & Metadata in 2026

arXiv hosts over 2.5 million open-access preprints across computer science, physics, mathematics, quantitative biology, statistics, and economics. The metadata is all public and there’s an official API — but that API returns Atom XML, paginates in a way you have to manage yourself, and enforces polite request pacing. If you’ve tried to assemble a few thousand papers for a literature review or an NLP dataset by hand, you’ve felt the gap between “the data is open” and “the data is in my CSV.” This guide covers what arXiv exposes, how its API is shaped, and how to pull structured paper records at scale.

What’s worth extracting

Each paper resolves to a clean, flat record of about 15 fields. Grouped by what you’ll actually use:

• Identity — arxivId (e.g. 2401.12345), title, and the arxivUrl abstract landing page.
• Content — the complete abstract (full text, not a snippet), ready for embeddings, summarization, or topic modeling.
• Authorship — authors, a comma-separated list of every co-author on the paper.
• Taxonomy — categories (all arXiv codes assigned) and primaryCategory (the single main one, e.g. cs.CL).
• Timestamps — publishedDate (original v1 submission) and updatedDate (most recent revision).
• Files — pdfUrl, the direct PDF download link for the paper.
• Publication metadata — doi, journalRef, and the author comment (page count, figures, accepted venue, code links) when the authors provide them.
• Provenance — searchQuery, the term that surfaced this paper, and a scrapedAt timestamp.

The searchQuery tag is worth calling out: because you can pass many search terms in one run, each returned paper is stamped with the query that found it, so you can trace any row back to its search without a separate bookkeeping step.

How the source actually works

The scraper talks directly to arXiv’s official public API at export.arxiv.org — the Atom/OAI-PMH-compatible endpoint arXiv provides specifically for programmatic access. There is no browser, no HTML parsing, and no key. You give the actor one or more searchQueries and it queries each term independently, pages through the results, and maps the Atom XML into clean fields.

A realistic dataset-building run — recent AI and NLP preprints, filtered to the categories that matter — looks like this:

{
  "searchQueries": ["large language models", "retrieval augmented generation"],
  "categories": ["cs.AI", "cs.CL", "cs.LG"],
  "sortBy": "submittedDate",
  "dateFrom": "2026-01-01",
  "maxResults": 500
}

The input surface is small and maps directly onto the API’s capabilities:

• searchQueries (required, array) — keywords, a topic, a paper title, or an author name. Each term is queried on its own.
• categories (array) — arXiv taxonomy codes like cs.AI, cs.CL, cs.CV, stat.ML, physics.optics, or q-bio.GN. Leave empty to search all fields.
• maxResults (integer, 1–1000) — total papers across all queries.
• sortBy — relevance, lastUpdatedDate, or submittedDate.
• dateFrom — keep only papers submitted on or after this date (YYYY-MM-DD).

Tracking a single researcher is just a query with the author’s name:

{
  "searchQueries": ["Yann LeCun"],
  "sortBy": "submittedDate",
  "maxResults": 200
}

Combining a focused query term with one or two categories gives the cleanest, most on-topic result set — a bare keyword across all of arXiv will drag in tangential matches. A broad topic sweep, by contrast, leans on the multi-query design: pass several terms in one run and let each pull its own ~1,000 papers.

{
  "searchQueries": ["quantum computing", "diffusion models", "protein folding"],
  "sortBy": "relevance",
  "maxResults": 300,
  "proxyConfiguration": { "useApifyProxy": true }
}

This is the intended path to a large corpus — many narrow queries rather than one impossibly broad one — and the searchQuery tag on every row keeps the provenance intact so you can slice the dataset back apart by term later.

▶ Open the arXiv Paper Scraper on Apify — turn a list of topics or author names into a structured paper dataset with full abstracts, author lists, categories, PDF links and DOIs. No key, no browser. Export to CSV or JSON.

The access reality

arXiv’s API is free and open — the friction is about volume limits and politeness, not anti-bot defenses:

• 1,000 papers per query, unlimited queries. A single query tops out at ~1,000 results, but there’s no cap on how many query terms you pass in one run. To build a dataset of tens of thousands of papers, combine many search terms rather than trying to page one query past its ceiling.
• Polite pacing is mandatory. arXiv publishes polite-use guidance and expects programmatic clients to throttle. The actor fetches pages sequentially with built-in delays and retries so you stay within that guidance without engineering it yourself.
• Proxy-friendly, datacenter is fine. arXiv does not treat datacenter IPs as hostile. Apify Proxy integration is available (proxyConfiguration) but you don’t need residential IPs.
• Preprint metadata is partial by nature. DOIs and journal references only exist once (and if) a paper is formally published, so those fields are frequently empty on fresh preprints — that’s the data, not a scraper bug.

Example output

One representative record:

{
  "arxivId": "2401.12345",
  "title": "Scaling Laws for Retrieval-Augmented Language Models",
  "authors": "Jane Doe, John Smith, Alice Zhang",
  "abstract": "We study how retrieval augmentation changes the scaling behavior of large language models across compute, data and parameter budgets...",
  "categories": "cs.CL, cs.AI, cs.LG",
  "primaryCategory": "cs.CL",
  "publishedDate": "2024-01-22T18:00:04Z",
  "updatedDate": "2024-02-15T09:31:12Z",
  "pdfUrl": "https://arxiv.org/pdf/2401.12345",
  "arxivUrl": "https://arxiv.org/abs/2401.12345",
  "comment": "18 pages, 7 figures. Accepted at ICML 2024.",
  "journalRef": "Proceedings of ICML 2024",
  "doi": "10.48550/arXiv.2401.12345",
  "searchQuery": "large language models",
  "scrapedAt": "2026-07-06T12:00:00.000Z"
}

Typical use cases

• Systematic literature reviews — pull every paper matching your topic and categories, filter by date, and export structured metadata instead of copy-pasting from the site.
• AI & tech intelligence — monitor the categories your team cares about to track emerging methods and competitor research output.
• Talent sourcing — search by domain, then mine the authors field for researchers publishing in your area; every co-author is a potential hire.
• NLP / ML dataset building — assemble titles and full abstracts for text classification, topic modeling, retrieval benchmarks, or citation-graph work.
• Trend & recency monitoring — sort by submittedDate with a rolling dateFrom to keep only fresh preprints and catch new work as it lands.
• Bibliometrics — join doi / journalRef with other sources to enrich a scholarly-publications database.

Build it yourself vs. use the actor

arXiv’s API is documented and free, so writing a client is entirely doable. What you actually maintain is the annoying part: an Atom XML parser, per-query pagination, multi-query batching, the retry/backoff loop that keeps you inside arXiv’s polite-rate guidance, date filtering, and the field mapping that turns nested XML into flat columns. That’s a small library you now own and have to keep green.

The managed actor is that library, already written. Pricing is pay-per-result — a small fraction of a cent per paper plus a negligible start fee — and the arXiv data itself is free. For a 500-paper review that’s a rounding-error cost, and you skip the XML-parsing and rate-limit engineering entirely.

Common pitfalls

Things to know before you commit to an arXiv pipeline:

• publishedDate vs updatedDate. publishedDate is the original v1 submission; updatedDate is the latest revision. Sort by lastUpdatedDate to catch papers revised with new results, not just newly submitted ones.
• Empty DOI / journalRef is normal. Preprints often haven’t been formally published, so these populate only later. Don’t filter them out expecting every paper to have a DOI.
• The 1,000-per-query ceiling. Don’t try to force one query past its limit. Split a broad topic into narrower queries (add categories, tighten the term) and let the multi-query batching do the volume.
• Author-name ambiguity. Searching an author name matches on the name string, so common names can pull in a different researcher. Combine with a category to disambiguate.
• PDF links, not PDF text. The actor returns the pdfUrl for every paper; full-text extraction from the PDFs is out of scope. Download them separately from those links if you need the body text.
• Category codes are exact. arXiv codes are case- and format-specific (cs.CL, not cs.cl). Browse the arXiv category taxonomy to get the precise code for your field.

Wrapping up

The arXiv API is a genuine public good, but it hands you Atom XML and a politeness contract, not a dataset. If you need a one-off pull, the endpoint is open and worth learning. If you need refreshed research metadata on a schedule — new preprints in your field every day, a reproducible corpus for training — let the actor own the XML parsing, pagination, and pacing and hand you export-ready rows.

▶ Run the arXiv Paper Scraper on Apify — queries arXiv’s official public API directly: titles, full abstracts, complete author lists, categories, PDF links and DOIs, up to 1,000 papers per query with unlimited queries. No API key, no browser. Start on Apify’s free monthly credit.