How to Scrape PubMed Abstracts, Authors & MeSH Terms (2026)

PubMed indexes more than 35 million biomedical citations, and for a systematic review, a bibliometric study, or an NLP training set you often need thousands of them as clean, structured rows — title, abstract, authors, MeSH terms, DOI. The website is built for reading one paper at a time. Exporting from it in bulk is clunky, and scraping the rendered pages is both fragile and unnecessary, because NCBI publishes an official API designed for exactly this. This guide covers how to pull PubMed records at scale through NCBI E-utilities: the two-step search-then-fetch pipeline, the fields you get, and the rate-limit reality of the keyless tier.

What’s worth extracting

The scraper outputs one structured row per article, with 12 fields:

• Identity — pmid (the PubMed unique identifier), title (full article title), and url (direct link to the article on PubMed).
• Content — abstract (full abstract text, which may include structured sections like Background/Methods/Results/Conclusions).
• Attribution — authors (semicolon-separated, LastName FirstName format) and journal (full journal title).
• Publication metadata — pubDate (YYYY-MM or YYYY-MM-DD), articleType (Journal Article, Review, Clinical Trial, …), doi (Digital Object Identifier), and pmcId (the PubMed Central ID, present when a free full-text version exists).
• Indexing terms — keywords (author-provided, semicolon-separated) and meshTerms (MeSH medical subject headings, semicolon-separated).

For a systematic review you need title, abstract, authors, journal, pubDate and doi. For an NLP corpus you want title, abstract and meshTerms as labels. The pmcId tells you which articles have free full text — you can build a link as https://www.ncbi.nlm.nih.gov/pmc/articles/{pmcId}/.

How the source actually works

The actor connects to NCBI’s public E-utilities API at eutils.ncbi.nlm.nih.gov and runs a two-step pipeline:

1. esearch.fcgi converts your keyword query into a paginated list of PMIDs.
2. efetch.fcgi retrieves the full XML records for those PMIDs in batches of up to 100, which the actor parses server-side into clean JSON.

Crucially, this uses the same search index as PubMed.gov, so full PubMed query syntax works verbatim — field tags, Boolean operators and publication-type filters all behave exactly as they do on the website.

A keyword search with a date window and abstracts looks like this:

{
  "query": "crispr gene editing cancer",
  "maxResults": 500,
  "dateFrom": "2022/01/01",
  "dateTo": "2024/12/31",
  "fetchAbstracts": true
}

If you already have a list of PMIDs — say, exported from a saved PubMed search — pass them directly and skip esearch entirely:

{
  "pmids": ["38912345", "38876543", "37654321", "36543210"],
  "fetchAbstracts": true
}

Query syntax is where the precision comes from. A few patterns that work because they’re standard PubMed tags:

• Field-scoped terms: KRAS[Gene Name] AND inhibitor[TI] AND 2023[PDAT]
• Boolean combinations (operators uppercase): (diabetes OR obesity) AND metformin AND clinical trial[PT]
• MeSH-anchored recall: cancer[MeSH] captures papers indexed under any cancer subheading, broader than a plain keyword.
• Author / journal / affiliation: Smith J[Author], Nature[Journal], Harvard[Affiliation].

Set fetchAbstracts: false for a fast metadata-only sweep when you don’t need the abstract text, and maxResults (1–10,000) to cap how deep the pagination goes.

▶ Run the PubMed Scraper on Apify — one run pulls thousands of biomedical articles with title, abstract, authors, DOI, MeSH terms and journal, by keyword or PMID. Keyless NCBI E-utilities, no login. Export to JSON, CSV, Excel or XML.

The access reality

E-utilities is genuinely built for automated querying, so the friction here is polite pacing rather than anti-bot walls:

• Keyless is three requests per second. NCBI allows up to three E-utilities requests per second without a key. The actor stays within that limit automatically and uses built-in retry logic for transient errors. (For heavier production use, NCBI offers a free API key that raises the ceiling — but the actor doesn’t need one.)
• maxResults caps at 10,000 per run. For a larger corpus, split the query by date range and run in parallel — separate jobs with different dateFrom/dateTo windows concatenate cleanly.
• Run time scales with volume. Because throughput is rate-limited, expect roughly 1–2 minutes for 200 articles, 5–8 minutes for 1,000, and 20–30 minutes for 5,000. That’s the rate limit, not a stall.
• Datacenter proxy is sufficient. NCBI doesn’t require residential IPs; the default datacenter proxy configuration works.
• It’s an API, not HTML scraping. The actor calls the official JSON/XML endpoints and never touches the PubMed website’s rendered pages, which is both faster and immune to front-end changes.

The naive alternative — scraping PubMed.gov search results — fights pagination and a changing DOM for data that E-utilities hands you as structured XML. There’s no reason to scrape the site when the API is this accessible.

Example output

One representative record (trimmed):

{
  "pmid": "39154321",
  "title": "Comparative immunogenicity of BNT162b2 and mRNA-1273 after booster vaccination in healthcare workers",
  "abstract": "We compared neutralizing antibody titers and T-cell responses in 312 healthcare workers who received a third dose of either vaccine ...",
  "authors": "Mendez A; Huang Y; Park S; Osei-Twum C; Hakim F",
  "journal": "Nature Medicine",
  "pubDate": "2024-01-09",
  "doi": "10.1038/s41591-023-02789-1",
  "pmcId": "PMC10876543",
  "keywords": "mRNA vaccine; booster; immunogenicity; COVID-19; healthcare workers",
  "meshTerms": "COVID-19; COVID-19 Vaccines; Antibodies, Neutralizing; Immunogenicity, Vaccine; Vaccination",
  "articleType": "Journal Article; Comparative Study",
  "url": "https://pubmed.ncbi.nlm.nih.gov/39154321/"
}

Use cases

Who pulls PubMed data in bulk and for what:

• Systematic literature reviews — pull hundreds to thousands of abstracts on a topic (e.g. immunotherapy in lung cancer, 2022–2024) for PRISMA-compliant screening pipelines.
• Competitive research intelligence — monitor all publications from a competitor research group or institution via an [Author] or [Affiliation] query.
• NLP training data — download thousands of titled, abstracted, MeSH-tagged articles to train biomedical language models or topic classifiers.
• Drug-target surveillance — track emerging research on a target with a query like (KRAS OR EGFR) AND inhibitor[TI] to watch new work on your compound.
• Grant-writing support — extract the most recent high-impact papers for a specific MeSH heading to build a background section fast.
• Bibliometrics & citation graphs — assemble metadata across a field for co-authorship networks and publication-trend analysis.

The actor is also MCP-compatible, so an AI agent can call it directly — “search PubMed for recent GLP-1 receptor agonist and weight-loss papers after 2022, get 300 results, and summarize the findings” — retrieve the dataset, and synthesize the literature without manual searching.

Build-it-yourself vs. managed actor

E-utilities is well-documented, so a two-step esearch → efetch script is a reasonable build. What accumulates:

• XML parsing. efetch returns nested PubMed XML; extracting abstracts (including structured sections), MeSH terms, author lists and DOIs cleanly into flat fields is more involved than it first looks.
• Rate-limit compliance. Staying under three requests per second, batching PMIDs into groups of 100, and retrying transient failures is glue you have to write and maintain.
• Pagination for large sets. Cycling through PMID pages to reach 5,000+ results, and splitting by date range for larger corpora, is manual orchestration.
• Export & scheduling. Wiring CSV/Excel export and recurring monthly sweeps.

The managed actor is pay-per-result and lightweight enough that thousands of searches stay cheap. If you’re comfortable owning the XML parser and pacing logic, build it — NCBI encourages programmatic use. If you want structured rows out of a query without writing the fetch-batch-parse loop, the managed path skips that work.

Common pitfalls

Specific gotchas with PubMed data:

• Not every article has an abstract. Conference abstracts, editorials, letters and older articles often lack indexed full text, so abstract will be an empty string. Don’t treat empty abstracts as scraping failures.
• MeSH terms lag for new papers. MeSH indexing is done manually by NLM staff and can trail publication by 6–12 months. Recent articles may have no meshTerms yet — if your query relies on [MeSH] tags, you’ll under-count the newest work.
• DOI isn’t universal. DOIs are populated when NCBI has them indexed (most articles from 2000 onward). Older and some non-English-journal articles may have no doi.
• [MeSH] recall differs from keyword recall. A MeSH query captures everything indexed under a heading and its subheadings, which is broader than a title/abstract keyword match. Choose deliberately: precision (keyword) vs. recall (MeSH).
• Date format is NCBI’s, not ISO. dateFrom/dateTo use YYYY/MM/DD (or YYYY), not YYYY-MM-DD. Getting the separators wrong silently returns the wrong window.
• pmids overrides query. If you pass both, the PMID list wins and the keyword search is skipped. Send one or the other for the run you intend.

Wrapping up

E-utilities is exactly the kind of API you want behind a scraping task — official, documented, and designed for automation. The work is the fetch-batch-parse pipeline and staying inside the rate limit. If you want that in your own stack, it’s a solid build. If you just need structured biomedical records out of a PubMed query — for a review, a corpus, or a monitoring feed — a managed actor that already handles the two-step pipeline, XML parsing and pacing gets you clean rows immediately.

▶ Open the PubMed Scraper on Apify — runs the NCBI E-utilities esearch → efetch pipeline to return title, abstract, authors, journal, DOI, PMC ID, MeSH terms and article type per article, by keyword or PMID. No API key, no login, MCP-ready. Export to JSON, CSV, Excel or XML. Start on Apify’s free monthly credit.