How to Scrape PubChem: Compounds, Properties & SMILES

PubChem is the largest open chemical database on the planet, run by the NCBI at the US National Institutes of Health. Its PUG REST API is free and keyless, so pulling one compound is trivial. Pulling a thousand is not: you have to resolve names to CIDs, fan out to two different endpoints per compound (one for properties, one for synonyms), throttle to NCBI’s fair-use limit so you don’t get blocked, and handle the compounds that simply aren’t found. This guide covers how the PUG REST API is shaped, the three ways to look up compounds, and how to build a clean chemical dataset in bulk without writing the rate-limiter yourself.

What’s worth extracting

The actor returns 10 fields per compound, the core identifiers and properties cheminformatics pipelines actually consume:

• Identity — cid (the PubChem Compound ID), name (the input name or CID used for the lookup), iupacName (the official IUPAC name, e.g. “2-acetyloxybenzoic acid”), pubchemUrl (a direct link to the compound page).
• Structure — molecularFormula (Hill notation, e.g. “C9H8O4”), canonicalSMILES (the SMILES string for structure representation), inchiKey (the InChIKey hash for exact compound identification).
• Physicochemical — molecularWeight (g/mol) and xlogp (computed octanol-water partition coefficient, the log P estimate used in drug-likeness assessment).
• Nomenclature — synonyms (a semicolon-separated list of up to 10 synonyms and trade names).

The canonicalSMILES and inchiKey fields are the ones that make this dataset composable — SMILES feeds directly into RDKit and similar toolkits, and the InChIKey is the canonical key for de-duplicating and cross-referencing compounds across datasets.

How the source actually works

The actor connects to the official NCBI PUG REST API at pubchem.ncbi.nlm.nih.gov/rest/pug. It supports three lookup modes:

• byName — resolve a list of chemical names to CIDs, then fetch full metadata. PubChem resolves synonyms, trade names and CAS numbers automatically, so “aspirin”, “acetylsalicylic acid” and a CAS number all land on the same compound.
• byCid — you already have numeric CIDs from another pipeline; skip name resolution and go straight to metadata (faster).
• search — keyword search that returns all matching compounds, good for exploring a chemical family like “penicillin” or “vitamin”.

For each compound the actor makes two calls: properties come from the /compound/.../property/ endpoint, synonyms from /compound/.../synonyms/. A name-based input looks like this:

{
  "mode": "byName",
  "names": ["aspirin", "caffeine", "ibuprofen", "glucose", "paracetamol"],
  "proxyConfiguration": { "useApifyProxy": true }
}

If you hold CIDs, use byCid:

{
  "mode": "byCid",
  "cids": ["2244", "2519", "3672", "5793", "1983"]
}

And to explore a family, use search with a cap:

{
  "mode": "search",
  "searchQuery": "penicillin",
  "maxResults": 50,
  "proxyConfiguration": { "useApifyProxy": true }
}

Results are streamed to the dataset as they arrive, so you can start consuming data while a long run is still going.

The access reality

PubChem is a government API built for programmatic access, so there’s no anti-bot wall — the constraints are politeness and data completeness:

• NCBI’s rate limit is roughly 5 requests per second per source. The actor throttles to at most 5 concurrent requests with a 200 ms delay between calls, and retries transient errors up to 3 times with exponential back-off. That pacing is why throughput lands around 10–15 compounds per minute; a 1,000-compound run takes roughly 70–100 minutes. byCid is faster than byName because it skips the name-resolution round trip.
• Datacenter proxy is sufficient. PubChem doesn’t block datacenter IPs, so residential proxy is unnecessary and would only add cost.
• Not every compound has every field. xlogp and iupacName in particular can be absent for inorganic compounds, salts, or newly added records without full computational data. That’s expected, not an error.
• Two endpoints per compound. Because properties and synonyms come from separate endpoints, each compound is two API calls — factor that into your rate-limit math for very large lists.

▶ Open the PubChem Scraper on Apify — look up compounds by name, CID or keyword; 10 fields per compound including SMILES, InChIKey and XLogP. Keyless NCBI PUG REST API, no login. Export to CSV, JSON or Excel.

Example output

A single compound:

{
  "cid": "2244",
  "name": "aspirin",
  "iupacName": "2-acetyloxybenzoic acid",
  "molecularFormula": "C9H8O4",
  "molecularWeight": "180.2",
  "canonicalSMILES": "CC(=O)Oc1ccccc1C(=O)O",
  "inchiKey": "BSYNRYMUTXBXSQ-UHFFFAOYSA-N",
  "xlogp": "1.2",
  "synonyms": "Aspirin; acetylsalicylic acid; Ecotrin; Bayer Aspirin; Disprin; 2-acetoxybenzoic acid; ASA; Aspro; Anacin; Entrophen",
  "pubchemUrl": "https://pubchem.ncbi.nlm.nih.gov/compound/2244"
}

Typical use cases

• Drug pipeline screening — feed 500 candidate CIDs and extract SMILES plus XLogP in one run for Lipinski rule-of-five filtering.
• Ingredient label auditing — input a list of cosmetic or food ingredient names and retrieve canonical InChIKeys for regulatory cross-referencing.
• Teaching and coursework — fetch molecular formulas and IUPAC names for an entire curriculum’s compound list from a single input.
• Similarity-search prep — extract canonicalSMILES for a set of scaffolds to feed into RDKit or another cheminformatics toolkit.
• Synonym enrichment — pull trade names, CAS numbers and other synonyms for the compounds in a product catalog.
• Knowledge graphs and ML sets — build a labeled chemical dataset keyed on CID and InChIKey for downstream modeling.

Making sense of the fields you pull

A PubChem dataset is only useful if you know what each field is for. Three of them do most of the work in downstream chemistry.

XLogP and drug-likeness. xlogp is a computed estimate of the octanol-water partition coefficient (log P) — how lipophilic, or fat-soluble, a molecule is. Higher values mean more lipophilic; lower (and negative) values mean more hydrophilic. It’s one of the pillars of Lipinski’s rule of five, the rough screen for oral drug-likeness, where an XLogP much above 5 is a red flag for poor absorption. Oral drugs typically sit between roughly −2 and 5. If you’re feeding candidates into a rule-of-five filter, xlogp plus molecularWeight and molecularFormula are the columns you filter on. Just remember it’s null for many inorganic compounds and salts — those aren’t rule-of-five candidates anyway.

InChIKey as the join key. inchiKey is a fixed-length hash derived from the compound’s structure, and it’s the field you should de-duplicate and cross-reference on — not the name, which is ambiguous, and not always the CID, which can differ across data sources. Two records with the same InChIKey are the same compound, full stop. When you merge a PubChem pull with an internal library or a regulatory list, join on InChIKey.

SMILES for the toolchain. canonicalSMILES is the machine-readable structure string that RDKit, Open Babel and similar cheminformatics libraries ingest directly. Pull SMILES for a set of scaffolds and you can compute fingerprints, run similarity searches or generate 2D depictions downstream without ever touching PubChem’s UI.

Choosing the right mode

The three modes exist because chemists arrive with different starting points:

• You have names (drug names, ingredient labels, common names) — use byName. PubChem’s name resolution is generous: trade names, systematic names and CAS numbers all resolve to the right CID. The cost is one extra round trip per compound for the resolution step.
• You have CIDs from a prior pipeline — use byCid. It skips resolution and is meaningfully faster, and it removes the ambiguity risk that a vague common name might resolve to the wrong compound.
• You want to explore a family (all the penicillins, all the vitamins) — use search with a searchQuery and a maxResults cap. Keep the cap modest (a few hundred) for a manageable run time, since search can expand to thousands of matching CIDs.

When precision matters — a regulatory cross-check, a specific salt form — prefer byCid with the exact PubChem ID over a name that might be interpreted loosely. Results stream to the dataset as they arrive in every mode, so a long byName run is usable before it finishes, and you can pipe the dataset to Google Sheets or a webhook for collaborative review.

Cost and effort math

The from-scratch version is mostly rate-limiting and error-handling discipline: a concurrency limiter that respects NCBI’s 5-req/sec ceiling, the two-endpoint fan-out per compound, name-to-CID resolution, graceful handling of 404s for compounds that don’t exist, exponential back-off on transient errors, and streaming results so a 90-minute run isn’t all-or-nothing. None of it is exotic, but getting it robust — and keeping it that way — is real work.

The managed actor is pay-per-result — a fraction of a cent per compound plus a tiny per-run start fee — and PubChem itself is free. A few-hundred-compound lookup costs well under a dollar. Because datacenter proxy is enough, there’s no residential-proxy bill. For a one-off library build you’d never write the throttling glue; for a recurring cross-check (say, a weekly ingredient-database refresh) you schedule the actor.

Common pitfalls

Specific to PubChem chemical data:

• XLogP is legitimately null for many compounds. Inorganic compounds and salts often have no computed log P. A null xlogp is expected — don’t treat it as a failed row.
• The synonym list is capped at 10 entries. Some compounds have hundreds of synonyms; the actor returns the top 10, which are the most useful. If you need the exhaustive list, that’s beyond this field.
• IUPAC names can be very long. For polymers and complex molecules iupacName can exceed 200 characters. Store it in a text column, not a fixed-width field.
• A missing compound just gets skipped. If a name is misspelled or not indexed, the actor logs a warning (“No CIDs found for name:”) and moves on — the compound won’t appear in the dataset. Check the log if your row count is lower than your input list.
• Name resolution is one-to-one, but ambiguous names exist. A vague common name may resolve to a different CID than you intended. When precision matters, use byCid with the exact PubChem ID.
• Mind the throughput on huge lists. At two calls per compound and NCBI’s politeness pacing, a 5,000-compound run is measured in hours, not minutes. Split very large jobs and set memory to at least 1 GB.

Wrapping up

PubChem is a first-class open data source, and the only things between you and a clean compound dataset are NCBI’s rate limits and the two-endpoint fan-out — exactly the kind of boilerplate worth handing off. If you need SMILES, formulas and log P values for a candidate list, an ingredient audit or a teaching set, the managed actor turns a list of names or CIDs into an exportable table without you writing a throttler.

▶ Run the PubChem Scraper on Apify — one run resolves thousands of names or CIDs to formula, molecular weight, SMILES, InChIKey and XLogP. No API key. Start on Apify’s free monthly credit.