How to Scrape PyPI Package Downloads & Metadata in 2026

Every Python project’s supply chain lives on PyPI, but pulling structured intelligence out of it is annoyingly fragmented. Package metadata is on one endpoint, download counts are on a completely separate service, and the “top packages” ranking isn’t in PyPI’s API at all — it’s a community-maintained list. If you want a clean table of the top 500 packages with versions, licenses, dependencies and monthly downloads, you’re stitching three sources together and handling the fact that dependency arrays don’t fit a flat schema. This guide covers how those sources work and how to turn them into one dataset — no API key, no login.

What’s worth extracting

The scraper returns 15 fields per package. Grouped by what you’ll query:

• Identity and version — name, version (latest published stable), summary (one-line description), and pypiUrl (direct project page).
• Maintainer — author, authorEmail (when provided), homePage, and projectUrls (a JSON object of named links: Source, Documentation, Tracker, etc.).
• Dependencies and requirements — requiresDist (a JSON array of dependency specifiers) and requiresPython (the Python version constraint, e.g. >=3.8).
• Classification — keywords (comma-separated tags) and classifiers (a JSON array of Trove classifiers covering language, topic, and license).
• Licensing — license (SPDX identifier or license string like MIT, Apache 2.0).
• Traction and timing — recentDownloads (30-day count) and releaseDate (ISO 8601 upload timestamp of the latest release).
• Provenance — scrapedAt (ISO 8601 collection timestamp).

One structural note that matters downstream: to stay compatible with the flat output schema, projectUrls, requiresDist and classifiers are serialized as compact JSON strings, not native objects. Parse them with JSON.parse() (or your language’s equivalent) before you use them.

How the source actually works

The actor pulls from two official, open endpoints and one community list:

https://pypi.org/pypi/{package}/json                 # full metadata (PyPI JSON API)
https://pypistats.org/api/packages/{package}/recent  # 30-day downloads (PyPI Stats)

The top-packages ranking comes from the widely-used hugovk/top-pypi-packages dataset, updated monthly with the top 5,000 packages by 30-day downloads. None of these require authentication.

There are three modes. topPackages fetches the top-N most-downloaded packages from that community list, then enriches each in parallel:

{
  "mode": "topPackages",
  "topN": 200,
  "maxResults": 200
}

packages enriches an explicit list — paste a requirements.txt in here for an SBOM-style audit:

{
  "mode": "packages",
  "packages": ["requests", "numpy", "pandas", "flask", "django", "fastapi", "pydantic", "boto3"],
  "maxResults": 100
}

packageDetail is a single-package alias for quick lookups. For each package the actor resolves the latest stable version, the full metadata (author, license, classifiers, requires-dist, requires-python, project URLs), and the last-month download figure from PyPI Stats. It runs at bounded concurrency (6 workers) to stay within rate limits, retries with exponential backoff on 429 and 5xx, logs a warning and skips any package that fails, and exits cleanly even when zero valid packages are found.

▶ Open the PyPI Package Scraper on Apify — one run enriches the top-N packages, or a whole requirements.txt, with version, license, deps and 30-day downloads. Keyless, no login. Export to CSV, JSON or Excel.

The access reality

Both endpoints are designed for programmatic access, so the friction is about rate limits and data joins, not anti-bot walls:

• Two services, one join. Metadata (PyPI JSON API) and downloads (PyPI Stats) are separate systems. Every enriched row is a join across both — and PyPI Stats is the slower, flakier of the two, so overall run time tracks its response times.
• recentDownloads can be null. PyPI Stats may return no data for very new packages (under 30 days old) or extremely low-download packages. The field is set to null rather than 0 in those cases — a null means “no data,” not “zero downloads.”
• Rate limits are real. Hitting PyPI and PyPI Stats hard from one IP will earn 429s. The actor’s bounded 6-worker concurrency plus exponential backoff is what keeps a 5,000-package run inside the limits.
• pypi.org only. The actor is hardcoded to pypi.org and pypistats.org. Private PyPI mirrors are out of scope without a custom fork.

The naive approach — a script calling both APIs in a tight loop — gets rate-limited fast, has no backoff, and dies on the first 404. The actor’s concurrency ceiling, retry logic, and graceful skip-on-failure are exactly the parts that make a large sweep survive.

Example output

One representative record:

{
  "name": "requests",
  "version": "2.31.0",
  "summary": "Python HTTP for Humans.",
  "author": "Kenneth Reitz",
  "authorEmail": "[email protected]",
  "license": "Apache 2.0",
  "homePage": "https://requests.readthedocs.io",
  "projectUrls": "{\"Documentation\":\"https://requests.readthedocs.io\",\"Source\":\"https://github.com/psf/requests\"}",
  "requiresDist": "[\"charset-normalizer<4,>=2\",\"idna<4,>=2.5\",\"urllib3<3,>=1.21.1\",\"certifi>=2017.4.17\"]",
  "requiresPython": ">=3.7",
  "keywords": "requests, http, https, rest, api",
  "releaseDate": "2023-05-22T15:12:44",
  "recentDownloads": 87432109,
  "pypiUrl": "https://pypi.org/project/requests/",
  "classifiers": "[\"License :: OSI Approved :: Apache Software License\"]",
  "scrapedAt": "2026-07-08T10:00:00.000Z"
}

Notice projectUrls, requiresDist and classifiers are strings holding JSON — parse them before use.

Typical use cases

Who pulls PyPI data and for what:

• Dependency auditing / SBOM — pull metadata for every package in a requirements.txt and flag those with missing license or an incompatible requiresPython.
• Ecosystem mapping — grab the top 500 packages and group by classifiers (web framework, ML, data, etc.) to visualize the Python landscape.
• Download-trend tracking — schedule weekly runs and diff recentDownloads over time to spot packages gaining or losing momentum.
• Tech-stack fingerprinting — enrich packages detected via import analysis to get author, homepage and project URLs.
• License compliance — read the License :: OSI Approved :: ... entries in classifiers for accurate license identification, more reliable than the free-text license field alone.
• OSS intelligence and outreach — identify active maintainers and cross-reference authorEmail against a CRM to find which open-source authors are already users or prospects.

Cost and effort: build vs. managed

The DIY version starts easy — two public APIs and a JSON list — but the production layer is the work: joining metadata to download stats, bounded concurrency to avoid 429s, exponential backoff and retry, graceful skip on 404, JSON-string serialization to fit a flat schema, and pulling and refreshing the hugovk/top-pypi-packages list. That’s a small pipeline that breaks quietly when PyPI Stats throttles you.

The managed actor is pay-per-result: a small per-run start fee plus a fraction of a cent per package row, no subscription (PyPI and PyPI Stats are free). The README puts throughput at roughly 100 packages per minute — a 500-package run in about 5 minutes, a full 5,000-package scan in 30–45 minutes depending on PyPI Stats latency. Because you pay per record, a targeted packages-mode audit of your own dependencies costs a rounding error.

Common pitfalls

Specific to PyPI:

• Parse the JSON-string fields. projectUrls, requiresDist and classifiers are strings, not objects. JSON.parse() them before building a dependency tree or reading URLs.
• null downloads ≠ zero. A null recentDownloads means PyPI Stats had no data (new or tiny package), not that nobody installed it. Don’t treat null as 0 in rankings.
• requiresDist can be null entirely. Some packages (see numpy in the README sample) publish no dependency list in the metadata. Handle the null before parsing.
• Trust classifiers for license over license. The free-text license field is inconsistent (MIT, Apache 2.0, full text, or blank). The License :: OSI Approved :: ... classifier is the structured, reliable signal.
• The top list is monthly, not live. topPackages pulls from a list refreshed monthly — it’s the freshest public ranking, but it isn’t a real-time leaderboard.
• No in-actor filtering by license or Python version. Apply those filters downstream (Pandas, Google Sheets, a Zapier step) using the license and requiresPython fields — the actor returns everything and you slice it.

Wrapping up

PyPI intelligence is really a three-source join — metadata, download stats, and a community ranking — wrapped in enough rate-limit engineering to survive a large sweep. If you’d rather not build the concurrency, backoff and JSON-normalization layer, the managed actor turns a mode and a package list into a clean, export-ready table, keyless.

▶ Run the PyPI Package Scraper on Apify — one run enriches up to 5,000 packages with version, license, dependencies, Python requirements and 30-day downloads. Keyless, no login. Export to CSV, JSON, Excel or XML. Start on Apify’s free monthly credit.