How to Scrape Crates.io Rust Package & Download Data

If you want to understand the Rust ecosystem — which crates dominate a category, how download counts trend, what a set of dependencies actually pulls in — crates.io is the source. It’s the official Rust package registry, home to 145,000+ published crates, and it exposes a fully public REST API. The good news is there’s no key and no login. The awkward part is that pulling a ranked list of the top 1,000 crates, or metadata for a curated dependency list, means paginating the API 100 records at a time, respecting the registry’s crawler etiquette, and flattening JSON into rows you can actually analyze. This guide covers what the API gives you and how to pull it cleanly.

What’s worth extracting

The scraper returns 13 fields per crate, straight from the crates.io API. They cover identity, popularity, and provenance:

• name — the crate’s unique registry name (e.g. serde).
• description — the author-provided short description.
• downloads — all-time total download count.
• recentDownloads — downloads in the last 90 days, a better proxy for current adoption than the cumulative total.
• maxVersion — the highest published semver version (excluding pre-releases).
• newestVersion — the most recently published version. For stable crates these two usually match; for crates in an active pre-release cycle they diverge.
• homepage — project homepage URL (often null).
• repository — source repository URL, usually GitHub or GitLab.
• documentation — docs.rs or a custom docs URL.
• categories — comma-separated category slugs (e.g. web-programming, http-client).
• keywords — comma-separated author-defined keywords.
• createdAt / updatedAt — ISO 8601 timestamps for first publish and most recent update.

How the source works

The scraper talks directly to the official crates.io public REST API at https://crates.io/api/v1/. It runs in three modes, each mapping to a way you’d want to slice the registry:

• list — paginate the whole registry ranked by a sort order. Sorts are downloads, recent-downloads, new-crates, updated, or alpha. This is how you build a “top N crates” dataset.
• search — a full-text query against crate names and descriptions, for niche discovery.
• crateDetail — fetch precise metadata for a known list of crate names, returning richer category and keyword taxonomy from the per-crate detail endpoint.

A top-crates run is as simple as:

{
  "mode": "list",
  "sort": "downloads",
  "maxResults": 1000
}

A keyword search, ranked by downloads:

{
  "mode": "search",
  "query": "async http client",
  "sort": "downloads",
  "maxResults": 200
}

And a targeted detail fetch for a known dependency set:

{
  "mode": "crateDetail",
  "crateNames": ["serde", "tokio", "reqwest", "axum", "clap"]
}

Each page request returns up to 100 crates, and the scraper paginates automatically until it hits your maxResults. Because the API is public and keyless, no credentials are needed — but a descriptive User-Agent header is sent on every request, which is exactly what crates.io’s published API guidelines ask bots to do.

Chaining modes for richer data

The three modes compose well, and the most useful pattern combines two of them. search and list return the standard crate fields quickly, but crateDetail hits the per-crate endpoint, which resolves the full category and keyword taxonomy rather than the abbreviated version in list results. So the high-fidelity workflow is: run a search to discover the crates relevant to your topic, collect their name values, then feed those names into a crateDetail run to pull the richer metadata. You get the breadth of search with the depth of the detail endpoint, without paginating the whole registry. The same trick works for tracking a fixed set of crates over time — a scheduled crateDetail run on a known name list gives you a clean download-count time series.

The access reality

crates.io is a well-behaved public registry, not a hostile scraping target. The constraints are about etiquette and pagination, not anti-bot walls:

• Crawler etiquette is a published policy, not a suggestion. crates.io asks bots to identify with a descriptive User-Agent and avoid hammering the API. The scraper enforces a 500 ms delay between requests and retries on 429 (Too Many Requests) automatically. That politeness is why runs stay reliable rather than getting rate-limited.
• 100 records per page. The API returns crates in pages of 100. A 1,000-crate run is roughly 10 requests — with the polite delay, that’s a few seconds of wait plus network latency, typically under 30 seconds total.
• The full registry is 145,000+ crates. A single run caps at maxResults (up to 10,000). To cover the entire namespace, you run multiple times with different sort orders — alpha is useful for walking the full name space.
• Fastly CDN, rare blocks. crates.io sits behind Fastly and rarely blocks well-behaved clients. A datacenter proxy is optional; enabling it just improves reliability for very large runs.
• Download counts lag up to 24 hours. Counts are updated periodically (typically daily) and reflect cumulative CDN-served downloads. They’re not real-time — plan any trend analysis around that cadence.

Example output

A representative record for tokio:

{
  "name": "tokio",
  "description": "An event-driven, non-blocking I/O platform for writing asynchronous I/O backed applications.",
  "downloads": 750000000,
  "recentDownloads": 38000000,
  "maxVersion": "1.37.0",
  "newestVersion": "1.37.0",
  "homepage": null,
  "repository": "https://github.com/tokio-rs/tokio",
  "documentation": "https://docs.rs/tokio",
  "categories": "asynchronous, network-programming",
  "keywords": "io, async, non-blocking, futures",
  "createdAt": "2016-08-04T00:00:00Z",
  "updatedAt": "2024-03-01T12:00:00Z"
}

Note homepage is null here — that field, along with documentation, is optional in the crates.io publishing spec and many authors omit it.

Use cases

• Ecosystem research — compile a ranked list of the top 1,000 crates by total downloads to map the shape of the Rust ecosystem, popular categories, and dependency trends.
• Competitive benchmarking — search a niche (async http client, serialization) and compare download counts and update frequency to pick the strongest library, or to see where your own crate ranks.
• Dependency auditing — pull version history and repository links for a curated list of internal dependencies, feeding a governance dashboard or SCA database.
• Trend tracking — schedule weekly runs with sort=recent-downloads to build a download-velocity time series for a set of crates.
• AI-agent workflows — feed crate metadata into an agent (the actor is available as an MCP tool) to generate dependency health reports or README summaries.

Cost and effort: build vs. managed

The crates.io API is free and keyless, so this is a question of time, not proxy bills:

• Building from scratch — an afternoon to learn the endpoints and write pagination, plus the polite-delay and retry-on-429 logic that keeps you within crawler etiquette. Then flattening the nested category/keyword arrays into clean columns, and the multi-run logic if you want the whole registry.
• Using the managed actor — pick a mode, set maxResults, run. Pricing is pay-per-result: a fraction of a cent per crate, plus a negligible per-run start fee. A 1,000-crate ranking costs a few cents and finishes in well under a minute.

The etiquette layer (User-Agent, 500 ms pacing, 429 retries) is the part people skip when building quickly — and it’s the part that gets you rate-limited. The managed actor bakes it in.

Common pitfalls

Specifics of crates.io that will catch you:

• maxVersion vs. newestVersion are not the same field. maxVersion is the highest stable semver; newestVersion is whatever was published most recently, which could be a 1.0.0-beta.1 pre-release. For stable crates they match; for crates in active pre-release, they diverge. Pick the one that matches your definition of “current.”
• Null fields are normal for small crates. homepage, documentation, categories, and keywords are all optional. Many utility crates omit them. If you need only fully-populated records, filter nulls in post-processing rather than treating them as scrape failures.
• recentDownloads is a 90-day window, not “this week.” It’s the last 90 days as reported by the API. It’s the right field for “is this crate currently adopted,” but don’t read it as a weekly figure.
• Download counts are cumulative and lag. All-time downloads reflect CDN-served totals updated roughly daily. For a velocity signal, diff recentDownloads (or cumulative totals) across scheduled runs rather than expecting live numbers.
• Categories are strings, not arrays. categories and keywords come back as comma-separated strings. Split on the comma to filter — easy in a spreadsheet or with a quick str.split in pandas.
• Covering the full 145K registry needs multiple runs. A single run tops out at 10,000. To walk the entire namespace, chain runs across different sort orders (alpha is the cleanest for full coverage).

Wrapping up

crates.io is a friendly, open registry, which makes Rust ecosystem data unusually accessible — the friction is pagination and crawler etiquette, not access. If you just want to poke at one crate, curl the API. If you want a ranked dataset of the top crates, a niche search, or a repeatable dependency-audit feed, a managed actor handles the pacing, retries, and flattening so your data lands clean.

▶ Open the Crates.io Scraper on Apify — one run yields hundreds to thousands of Rust crates with downloads, versions, categories, and repo URLs; no API key, no login. Export to CSV, JSON, or Excel, and start on Apify’s free monthly credit.