How to Extract Wayback Machine Archived URLs in 2026

When a site gets migrated, redesigned, or quietly deleted, the URLs don’t vanish — they live on in the Internet Archive’s Wayback Machine. That makes it the single best source for recovering a domain’s full historical URL footprint: the deleted blog posts, the retired product pages, the old endpoints that still earn backlinks. The Archive exposes this through its public CDX API, but querying it by hand is genuinely tedious. You have to construct CDX query strings, handle resumeKey pagination for large sites, de-duplicate captures, and build snapshot links yourself. This guide covers what the CDX API exposes and how to pull a clean, de-duped URL inventory for any domain.

What’s worth extracting

The extractor returns one row per unique archived URL — de-duplicated so each URL appears once, not once per capture. Each row carries eight fields:

• domain — the normalized domain this URL belongs to.
• url — the original archived URL.
• timestamp — the raw 14-digit Wayback capture timestamp (YYYYMMDDhhmmss).
• capturedAt — the ISO 8601 form of that timestamp.
• statusCode — the HTTP status the archive recorded for that capture (200, 301, 404, or -).
• mimeType — the content type recorded at capture time (e.g. text/html).
• digest — the Wayback content digest, used internally for de-duplication.
• snapshotUrl — a direct, ready-to-open link to the archived snapshot on web.archive.org.

The snapshotUrl is the field that makes the output immediately useful: every row links to the exact archived page, so you can open any recovered URL and see what was there.

How the source works

The extractor is a front end over the Internet Archive’s CDX API — the index of everything the Wayback Machine has captured, going back to 1996. You give it a domain and it builds the CDX query, pages through the results, and constructs snapshot links for each row.

A full historical inventory of a domain looks like this:

{
  "domains": ["nasa.gov"],
  "matchType": "subdomains",
  "maxResults": 0,
  "proxyConfiguration": { "useApifyProxy": true }
}

The matchType field controls scope, and it maps to the CDX API’s matching modes:

• subdomains (default, broadest) — the host plus all subdomains and paths.
• host — the exact host only.
• domain — the exact domain.
• prefix — URLs starting with a given path prefix.

You can also filter by capture window and status. A redirect-map run — only live 200 pages from a date range — looks like this:

{
  "domains": ["example.com"],
  "matchType": "subdomains",
  "fromDate": "20100101",
  "toDate": "20201231",
  "filterStatus": "200",
  "maxResults": 5000
}

fromDate and toDate take YYYYMMDD bounds; filterStatus keeps only captures with a given HTTP status. Under the hood, the query requests the original, timestamp, statuscode, mimetype, and digest fields with collapse=urlkey, so each URL appears once instead of returning every capture of it. Domains are normalized first — scheme, www., paths, and wildcards are reduced to a bare host.

▶ Run the Wayback Machine URL Extractor on Apify — one run turns a single domain into its full historical URL inventory, each with a capture date and a direct snapshot link. No API key, no login. Export to CSV or JSON.

The access reality

The CDX API is public and free, but it has real operational quirks that a naive script hits hard:

• resumeKey pagination for large sites. The CDX API doesn’t hand you everything in one response. For a domain with hundreds of thousands of archived URLs, you page through using the showResumeKey / resumeKey mechanism. The extractor streams each page to the dataset in batches, so memory stays flat even on 100k+ URL domains — build it yourself and you’ll write this streaming logic or run out of memory.
• The index can be slow, and it rate-limits. The CDX index is not a fast endpoint, and large queries can time out or hit 429. The extractor retries slow responses, 5xx, and 429 errors with exponential backoff on a fresh proxy IP. Without that, big runs are flaky.
• De-duplication is on you (or the tool). Without collapse=urlkey, the CDX API returns every capture of every URL — the same page a thousand times across a thousand snapshots. The extractor collapses to one row per unique URL; a hand-rolled query has to do the same or drown in duplicates.
• No API key, but proxy strongly recommended. The CDX API needs no auth, but at volume you want a proxy to avoid rate limiting from the Archive. Proxy is on by default.
• Result caps matter for cost. Large domains can yield hundreds of thousands of URLs. maxResults per domain (or 0 for unlimited) is your lever — combine it with date ranges and status filters to keep big runs lean.

Example output

A trimmed slice showing two archived URLs — one live, one redirected:

[
  {
    "domain": "nasa.gov",
    "url": "http://www.nasa.gov/mission_pages/station/main/index.html",
    "timestamp": "20120114043915",
    "capturedAt": "2012-01-14T04:39:15.000Z",
    "statusCode": "200",
    "mimeType": "text/html",
    "digest": "AB23CD45EF67GH89IJ01KL23MN45OP67",
    "snapshotUrl": "https://web.archive.org/web/20120114043915/http://www.nasa.gov/mission_pages/station/main/index.html"
  },
  {
    "domain": "nasa.gov",
    "url": "http://www.nasa.gov/topics/earth/features/index.html",
    "timestamp": "20150803121044",
    "capturedAt": "2015-08-03T12:10:44.000Z",
    "statusCode": "301",
    "mimeType": "text/html",
    "digest": "ZZ98YY76XX54WW32VV10UU98TT76SS54",
    "snapshotUrl": "https://web.archive.org/web/20150803121044/http://www.nasa.gov/topics/earth/features/index.html"
  }
]

Use cases

• SEO migration and redirect maps — recover lost or old URLs after a site move and rebuild a complete 301 redirect map so you don’t shed link equity.
• Content recovery — find and restore blog posts, product pages, or docs that were deleted but still live in the archive.
• OSINT and research — enumerate a target domain’s historical footprint: old endpoints, removed pages, and forgotten subdomains.
• Link reclamation — surface old URLs that still attract backlinks, then redirect them to reclaim the value.
• Finding legacy endpoints — reveal admin paths, old APIs, and orphaned pages that no longer appear on the live site, for security review or cleanup.
• Web archaeology and competitive research — reconstruct how a competitor’s URL structure and content changed across years of snapshots.

Cost and effort: build vs. managed

The CDX API is free, so the cost is entirely in the query engineering:

• Building from scratch — you’d hand-craft CDX query strings for each matchType, implement resumeKey pagination with streaming so you don’t blow memory on large domains, add exponential-backoff retries for the slow, rate-limiting index, de-duplicate captures with collapse=urlkey, and construct snapshot links. It’s more code than it sounds, and the pagination/retry logic is where hand-rolled versions break.
• Using the managed actor — give it a domain and optional filters, run. Pricing is pay-per-result: you pay per archived URL returned plus the platform compute. Use maxResults, date ranges, and status filters to keep large-domain runs cheap.

The resumeKey streaming and the retry-on-slow-CDX logic are the two things that make a real difference on big domains, and both are already handled.

Common pitfalls

Specifics of the Wayback CDX API that catch people out:

• Some statusCode values are -. The Wayback index occasionally records captures without a stored status (revisit records, for instance). Those rows are still valid archived URLs — don’t discard them as broken. If you’re filtering for live pages, set filterStatus to 200 rather than assuming a missing status means dead.
• De-dup is by URL, not by content. The extractor collapses to one row per unique URL. A URL that changed content over the years still appears once — you get the URL inventory, not every content revision. If you need the version history of a single page, that’s a different query.
• matchType scope changes result size dramatically. subdomains is the broadest and can pull enormous inventories; host is exact and much smaller. Pick the narrowest scope that answers your question, or a full-site query on a big domain will run long and cost more.
• Date filters use YYYYMMDD, not ISO. fromDate/toDate are 8-digit dates (20100101), not 2010-01-01. Get the format wrong and the filter is ignored.
• The archive is incomplete by nature. The Wayback Machine captured what it crawled, not necessarily every URL a site ever had. A URL missing from the results may simply never have been archived — absence isn’t proof it never existed.
• Large maxResults: 0 runs need patience. A 100k+ URL domain is a legitimate result set, but the slow CDX index plus streaming pagination means it takes time. Cap it or window it by date if you don’t need the full history.

Wrapping up

The Wayback Machine’s CDX API is a goldmine for recovering a domain’s lost URLs, but the pagination, retries, and de-duplication are exactly the kind of plumbing that turns a quick idea into an afternoon of debugging. For a small domain, a hand-rolled CDX query works. For a full historical inventory — especially for a redirect map or OSINT sweep on a large site — a managed extractor handles the resumeKey streaming and retries so you get one clean, de-duped list with snapshot links.

▶ Open the Wayback Machine URL Extractor on Apify — one run yields a domain’s full de-duped archived-URL inventory with capture dates, HTTP status, and direct snapshot links; no API key, no login. Export to CSV or JSON, and start on Apify’s free monthly credit.