How to Monitor Certificate Transparency Logs for Recon

Every time an organization issues a TLS certificate, the hostname is published to public Certificate Transparency logs within minutes. That makes CT logs the single richest passive source of infrastructure intelligence on the internet: new services, staging environments, shadow IT, and lookalike phishing domains all show up the moment a cert is minted, often before DNS or a port scan would catch them. The problem is access. crt.sh aggregates all the major CT logs, but it has no clean, structured, automation-friendly API — you get an overloaded web endpoint that times out on large targets and returns duplicate precertificate rows you have to dedup yourself. This guide covers how to pull CT-log data reliably and what to do with it.

What data is worth extracting

Each certificate becomes one structured row with 13 fields. Grouped by what they tell you:

• Identity — certId (the crt.sh certificate ID), commonName (the CN, e.g. *.stripe.com), and serialNumber.
• The hostname goldmine — nameValues: the Subject Alternative Names, an array of every hostname the certificate is valid for. This is the richest source for subdomain discovery.
• Issuer — issuerName, the CA that signed it (Let’s Encrypt, DigiCert, Google, Cloudflare, and so on).
• Validity window — notBefore and notAfter (ISO 8601), plus isExpired, a boolean shortcut for a current attack-surface view.
• Timing — entryTimestamp (when the cert was logged to the CT log) and loggedDate (the YYYY-MM-DD part), which is what “new cert today” is keyed on.
• Context — query (the exact crt.sh query used), domain (the target), and scrapedAt (capture time).

How the source actually works

The actor queries crt.sh’s JSON search endpoint directly over HTTP — no browser, no key. The query shape is the thing to understand:

https://crt.sh/?q=<query>&output=json

For a full history you query the bare domain; for subdomain discovery you query a wildcard %.domain, which surfaces every hostname ever certified under that domain. crt.sh already aggregates and deduplicates across all the major CT logs (Google Argon, Cloudflare Nimbus, Let’s Encrypt Oak, DigiCert), so you avoid the alternative — parsing Merkle trees and merge-deduplicating individual logs yourself.

Four operational modes cover the recon workflows:

Mode	What it does
watch	Keeps only certs logged in the last N days (daysBack). 1 = today’s new certs.
subdomains	Wildcard %.domain query, extracts SAN hostnames — passive subdomain enumeration.
domain	Full certificate history for a target (or a small list).
bulk	Sweeps a whole watchlist of domains.

A daily monitoring run you would schedule:

{
  "mode": "watch",
  "domain": "google.com",
  "daysBack": 1,
  "maxPerDomain": 500
}

A wildcard subdomain sweep, dropping expired certs for a current-surface view:

{
  "mode": "subdomains",
  "domain": "stripe.com",
  "maxPerDomain": 2000,
  "excludeExpired": true
}

And a batch history pull across several targets:

{
  "mode": "domain",
  "domains": ["stripe.com", "linear.app", "vercel.com"],
  "maxPerDomain": 1000
}

Two filters shape the output. excludeExpired drops certs past their notAfter so you only see live surface; dedupBySerial (on by default) collapses the precertificate and final-certificate pair that CT logs record for every issuance into a single row.

▶ Open the Certificate Transparency Monitor on Apify — point it at a domain in watch or subdomains mode and get one structured row per certificate with issuer, SAN hostnames, validity, and serial. No key, no browser.

The access reality

CT-log data is public by design — the CA/Browser Forum Baseline Requirements mandate logging, and certificate holders consent to it at issuance — so there is no auth wall. The friction is entirely operational, and it comes from crt.sh:

• crt.sh gets slow under load. For popular domains (google, microsoft) with tens of thousands of certs, a single query can take 10–60 seconds. The actor uses a 60-second timeout plus exponential-backoff retry (up to 5 attempts) on 5xx errors and timeouts, so a slow response does not kill the run.
• It runs behind Apify’s datacenter proxy by default, because crt.sh can be overloaded and a naive single-IP scraper hits its limits quickly.
• Volume can be enormous. maxPerDomain goes up to 20,000 certs per domain, and popular domains genuinely have that many. A wildcard subdomains query on a large org routinely yields hundreds to thousands of hostnames per run.
• Duplicate rows before dedup. Without dedupBySerial, you will see each certificate twice — the precert and the final cert share a serial. Leave dedup on unless you specifically want both.

The data is near-real-time: a certificate logged minutes ago typically appears in crt.sh within minutes to a few hours, which is what makes watch with daysBack: 1 a meaningful daily monitor.

Example output

One certificate row, trimmed:

{
  "query": "stripe.com",
  "domain": "stripe.com",
  "certId": 12345678901,
  "commonName": "*.stripe.com",
  "nameValues": ["stripe.com", "www.stripe.com", "api.stripe.com", "dashboard.stripe.com"],
  "issuerName": "DigiCert TLS RSA SHA256 2020 CA1",
  "serialNumber": "0abc123def456...",
  "notBefore": "2026-06-28T00:00:00.000Z",
  "notAfter": "2027-07-03T23:59:59.000Z",
  "entryTimestamp": "2026-06-28T12:34:56.789Z",
  "loggedDate": "2026-06-28",
  "isExpired": false,
  "scrapedAt": "2026-07-06T12:00:00.000Z"
}

The Overview dataset view scans all certs with issuer and validity; the Subdomains (SAN) view reads the discovered hostnames per cert.

Use cases

• New-infrastructure detection — run watch daily and alert when a cert appears for a hostname you have not seen before, catching new services, staging environments, and shadow IT.
• Subdomain enumeration — subdomains mode builds a hostname inventory from SANs, passively collected and often broader than DNS brute-forcing.
• Brand-protection / lookalike detection — query a brand string to flag impersonation domains (stripe-login.com, secure-stripe.com) the moment they are certified.
• Issuer / CA tracking — monitor which CAs a target uses and alert on anomalies, such as a sudden shift to a cheap CA often used for phishing.
• Attack-surface diffing — schedule weekly domain runs, diff the cert sets, and report added or removed hostnames.
• RAG over infrastructure — embed the cert feed into a vector store so an agent can answer “when did X hostname first appear?” with a citation to the exact certificate.
• Threat-intel pipeline — feed new certs into a SIEM and correlate issuer/hostname patterns with known attacker behavior.

This is built for the wave of AI agents that reason about external attack surface — recon agents watching a target’s new certificates, brand-protection agents detecting lookalikes, asset-discovery agents enumerating subdomains — plus RAG pipelines grounding on live infrastructure rather than stale scans. Expose it through an MCP server and an agent passes a domain plus mode and receives structured certs back, with no crt.sh scraping or SAN parsing on the agent side.

Cost and effort: build vs. managed

Rolling your own means wrapping crt.sh’s flaky endpoint in retry logic, handling the 60-second query times on big domains, splitting SANs out of the name_value blob, deduping precert/final pairs by serial, and normalizing dates to ISO — then keeping it alive as crt.sh’s load and response format shift. It is not hard, but it is fiddly, and the failure mode (silent timeouts on your most interesting targets) is exactly where you cannot afford gaps.

The actor is pay-per-result: one event per saved certificate, and runs that yield zero certs are free. That economics is what makes scheduled watch monitoring cheap and predictable — a daily watch on a normal domain that logs a handful of new certs costs a fraction of a cent, and a quiet day costs nothing.

Common pitfalls

• CT logs are historical, not live-state. A cert in the logs may have been issued for a host that no longer resolves. Use excludeExpired for a current-surface view, and pair CT enumeration with an active resolver if you need liveness.
• SANs contain the whole hostname set, including the CN. For a wildcard cert, nameValues holds *.stripe.com plus every explicit name. Parse the array, do not assume one hostname per cert.
• daysBack keys on entryTimestamp, not notBefore. “New today” means logged today, which is what you want for monitoring — but a backdated cert reissue can log later than its validity start.
• Private/internal CAs never appear. CT only covers publicly-trusted certificates. Hosts using an internal CA are invisible here — true of any CT-based tool.
• Big-org watches can flood. Large organizations log dozens of certs a day; set maxPerDomain and daysBack deliberately so a watch run stays focused.
• Dedup is on by default for a reason. Turn it off only if you explicitly want to see precert and final-cert rows separately.

Wrapping up

CT logs are the closest thing to a free, real-time feed of an organization’s new infrastructure, and crt.sh is the aggregator — but its unstructured, timeout-prone endpoint is the reason people give up on automating it. A retry-hardened, deduping layer turns that into a reliable per-certificate feed you can schedule, diff, or hand to an agent.

▶ Run the Certificate Transparency Monitor on Apify — one run yields hundreds to thousands of certificates with SAN hostnames, issuer, and validity; empty runs are free. No key, no browser, MCP-callable, export to JSON, CSV, or Excel. Start on Apify’s free monthly credit.