How to Audit HTTP Security Headers in Bulk (CSP, HSTS)

Checking one site’s security headers is easy — paste a URL into any online checker and read the grade. Doing it for a portfolio of 500 domains, on a schedule, and getting the results as structured rows you can diff over time is where the free web tools fall over. They’re one-URL-at-a-time, they rate-limit you, and they give you a screenshot instead of a dataset. This guide covers how to audit HTTP security headers at scale: which headers actually matter, how the scoring works, and how to turn a list of URLs into a clean per-URL security report you can pipe into a spreadsheet, a SIEM, or a CI gate.

What’s worth extracting

For each URL, the scanner returns a full security-header report — 30+ fields. The ones that carry the analysis:

• Score & grade — securityScore (0–100) and grade (A+, A, B, C, D, F). These are the headline numbers for ranking and tracking.
• Content-Security-Policy — cspHeader (raw value or not set), cspPass (present and well-formed), and cspReportOnly if a report-only policy is in place.
• HSTS, deep-parsed — hstsHeader, hstsPass, plus the directives broken out: hstsMaxAge, hstsIncludeSubdomains and hstsPreload.
• Framing & sniffing — xFrameOptions / xFrameOptionsPass (pass on DENY or SAMEORIGIN) and xContentTypeOptions / xContentTypeOptionsPass (pass on nosniff).
• Privacy & permissions — referrerPolicy / referrerPolicyPass and permissionsPolicy / permissionsPolicyPass.
• Cross-origin isolation — crossOriginOpenerPolicy (COOP), crossOriginResourcePolicy (CORP) and crossOriginEmbedderPolicy (COEP), each with a pass flag where applicable.
• Information-leak checks — server and xPoweredBy. These are flagged when present, because exposing your stack version is a mild information leak.
• Report summary — missingHeaders (a comma-separated list of exactly which recommended headers are absent), passCount, totalChecks (11), and cacheControl.
• Request metadata — url, finalUrl (after redirects), statusCode, latencyMs, and checkedAt (ISO 8601).

The missingHeaders field is the operationally useful one: it tells you exactly what to fix per URL, so you can generate remediation tickets straight from the dataset.

How the source actually works

This isn’t a scraper of any third-party site — it inspects the target URLs’ own HTTP responses. For each URL, the actor sends an HTTP HEAD request (not GET), which returns the response headers without downloading the page body. All security-relevant information lives in those headers, so HEAD is both faster and lighter. Redirects are followed automatically (up to 5), and the actor reports the landing finalUrl, so an http:// to https:// upgrade is handled transparently.

Input is just a list of URLs — the scheme is added automatically if you omit it:

{
  "urls": ["https://github.com", "https://stackoverflow.com", "https://example.com"],
  "maxConcurrency": 20,
  "proxyConfiguration": { "useApifyProxy": true }
}

For a large list, raise maxConcurrency (default 20, max 100) so more URLs are checked in parallel:

{
  "urls": ["https://site1.com", "https://site2.com", "https://site3.com", "https://site4.com"],
  "maxConcurrency": 50,
  "proxyConfiguration": { "useApifyProxy": true }
}

A natural pattern is to feed the output of a sitemap expander or crawler into urls, so you audit an entire domain’s pages in one run rather than hand-listing them.

How the score is computed

Each of the 11 checks carries a weight reflecting its security impact, and the score is the weighted percentage of checks that pass, normalized to 0–100:

• CSP — present and well-formed. Weight 20.
• HSTS — max-age of at least 31,536,000 (one year). Weight 15.
• X-Frame-Options — DENY or SAMEORIGIN. Weight 12.
• X-Content-Type-Options — equals nosniff. Weight 10.
• Referrer-Policy — restricts referrer data. Weight 10.
• Permissions-Policy — present. Weight 8.
• COOP / CORP / COEP — cross-origin isolation. Weight 5 each.
• Server / X-Powered-By — information leak; points are awarded when these are not present. Weight 5 each.

Grades map from the score: A+ at 90 and above, A at 80+, B at 70+, C at 50+, D at 30+, and F below 30.

▶ Run the Bulk HTTP Security Headers Analyzer on Apify — one run scores thousands of URLs across 11 header checks and returns a 0-100 score plus an A+ to F grade per URL. HEAD-only, no API key, no login. Export to JSON, CSV, Excel or XML.

The access reality

Reading a site’s HTTP response headers is a standard, lightweight request that downloads no page content, so this is about as low-impact as web requests get. Still, a few practical notes:

• Use a proxy for large lists. Hitting many hosts quickly can trigger rate limiting or WAF friction. Apify Proxy (datacenter is the default) is recommended to spread requests. Keep it enabled for anything past a handful of URLs.
• HTTPS URLs give meaningful results. Plain http:// sites will never return HSTS — HSTS is an HTTPS-only header — so feed HTTPS URLs when you want the full picture.
• Some hosts reject HEAD. A minority of servers don’t handle HEAD cleanly and behave differently than on GET. Where that happens you’ll see it in statusCode; treat those rows as needing a manual re-check rather than a real failing grade.
• Concurrency is a throughput lever, not a magic one. Raising maxConcurrency toward 100 speeds up big lists, but very aggressive parallelism against a single origin can look hostile. Spread across many distinct hosts and it’s a non-issue.

The naive approach — a shell loop of curl -I piped through grep — technically works for ten URLs, but it gives you raw header text, no scoring, no pass/fail normalization, and no dataset. The value here is turning headers into a weighted, comparable score across thousands of URLs in one structured output.

Example output

One representative record (trimmed):

{
  "url": "https://github.com",
  "finalUrl": "https://github.com/",
  "statusCode": "200",
  "securityScore": "78",
  "grade": "B",
  "cspHeader": "default-src 'none'; ...",
  "cspPass": "true",
  "hstsHeader": "max-age=31536000; includeSubdomains; preload",
  "hstsPass": "true",
  "hstsMaxAge": "31536000",
  "hstsIncludeSubdomains": "true",
  "hstsPreload": "true",
  "xFrameOptions": "deny",
  "xFrameOptionsPass": "true",
  "xContentTypeOptions": "nosniff",
  "xContentTypeOptionsPass": "true",
  "permissionsPolicy": "not set",
  "permissionsPolicyPass": "false",
  "server": "GitHub.com",
  "missingHeaders": "Permissions-Policy, Cross-Origin-Opener-Policy",
  "passCount": "8",
  "totalChecks": "11",
  "checkedAt": "2026-07-06T12:00:00.000Z"
}

Use cases

Who runs bulk header audits and why:

• Security posture audits — scan an entire web portfolio and get a numeric score per URL, then track improvement over time by diffing scheduled runs.
• Compliance scanning — SOC 2, ISO 27001 and PCI DSS controls reference headers like HSTS and CSP; a per-URL pass/fail report is direct evidence of coverage (or gaps).
• Vendor / third-party risk — score the SaaS tools and partners you depend on for basic HTTP security hygiene on the same 0–100 scale.
• Bug-bounty recon — quickly surface sites with weak or missing security headers that broaden the attack surface (clickjacking on a missing X-Frame-Options, for example).
• DevSecOps CI gate — run as a post-deploy check and fail the build if a security header regresses; parse missingHeaders to know exactly what broke.
• Competitor benchmarking — compare your header posture against competitors on the same scale for a leaderboard.

Build-it-yourself vs. managed actor

You can write a HEAD-request loop with a header parser in an afternoon. What you then own:

• The scoring rubric. Deciding weights, defining what “well-formed CSP” means, parsing HSTS directives, and keeping the grade bands sensible — that’s the actual product, and it’s easy to get subtly wrong.
• Concurrency and redirect handling. Parallelizing safely, following redirect chains, and recording the final URL without losing the original are all small tasks that add up.
• Proxy and rate-limit resilience. For big lists you’ll want proxy rotation so you aren’t blocked mid-run.
• Output normalization. Turning inconsistent header casing and formats into clean, one-row-per-URL columns that drop straight into a spreadsheet.

The managed actor is pay-per-result: each URL is one row at a fraction of a cent, so auditing a few thousand URLs weekly is trivial in cost. If you only ever check one site once, an online checker is fine. If you need a repeatable, dataset-shaped audit across a portfolio, the managed path skips building and maintaining the scoring engine.

Common pitfalls

Specific gotchas when auditing headers:

• HSTS needs a real max-age. A present Strict-Transport-Security header with a short or zero max-age is effectively not protection. The check only passes at max-age of a year or more — read hstsMaxAge, don’t just check for the header’s presence.
• CSP presence isn’t CSP quality. cspPass confirms a policy exists and is well-formed, not that it’s tight. A permissive default-src * will still pass the presence check, so review the raw cspHeader for actual strength.
• Report-Only doesn’t enforce. A Content-Security-Policy-Report-Only policy (in cspReportOnly) monitors violations but blocks nothing. Don’t count it as an enforced CSP.
• Redirects change the graded host. The score reflects finalUrl, not the URL you submitted. If http://example.com redirects to a CDN, you’re grading the CDN’s headers — check finalUrl before drawing conclusions about the origin.
• Information-leak checks invert. For server and xPoweredBy, present is bad — points are awarded for absence. Don’t misread a populated server field as a “pass.”
• Plain HTTP skews low. An http://-only URL can’t return HSTS and will score lower by design. Compare like with like; audit the HTTPS endpoint.

Wrapping up

Auditing HTTP security headers is a lightweight, non-invasive request — the challenge is doing it at portfolio scale, consistently, with a comparable score you can track. If that’s a one-time curiosity, an online checker does the job. If it’s a recurring audit feeding compliance evidence, remediation tickets or a CI gate, a managed actor that already encodes the weighted scoring and outputs clean per-URL rows is the faster route.

▶ Open the Bulk HTTP Security Headers Analyzer on Apify — sends HEAD requests to every URL, checks 11 security headers, and returns a 0-100 score, an A+ to F grade, per-header pass/fail and a missingHeaders list per URL. No API key, no login. Export to JSON, CSV, Excel or XML. Start on Apify’s free monthly credit.