How to Convert Any URL to Clean Markdown for LLMs 2026

Feeding raw HTML to an LLM is a waste of tokens and a source of noise. A news article is 3% content and 97% navigation, cookie banners, ad slots, share buttons and footer links — and the model has to read all of it. The tidy fix is a “reader” that strips the chrome and returns just the article as Markdown, but the popular ones (Firecrawl, Jina Reader) want an API key and bill per page in a way that adds up fast inside an agent loop. This guide covers how to turn any URL into clean, LLM-ready Markdown plus structured metadata, keyless, over plain HTTP.

The job is single-purpose and worth doing well: point it at a page, it fetches the HTML, removes the boilerplate, and hands back the main article as tidy Markdown — headings, lists, tables, code blocks, links and images preserved — alongside metadata your pipeline can index on. It’s the “read this page for me” tool an agent or a RAG ingester actually needs. (If you need to crawl an entire site rather than convert specific URLs, that’s a different, crawler-shaped job — this is the precise, per-URL reader.)

What’s worth extracting

One structured record per URL. The markdown field is the payload; the rest is metadata for filtering, dedup and display:

The content

• markdown — the clean, LLM-ready Markdown of the page. Feed this straight to a model or a chunker.

Page metadata

• title, description — page title and meta description.
• author, publishedDate — byline and publish time, when the page exposes them.
• siteName, lang, canonicalUrl — source name, detected language, and canonical URL.

Length and structure

• wordCount, readingTimeMin — length and estimated reading time; useful for token budgeting and for skipping stubs.
• links, images — in-content links and images as structured lists (each link is a text/URL pair), so you can extract citations or crawl outward without re-parsing.

Request result

• statusCode, finalUrl — HTTP status and the final URL after redirects, so a 404 or a redirect is visible in the row rather than silently producing empty content.

How the source actually works

The design decision that keeps this fast and cheap: it reads server-rendered HTML over plain HTTP — no headless browser. There’s no Chrome to keep warm and no idle cost, and it covers the vast majority of articles, docs and blogs, which serve real content in their initial HTML. Boilerplate removal then isolates the main article: navigation, headers, footers, sidebars, ads and cookie banners are stripped so the model sees content, not chrome.

Give it at least one URL — the urls batch list (up to 500) or the single url field — and everything else is optional. A batch for RAG ingestion:

{
  "urls": [
    "https://en.wikipedia.org/wiki/Model_Context_Protocol",
    "https://en.wikipedia.org/wiki/Large_language_model"
  ],
  "onlyMainContent": true,
  "includeLinks": true,
  "includeImages": false,
  "concurrency": 5
}

A single URL, token-capped for an agent:

{
  "url": "https://news.ycombinator.com",
  "onlyMainContent": true,
  "maxCharacters": 8000
}

The knobs map directly to token cost and cleanliness:

• onlyMainContent (default true) — strip nav/header/footer/sidebars/ads/cookie banners and keep just the article.
• includeLinks (default true) — keep hyperlinks in the Markdown as [text](url).
• includeImages (default true) — keep images as ![alt](url).
• maxCharacters (default 0 = no cap) — cap Markdown length per page to control token cost.
• concurrency (default 5, 1–20) — how many URLs to fetch in parallel.
• useApifyProxy / proxyGroups — route through Apify’s datacenter proxy by default; switch to a residential group only for bot-walled sites.

▶ Run URL to Markdown on Apify — a URL in, clean article Markdown plus metadata out. No API key, no browser. Batch up to 500 pages, export to JSON, CSV or Excel, or call it live via MCP.

The access reality

The honest boundary is JavaScript rendering, and it’s worth being upfront about. This reads server-rendered HTML over plain HTTP — that’s the choice that makes it fast and inexpensive, and it handles the overwhelming majority of articles, documentation and blogs, because those ship their content in the initial HTML response. What it deliberately does not do is execute JavaScript. For a heavy client-rendered app where the content only materializes after JS runs, you want a browser-based scraper instead; this tool trades that edge case for speed and cost on the common case.

The rest of the access story is mild:

• Keyless. No Firecrawl, Jina or third-party reader key — only an Apify account. Datacenter proxy is the default and fine for public pages.
• Bot-walled pages. If a site actively blocks datacenter IPs, set proxyGroups to ["RESIDENTIAL"]. Reach for that only when you actually hit a wall, since residential adds cost.
• Visible failures. Because statusCode and finalUrl are in every record, a blocked or redirected page shows up as data — a 403 or a 301 chain — rather than a mysterious empty markdown. That’s what you want in an ingestion pipeline: failures you can filter, not silent gaps.

Example output

A trimmed record for one page:

{
  "url": "https://en.wikipedia.org/wiki/Model_Context_Protocol",
  "finalUrl": "https://en.wikipedia.org/wiki/Model_Context_Protocol",
  "statusCode": 200,
  "title": "Model Context Protocol",
  "description": "An open standard for connecting AI assistants to tools and data.",
  "author": null,
  "publishedDate": null,
  "siteName": "Wikipedia",
  "lang": "en",
  "canonicalUrl": "https://en.wikipedia.org/wiki/Model_Context_Protocol",
  "markdown": "# Model Context Protocol\n\nThe **Model Context Protocol (MCP)** is an open standard...",
  "wordCount": 1240,
  "readingTimeMin": 6,
  "links": [{ "text": "Anthropic", "url": "https://www.anthropic.com" }],
  "images": []
}

Use cases

• RAG ingestion — convert documentation, blogs and news into Markdown before chunking and embedding, so your vector store holds content, not HTML noise.
• AI agents — give an agent a “read this URL” tool it calls mid-conversation and pays for per page.
• Research and summarization — feed clean article text to an LLM for summaries or Q&A without the model wading through nav menus.
• Content monitoring — snapshot pages as Markdown on a schedule and diff them for change detection and alerts.
• Knowledge bases — bulk-convert a list of URLs into a tidy Markdown corpus for indexing.

Cost and effort math

The build-it-yourself version isn’t a weekend project done well. Boilerplate removal that reliably isolates the article across thousands of site templates is genuinely hard — it’s the part Firecrawl and Jina charge for. Add metadata extraction (author, publish date, canonical), Markdown conversion that keeps tables and code blocks intact, language detection, redirect handling, and a proxy layer, and you’ve built a maintained service that decays as site templates change.

The managed tool is pay-per-event: one charge per converted URL, no monthly fee, no idle cost — which is exactly the shape an agent’s usage takes, one page at a time. Batch up to 500 URLs per run with tunable concurrency, and use maxCharacters to cap per-page length so a single long page can’t blow your token budget. Compared to a keyed reader API, you skip the per-call subscription and pay only per page; compared to running your own extractor, you skip the extraction-quality maintenance entirely.

Common pitfalls

• No JavaScript rendering. If a page’s content is injected client-side, plain-HTTP fetch returns the shell. Check wordCount — a near-zero count on a page you know is long usually means JS-rendered content; reach for a browser scraper there.
• Cap tokens with maxCharacters. The default is uncapped. A single very long page can dominate an LLM context or a batch’s cost — set a per-page cap for agent use.
• author and publishedDate are often null. Many pages don’t publish byline or date metadata. Treat these as best-effort, not guaranteed.
• Trim links/images you don’t need. includeLinks and includeImages default on. For pure text embeddings, turn images off (and often links) to keep the Markdown lean and the token count down.
• Watch statusCode before trusting markdown. A 403 or 404 produces a row with little or no content. Filter on statusCode === 200 in your pipeline rather than assuming every row converted.
• Residential proxy only when walled. Don’t default to residential — it costs more. Use datacenter first and switch proxyGroups only for sites that actually block you.
• Batch size vs concurrency. You can queue up to 500 URLs, but concurrency (1–20) controls how many fetch at once. For a large, fragile set, a lower concurrency is gentler on the target sites and less likely to trip rate limits; for a big batch of robust sites, raise it to finish faster.

Scheduling and integration

Because it’s a standard Apify Actor, you can schedule it to refresh a Markdown corpus daily or weekly — point it at a fixed URL list and let each run re-convert the pages. Export the dataset to JSON, CSV, Excel, HTML or RSS, sync it to Google Sheets, or push the markdown field directly to your vector database, warehouse or a webhook through the Apify API. It slots into Make, n8n or Zapier for a no-code ingestion pipeline, and as an MCP tool an agent in Claude or Cursor can call it the moment it needs to read a page. A common pattern is to pair it with a sitemap expander or a URL-status checker upstream: one tool discovers the URLs, this one turns each into clean Markdown for indexing.

Wrapping up

Clean Markdown is the difference between an LLM reading an article and an LLM reading an article buried in a page’s furniture. The extraction quality is the hard part, and it’s the part you’d otherwise maintain forever. If you have one page to convert, any reader will do; if you’re ingesting URLs at scale for RAG or handing an agent a read-this-page tool, a keyless, browser-free, pay-per-page converter with visible status codes is the piece worth buying.

▶ Open URL to Markdown on Apify — any URL to clean, LLM-ready Markdown plus metadata, batchable to 500 pages. Keyless, no browser, pay per page. Start on Apify’s free monthly credit.