How to Build a Deep-Research Retrieval Layer for AI Agents

Large language models have a knowledge cutoff and confidently hallucinate about anything recent or niche. The standard fix is a retrieval layer: search the live web, read the pages, hand the model real sources with citations. But the hosted grounding APIs — Tavily, Exa, Perplexity, Brave, SerpAPI — each want their own key, their own billing, and often bundle synthesis you did not ask for and cannot fully control. This guide covers a keyless alternative that does exactly one job: search, fetch, and read sources into clean, cited material your own model reasons over.

The important design decision up front: this actor does not synthesize. There is no LLM inside it. It returns raw, cited source material so your agent’s model does the thinking. Think of it as the retrieval-and-reading layer of a deep-research stack — it searches, fetches, dedupes, and converts to Markdown; your LLM writes the answer.

What data comes back per topic

You send one topic (or a batch), and for each you get a self-contained research bundle with three signals:

• Ranked web sources — an array of sources, each with rank (1 = top), title, url, snippet, and source (the domain).
• Full page content — with includeContent on (the default), each source carries content: the page’s main text converted to clean Markdown, capped around 6,000 characters, plus a contentWordCount. This is what makes it “deep” rather than a snippet scan.
• Recent news — an array of news items from Google News RSS (last 30 days), each with title, url, publishedAt, and source.
• Citations — a ready-to-footnote citations list, each entry { n, title, url }, so the agent can reference sources by number.
• Bundle metadata — sourceCount, newsCount, engine (which search served the results), an error field (null on success), and a generatedAt timestamp.

One row per topic. That row is everything an agent needs to reason and cite.

How the retrieval actually works

The source layer is a live web search with a resilient fallback. DuckDuckGo HTML is the primary engine; if it returns nothing, Bing takes over automatically, so you get sources instead of an empty response. The engine field on each bundle records which one served it. Recent news comes separately from Google News RSS, filtered to the last 30 days.

For each source that survives the search-and-dedupe step, the actor fetches the page and extracts the main body text, converting it to Markdown. Fetches run in parallel per topic, which is why concurrency is kept deliberately low by default — each topic already fans out into a search call plus one fetch per source plus a news call.

A batch configuration for a report-writing agent looks like this:

{
  "topics": [
    "model context protocol adoption",
    "vector database market 2026",
    "AI agent orchestration frameworks"
  ],
  "maxSources": 8,
  "includeContent": true,
  "includeNews": true,
  "maxNews": 8,
  "region": "us-en",
  "concurrency": 2
}

For a single deep dive where you want maximum reading material for a vector store, push maxSources up and drop the news:

{
  "topic": "post-quantum cryptography standards NIST",
  "maxSources": 12,
  "includeContent": true,
  "includeNews": false,
  "region": "us-en"
}

And for a fast links-and-news scan where you do not need full page bodies, turn content off and raise concurrency:

{
  "topics": ["OpenAI funding round", "Anthropic enterprise deals"],
  "maxSources": 6,
  "includeContent": false,
  "includeNews": true,
  "maxNews": 15,
  "region": "us-en",
  "concurrency": 4
}

The input fields you will actually tune: maxSources (1–20, default 6), includeContent (boolean, default true), includeNews (boolean), maxNews (0–30, set 0 to skip), region (e.g. us-en, uk-en, de-de, fr-fr, es-es), and concurrency (1–8). Requests route through Apify’s datacenter proxy by default, with a proxyGroups override to ["RESIDENTIAL"] when you need it.

▶ Open AI Deep Research on Apify — one topic in, one cited bundle out: ranked sources, full-page Markdown, and recent news. Keyless, no LLM inside, callable as an MCP tool by an agent.

The access reality

The friction that hosted grounding APIs hide behind their pricing is real, and this actor absorbs it:

• Search engines rate-limit by IP. DuckDuckGo HTML and Bing will throttle a naive scraper fast. Routing through Apify’s proxy is why useApifyProxy defaults to on — it spreads requests across IPs so you get sources, not blocks.
• Content fetching is the expensive part. Every source is a full page fetch. A topic with 12 sources means 12 fetches plus the search and news calls. That is why the default concurrency is low: it keeps runs stable and friendly to the sites you are reading. Raise it only for light topics or when includeContent is off.
• Markdown is capped for a reason. Each source’s content is truncated near 6,000 characters. That is a deliberate ceiling to keep bundles token-efficient for prompt context — you get the substance of a page, not a 40,000-word dump that blows your context window.
• Some topics yield nothing. When a topic returns no sources, the bundle still pushes a row (with error populated) but is not charged.

Example output

A trimmed bundle for one topic (arrays and content shortened with an ellipsis):

{
  "topic": "model context protocol adoption",
  "sources": [
    {
      "rank": 1,
      "title": "Model Context Protocol — An open standard for AI tools",
      "url": "https://modelcontextprotocol.io",
      "snippet": "MCP standardizes how applications provide context to LLMs...",
      "source": "modelcontextprotocol.io",
      "content": "# Model Context Protocol\n\nMCP is an open protocol that standardizes...",
      "contentWordCount": 812
    }
  ],
  "news": [
    {
      "title": "MCP gains traction across IDEs",
      "url": "https://news.example.com/mcp",
      "publishedAt": "Wed, 02 Jul 2026 09:00:00 GMT",
      "source": "Example News"
    }
  ],
  "citations": [
    { "n": 1, "title": "Model Context Protocol — An open standard", "url": "https://modelcontextprotocol.io" }
  ],
  "sourceCount": 6,
  "newsCount": 8,
  "engine": "duckduckgo",
  "generatedAt": "2026-07-04T10:00:00.000Z"
}

Use cases

• Deep-research agents — the retrieval and reading layer for an agent that writes long, cited reports. It gathers and reads; your LLM synthesizes.
• RAG & grounding — drop full-text, cited sources straight into a prompt or vector store instead of stale training data.
• Research automation — batch dozens of questions and collect ranked, read sources plus fresh news in one run.
• Market & competitive intel — one topic yields sources plus recent news, ready to summarize.
• Answer engines & chatbots — power a bot that answers with links and citations, not hallucinations.
• Fact-checking & briefings — pull current top sources and last-30-days news on any subject on demand.

Cost and effort: build vs. managed

Building this yourself is deceptively deep. You would need a search scraper that survives DuckDuckGo and Bing rate limits, a fallback chain between them, a readability extractor that turns arbitrary HTML into clean Markdown, a Google News RSS parser, dedup logic, a proxy pool, and a citation assembler — then you maintain all of it as engines change their HTML. And you would still be paying a proxy bill on top.

The managed actor is pay-per-event: one charge per topic bundle returned. Topics that yield no sources still push a row but are not charged. There is no monthly fee and no idle cost. Because pricing is per-event, an AI agent can discover and call it autonomously through the Apify MCP server — research a topic, pay for that one bundle, reason over the output — with no account setup or key handling on the agent’s side. Wire it into Claude, Cursor, or any MCP client.

Common pitfalls

• Do not expect synthesis. There is no LLM here. If your pipeline assumed the grounding API would write the answer, this one deliberately does not — that is the point, and it is what keeps it keyless and cheap.
• Concurrency is a stability lever, not a throughput dial. Each topic already fans out into many fetches. Cranking concurrency on heavy topics with includeContent on will make runs less stable, not faster.
• The Markdown cap is per source. If you need the entire text of a very long document, this is the wrong tool — it is tuned for prompt-context grounding, not archival.
• engine matters for reproducibility. The same topic can be served by DuckDuckGo one run and Bing the next depending on availability; check the engine field if results shift between runs.
• Region changes results. us-en and de-de return different source sets and news. Pin the region deliberately for consistent grounding.
• You own the content compliance. The actor retrieves publicly available pages and news; using the returned text is subject to each source’s terms, copyright, and data-protection law.

Wrapping up

If your agent hallucinates about anything recent, the fix is not a bigger model — it is fresh, cited sources in the prompt. This actor is the narrow, keyless retrieval layer for exactly that: search, read, cite, return. Keep synthesis in your own model and pay only per topic bundle.

▶ Run AI Deep Research on Apify — one run returns up to 50 cited bundles, each with ranked sources, full-page Markdown, and recent news. Keyless, MCP-callable, export to JSON, CSV, or Excel. Start on Apify’s free monthly credit.