How to Add Live Web Search to AI Agents (No API Key)

Every LLM has a knowledge cutoff, and it will confidently hallucinate about anything past it. The fix is grounding: hand the model live search results with real URLs it can cite, instead of letting it guess. The problem is that the tidy “search API for agents” products — Tavily, Exa, Brave Search, SerpAPI — all want a key, a signup, and a per-call bill that scales badly when an agent searches on every turn. This guide covers how to get ranked, citable web results plus optional page content as clean Markdown, keyless, on a pay-per-result basis.

The core job is narrow and well-defined: send a query, get back the top-ranked results a search engine is serving right now — title, URL, snippet, source domain, rank — in a flat schema that drops straight into a prompt, a tool call, or a vector store. Optionally, attach each result page’s main text as Markdown so a RAG pipeline has something to embed without a second fetch.

What’s worth extracting

The scraper returns one row per search result, twelve fields each. Grouped by what they’re for:

Ranking and provenance

• query — the query that produced this row (essential when you batch dozens of queries into one dataset).
• rank — position in the results, 1 = top.
• engine — which engine served it, duckduckgo or bing.
• fetchedAt — ISO 8601 capture timestamp, so you know how fresh the result is.

The result itself

• title — result title.
• url — destination URL.
• displayUrl — the human-readable URL as shown in the SERP.
• source — the source domain, e.g. modelcontextprotocol.io. Handy for filtering or deduping by site.
• snippet — the summary/description text.

Grounding payload (optional)

• content — the result page’s main content as clean Markdown, populated only when you enable includeContent.
• contentWordCount — word count of that Markdown, so you can budget tokens.
• error — per-row error message if a page fetch failed (otherwise null); a failed content fetch doesn’t kill the whole run.

When includeContent is off, content and contentWordCount are null and you get pure ranked SERP rows — fast and cheap. Turn it on and each result carries the page text ready to paste into a grounding prompt, at the cost of one extra fetch per result.

How the source actually works

There are two design decisions here that matter for reliability. First, no headless browser: the scraper pulls results over direct HTTP, which is what keeps it fast and inexpensive — no Chrome to keep warm, no idle cost. Second, a dual-engine pipeline: DuckDuckGo HTML is the primary source, and if it returns nothing, the scraper automatically falls back to Bing. You get results, not empty responses on a bad query.

Input is deliberately minimal. Give it at least one query — either the queries batch list (up to 100) or the single query field — and everything else is optional. A typical agent grounding call:

{
  "queries": ["what is the model context protocol", "MCP server security best practices 2026"],
  "maxResults": 10,
  "includeContent": true,
  "region": "us-en"
}

A batch research run, ranked sources only, with concurrency for speed:

{
  "queries": [
    "best open source vector databases 2026",
    "RAG chunking strategies compared",
    "hybrid search BM25 vs embeddings"
  ],
  "maxResults": 15,
  "concurrency": 5,
  "region": "us-en"
}

The knobs, all optional:

• maxResults — ranked results per query, 1 to 30 (default 10).
• includeContent — fetch each result page and attach Markdown. Slower, one fetch per result.
• region — region/language code like us-en, uk-en, de-de, fr-fr, es-es. Search a topic the way a local would.
• safeSearch — filter explicit results (default on).
• concurrency — how many queries run in parallel, 1 to 10 (default 3).
• useApifyProxy / proxyGroups — route through Apify’s datacenter proxy (recommended, avoids per-IP rate limits); optionally override to a residential group.

▶ Run AI Web Search on Apify — one query in, ranked live results out with optional page Markdown for grounding. No Tavily/Exa/SerpAPI key. Export to JSON, CSV or Excel, or call it live via MCP.

The access reality

The honest friction with search extraction is per-IP rate limiting on the engines, not an elaborate anti-bot wall. A naive script hitting DuckDuckGo or Bing from a single IP in a tight loop gets throttled quickly. Two things address that here: routing through Apify’s datacenter proxy spreads requests across IPs, and the dual-engine fallback means a soft block on one engine doesn’t leave you empty-handed.

Because it’s direct HTTP with no browser, it covers the standard SERP job cleanly and cheaply — this is a search-and-grounding tool, not a full render-and-interact scraper. If you need JavaScript-heavy destination pages captured perfectly, that’s a job for a browser-based reader; here, includeContent extracts the main article text over HTTP, which is exactly what a RAG pipeline wants and nothing more.

Being keyless is the point: no search-engine account, no API key to provision or rotate. An AI agent can discover the tool through the Apify MCP server and call it autonomously, paying one charge per returned result — no human in the loop to set up credentials.

Localizing a query

Region control is more than cosmetic when you’re grounding an answer for a specific market. Setting region to a value like de-de or fr-fr changes which pages rank, which is exactly what you want when the user’s question is local:

{
  "query": "beste günstige elektroautos 2026",
  "maxResults": 20,
  "region": "de-de",
  "safeSearch": true
}

The supported codes pair a locale with a language — us-en, uk-en, de-de, fr-fr, es-es and more — and safeSearch (on by default) filters explicit results. If your agent serves multiple markets, pass the caller’s region through so the sources it cites match the market it’s answering for.

Example output

A single ranked result with grounding content enabled:

{
  "query": "what is the model context protocol",
  "rank": 1,
  "engine": "duckduckgo",
  "title": "Model Context Protocol — An open standard for AI tools",
  "url": "https://modelcontextprotocol.io",
  "displayUrl": "modelcontextprotocol.io",
  "source": "modelcontextprotocol.io",
  "snippet": "MCP is an open protocol that standardizes how applications provide context to LLMs...",
  "content": "# Model Context Protocol\n\nMCP is an open standard that lets AI applications connect to tools and data sources...",
  "contentWordCount": 812,
  "error": null,
  "fetchedAt": "2026-07-06T12:00:00.000Z"
}

Use cases

• RAG and grounding — hand an LLM fresh, citable sources with the page text already as Markdown, instead of stale training data.
• AI agents — a “search the web” tool an agent calls mid-conversation and pays per result, no key provisioning.
• Research automation — fire dozens of questions in one batched run and collect ranked sources across all of them.
• Answer engines and chatbots — power a bot that replies with links and citations rather than hallucinations.
• Monitoring and discovery — track what ranks for a topic over time, or surface newly indexed pages on a subject.
• Fact-checking pipelines — pull the top sources for a claim and let a downstream model reconcile them.

Cost and effort math

Rolling your own means writing SERP parsers for two engines, keeping them working as the HTML shifts, wiring a proxy pool to dodge rate limits, and building a boilerplate-stripping content extractor for the grounding step. It’s a real project, and it’s the kind that quietly rots — engines tweak their markup and your parser goes silent.

The managed version is pay-per-event: one charge per returned result, no monthly fee, no idle cost. That model fits an agent’s usage curve exactly — you pay for the results you actually get, whether that’s five rows on one call or a few thousand across a batch. There’s no proxy invoice and no parser maintenance on your side; the developer keeping the dual-engine pipeline green is carrying that. Batch up to 100 queries at 30 results each and you’re looking at hundreds to low thousands of rows per run for a few dollars.

Common pitfalls

• Snippets are not answers. The snippet is the SERP description, sometimes truncated or marketing copy. For grounding, enable includeContent and reason over content, not the snippet.
• Region changes the results. A us-en query and a de-de query for the same term return different pages. Pin region deliberately; don’t let it drift between runs if you’re comparing over time.
• Engine attribution matters. When the fallback fires, engine reads bing instead of duckduckgo. If you’re analyzing ranking consistency, group by engine — you’re not comparing like with like across engines.
• Content fetch can fail per row. A page behind a wall or a timeout sets error on that row with content null while the rest of the run succeeds. Check error before treating content as present.
• maxResults is per query, not per run. With a batch of 100 queries and maxResults: 30, you can produce thousands of rows. Size it against your token and cost budget.
• Concurrency trades speed for politeness. concurrency (1–10) sets how many queries run in parallel. Higher is faster on a big batch but leans harder on the engines; if you start seeing empty or degraded results on large runs, ease it back down toward the default of 3.
• Grounding costs an extra fetch per result. includeContent is what makes this useful for RAG, but each result now triggers a page fetch on top of the search. For a pure ranking or discovery job — tracking what shows up for a term — leave it off and keep the run fast and cheap.

Scheduling and integration

Because it’s a normal Apify Actor, you can schedule it to refresh a research corpus or watch a topic’s rankings on any cadence, export the dataset to JSON, CSV or Excel, or push results straight to a vector store, database or BI tool through the Apify API. Wire it into Make, n8n or Zapier for an automated grounding pipeline, or expose it as an MCP tool so an agent in Claude, Cursor or any MCP client calls it live mid-conversation. The pay-per-result model means a scheduled monitor that runs hourly only costs you for the rows it actually returns.

Wrapping up

Grounding is the difference between an agent that cites a real URL and one that invents a plausible-looking one. If you’re prototyping, you can stitch together an engine parser and a proxy yourself. If you’re shipping an agent that searches on every turn, a keyless, pay-per-result tool with a dual-engine fallback and optional Markdown grounding is the piece you don’t want to build and maintain.

▶ Open AI Web Search on Apify — ranked live results with optional page content as Markdown, up to 100 queries per run. Keyless, MCP-callable, pay per result. Start on Apify’s free monthly credit.