Why We Put Page Content Inside Search

Most search APIs hand you snippets and leave the fetching to you. In Serpex, our web search API for AI agents, we put page content inside the search call itself: set include_content to true and the same request that returns ranked results also returns each page's content as markdown, with billing based on what we actually deliver, not what you asked for. This post is why we built it that way, what it costs, and when you should reach for something else instead.

What was the problem, actually?

The usual agent pipeline does the same two-step dance: search, look at the snippets, then fire off a separate fetch for whichever URLs looked worth reading. That's a second network round trip, a second client call to write and retry, and a second place to handle failures. None of it is hard. All of it is glue code that exists only because search and content-fetching happened to live in different endpoints.

Once we had both a search index and a content fetcher behind the same API, keeping them apart stopped making sense. So we merged them at the request level: one call, one price, one place to check whether it worked.

What does include_content actually do?

Add include_content: true to a /api/search request and set content_results to 5 or 10 (5 is the default). Serpex fetches the page for that many of the top results and returns it as markdown in each result's content field. A result we couldn't fetch gets content_error instead, with a reason like blocked, timeout, or robots_disallowed. A result only ever carries one of the two keys, never both, and both are absent entirely on a plain search.

import os
from serpex import SerpexClient

client = SerpexClient(os.environ["SERPEX_API_KEY"])

response = client.search(
    {
        "q": "latest AI news",
        "include_content": True,
        "content_results": 5,
    }
)

for r in response.results:
    if r.content:
        print(f"{r.title}: {len(r.content)} chars of markdown")
    elif r.content_error:
        print(f"{r.title}: content unavailable ({r.content_error})")

print(f"delivered {response.metadata.content_delivered} of {response.metadata.content_requested}")

The title, URL, and snippet still come back for every result, content or not. Content fetching is the part that's best-effort; ranking isn't.

How are we billing this?

By what actually came back, not by what you asked for. You pick content_results: 5 or 10, and you pay 3 credits when 1 to 5 results come back with content, 6 credits when 6 to 10 do. If content fetching fails for every single result in the request, you're only charged the plain-search rate of 1 credit, because you didn't get anything beyond what a plain search already gives you. A plain search, with include_content left off entirely, is always 1 credit.

Your own organization repeating the identical request within about 5 minutes costs 0 credits, same as a plain search. That includes include_content requests.

How reliable is it?

Honestly: content comes back for roughly 70% of the results you ask for. That's the number in our own docs, and we'd rather you plan around it than be surprised by it. Some pages block automated fetches outright. Some disallow it in robots.txt, and we honor that. Some just time out. When that happens you get content_error on that result and you're not charged for it, but you should write your code assuming a chunk of requested results will come back without content, not treat it as an edge case.

When should you use extract instead?

If you already know the URLs you want, don't run a search to get content for them. Call /api/crawl (the extract() method in the SDK) directly with those URLs. It's built for exactly that, handles up to 10 URLs per request, and costs 1 credit per successfully extracted URL. Reaching for search-with-content when you already have the URL just adds a ranking step you don't need. See the extract API reference for the full parameter set.

When is plain search enough?

If your agent just needs to know what's out there, snippets usually answer that. "What's the latest version of X," "who announced Y," "is Z still maintained" are all snippet questions. Turn on include_content when the task genuinely needs to read the page, not just know it exists: summarizing an article, pulling a specific fact out of body text, or feeding a retriever for RAG. Snippets are also the cheaper and faster path, so treat include_content as something you turn on for specific tasks, not a default you leave on for every query.

FAQ

Does turning on include_content slow the request down? Yes, compared to a plain search, because we're fetching pages in addition to ranking results. Budget for it in your timeouts if you're setting content_results: 10.

Am I ever charged for a page that came back empty? No. A failed content fetch on its own is never billed; you only pay the content rate for results that actually delivered content, and errors are never charged at all.

Can I request content for more than 10 results? No. content_results has to be exactly 5 or 10; anything else is rejected before the request runs.

Is content_delivered always equal to content_requested? No, and that's the whole point of billing on delivery. Check metadata.content_delivered against metadata.content_requested in every response if you need to know exactly what you got.

Try Serpex

Get an API key and you'll have 200 free credits to try include_content yourself, no card needed. The docs have the full parameter reference in the Search API page, and pricing lists the per-1,000 rates, with credits that never expire.