A Web Search Tool for CrewAI Agents

Subclass BaseTool from crewai.tools, call SerpexClient.search() inside _run, and any CrewAI agent can search the live web as part of its task. Give the tool an args_schema so CrewAI knows exactly what input it expects, hand it to an Agent, and run it inside a Crew. Below is a working researcher crew, plus a second tool for when you want full page content instead of snippets.

Crews are good at exactly this kind of work: give an agent a role, a goal, and a way to check the actual web instead of guessing. The tool is the part that has to be right, because a crew is only as current as what it can look up.

What do you need before you start?

• Python 3.10+ (CrewAI's own requirement)
• pip install serpex crewai
• A Serpex API key from app.serpex.dev. New accounts get 200 free credits, no card required.
• The key read from an environment variable, SERPEX_API_KEY.

How do you build a custom CrewAI tool with Serpex?

CrewAI's current tool API wants a Pydantic input schema and a BaseTool subclass with a _run method. That's what makes the tool's arguments explicit to the agent instead of implicit in a docstring.

import os
from typing import Type

from crewai.tools import BaseTool
from pydantic import BaseModel, Field
from serpex import SerpexClient

client = SerpexClient(os.environ["SERPEX_API_KEY"])


class WebSearchInput(BaseModel):
    query: str = Field(..., description="The search query to run.")


class WebSearchTool(BaseTool):
    name: str = "web_search"
    description: str = (
        "Search the live web for a query and return titles, URLs, and "
        "snippets for the top results."
    )
    args_schema: Type[BaseModel] = WebSearchInput

    def _run(self, query: str) -> str:
        response = client.search({"q": query})
        if not response.results:
            return "No results found."
        lines = [f"{r.title}\n{r.url}\n{r.snippet}" for r in response.results]
        return "\n\n".join(lines)

client.search() returns a SearchResponse, and each item in response.results is a SearchResult with title, url, and snippet fields. That's all a snippet-level search tool needs.

How do you wire it into a researcher crew?

A minimal crew: one agent, one task, one tool.

from crewai import Agent, Task, Crew

researcher = Agent(
    role="Web Researcher",
    goal="Find accurate, current information and cite where it came from",
    backstory=(
        "A careful researcher who checks the live web before answering "
        "instead of relying on memory."
    ),
    tools=[WebSearchTool()],
    verbose=True,
)

research_task = Task(
    description="Find the latest documentation changes for {topic} and summarize them.",
    expected_output="A short summary of what changed, with source URLs.",
    agent=researcher,
)

crew = Crew(agents=[researcher], tasks=[research_task])
result = crew.kickoff(inputs={"topic": "LangGraph create_agent"})
print(result)

The agent decides when to call web_search based on the task description and its own reasoning. You don't route that call yourself.

When should you turn on page content?

Snippets are usually enough for "what's the current version" or "who announced this." They're not enough for a task that has to summarize a whole page or pull a specific detail out of it. That's when you want include_content instead of a second extract step.

class WebSearchWithContentInput(BaseModel):
    query: str = Field(..., description="The search query to run.")


class WebSearchWithContentTool(BaseTool):
    name: str = "web_search_with_content"
    description: str = (
        "Search the live web and return each top result's page content as "
        "markdown, for tasks that need to read the full page, not a snippet."
    )
    args_schema: Type[BaseModel] = WebSearchWithContentInput

    def _run(self, query: str) -> str:
        response = client.search(
            {
                "q": query,
                "include_content": True,
                "content_results": 5,
            }
        )
        chunks = []
        for r in response.results:
            if r.content:
                chunks.append(f"# {r.title}\n{r.url}\n\n{r.content}")
            elif r.content_error:
                chunks.append(f"# {r.title}\n{r.url}\n(content unavailable: {r.content_error})")
        return "\n\n---\n\n".join(chunks)

Give this tool to an agent instead of the plain one when the task description says something like "summarize the article" or "extract the pricing details from the page," not "find out if X happened." Page-content fetching is best-effort (roughly 79% of requested results come back with content), so a summarizing agent should be able to handle a content_error on a result and move on rather than treat it as a failure.

What does this cost, and how fast can I run it?

A plain search is 1 credit. With include_content on, you're billed for pages actually delivered, not requested: 3 credits for 1 to 5 delivered pages, 6 credits for 6 to 10. If nothing comes back with content, you're only charged the 1-credit plain-search rate. Rate limits scale with your plan (5, 25, or 50 requests per second), which matters more once you have a crew of agents all calling the tool in parallel. Full details are in the Search API reference.

FAQ

Does CrewAI cache repeated searches? Not on its own, but Serpex does: your organization's identical repeat requests within about 5 minutes cost 0 credits regardless of which framework called them.

Can one crew use both tools? Yes. Give a researcher agent both WebSearchTool and WebSearchWithContentTool and let the task description or the agent's own reasoning decide which to call.

What LLM does this example assume? None specifically. The llm parameter on Agent accepts whatever model string or LLM instance your CrewAI setup is already configured with; the tool itself doesn't care.

What happens if a page is disallowed by robots.txt? You get content_error on that result (for example robots_disallowed) instead of content. It's not billed, and the rest of the results in the same call are unaffected.

Get an API key at app.serpex.dev and you'll have 200 free credits to build this yourself, no card needed. Setup steps are in the quickstart.