Tool Selection

How LLMs select tools

When you pass tools to an LLM, it selects which to call (and with what arguments) based entirely on the tool descriptions and the current conversation. There's no explicit "tool selection algorithm" in the model — it's all inference from text.

This means tool selection quality is a prompt engineering problem: write clearer, more specific descriptions and the model selects tools more reliably. The model asks itself (implicitly): "Given the user's request and these tool descriptions, which tool, if any, should I call right now?"

For small sets of tools (under ~20), this works excellently. The challenge comes with scale: a production agent system might have hundreds of tools, and including all of them in every prompt is expensive and confusing.

Writing tool descriptions that work

Tool descriptions are the primary lever for improving selection accuracy. Effective descriptions answer four questions:

What does this tool do? ("Searches the product catalogue for items matching a query")

When should it be used? ("Use when the user asks about product availability, pricing, or specifications")

When should it NOT be used? ("Do not use for order status — use get_order_status instead")

What does it return? ("Returns a list of up to 10 matching products with name, price, and stock level")

# Bad description — too vague
{
    "name": "search",
    "description": "Search for information.",
    ...
}

# Good description — specific, actionable, with negative examples
{
    "name": "search_products",
    "description": (
        "Search the product catalogue by keyword. "
        "Use when the user asks about product availability, pricing, features, or specifications. "
        "Returns up to 10 matching products. "
        "Do NOT use for order status, returns, or account issues — use the appropriate tools for those."
    ),
    ...
}

Scaling to many tools with RAG

For large tool catalogues (50+ tools), include all descriptions in every prompt is wasteful and degrades selection quality. Instead, treat tool selection as a retrieval problem: embed all tool descriptions, then retrieve only the most relevant ones for each user query.

import numpy as np
from sentence_transformers import SentenceTransformer
import anthropic

# Tool catalogue
ALL_TOOLS = [
    {"name": "search_products", "description": "Search product catalogue...", "schema": {...}},
    {"name": "get_order_status", "description": "Check order delivery status...", "schema": {...}},
    {"name": "process_return", "description": "Initiate a product return...", "schema": {...}},
    # ... 100+ more tools
]

# Embed all tool descriptions at startup
embedder = SentenceTransformer("all-MiniLM-L6-v2")
tool_descriptions = [t["description"] for t in ALL_TOOLS]
tool_embeddings = embedder.encode(tool_descriptions)  # shape: (N, 384)

def retrieve_relevant_tools(query: str, k: int = 8) -> list[dict]:
    '''Retrieve top-K most relevant tools for the user query.'''
    query_embedding = embedder.encode([query])
    # Cosine similarity
    norms = np.linalg.norm(tool_embeddings, axis=1) * np.linalg.norm(query_embedding)
    similarities = (tool_embeddings @ query_embedding.T).flatten() / (norms + 1e-8)
    top_k_indices = np.argsort(similarities)[-k:][::-1]
    return [ALL_TOOLS[i] for i in top_k_indices]

client = anthropic.Anthropic()

def agent_with_tool_retrieval(user_query: str) -> str:
    # Retrieve only relevant tools
    relevant_tools = retrieve_relevant_tools(user_query, k=8)
    api_tools = [{"name": t["name"], "description": t["description"],
                  "input_schema": t["schema"]} for t in relevant_tools]

    response = client.messages.create(
        model="claude-sonnet-4-5", max_tokens=1024,
        tools=api_tools,
        messages=[{"role": "user", "content": user_query}]
    )
    return response.content[0].text

Hierarchical tool routing

For very large catalogues (hundreds to thousands of tools), even RAG retrieval can be slow. Hierarchical routing adds a fast first-stage classifier that narrows down the category before retrieving specific tools:

TOOL_CATEGORIES = {
    "orders": ["get_order_status", "cancel_order", "update_shipping"],
    "products": ["search_products", "get_product_details", "check_stock"],
    "returns": ["process_return", "get_return_status", "refund_timeline"],
    "account": ["update_account", "reset_password", "billing_history"],
}

def classify_query_category(query: str) -> str:
    categories = list(TOOL_CATEGORIES.keys())
    response = client.messages.create(
        model="claude-haiku-4-5-20251001", max_tokens=32,
        messages=[{"role": "user", "content": f"Classify into one category: {categories}
Query: {query}
Category:"}]
    )
    return response.content[0].text.strip().lower()

def route_to_tools(query: str) -> list[str]:
    category = classify_query_category(query)
    return TOOL_CATEGORIES.get(category, TOOL_CATEGORIES["products"])

# Now load only the 3-5 tools in the matched category
tool_names = route_to_tools("Where is my order #12345?")
# → ["get_order_status", "cancel_order", "update_shipping"]

The classification call uses a cheap, fast model (Haiku). The second call uses the right tool subset with a more capable model. Total latency is lower than loading all tools, and accuracy improves because the model has fewer tools to confuse.

Tool selection metrics

Measure tool selection quality before optimising it. Key metrics:

Selection accuracy: what % of time does the model call the correct tool for a given query? Build a test set of (query, expected_tool) pairs and measure.

False positive rate: how often does the model call a tool when no tool is needed (the query could be answered from knowledge)?

Wrong-tool rate: how often does the model call a plausible-but-wrong tool (e.g., calling search_products instead of get_order_status for an order query)?

test_cases = [
    {"query": "Where is my order 123?", "expected_tool": "get_order_status"},
    {"query": "Do you have red sneakers in size 10?", "expected_tool": "search_products"},
    {"query": "What is your return policy?", "expected_tool": None},  # No tool needed
]

correct = sum(1 for case in test_cases
              if get_selected_tool(case["query"]) == case["expected_tool"])
accuracy = correct / len(test_cases)
print(f"Tool selection accuracy: {accuracy:.1%}")

Dynamic tool loading

Instead of defining tools statically, load them from a registry at runtime. This enables tool versioning, A/B testing tool descriptions, and feature flags for tool availability:

import json

class ToolRegistry:
    def __init__(self, registry_path: str):
        with open(registry_path) as f:
            self._tools = json.load(f)  # loaded from DB or config file

    def get_tools_for_user(self, user_id: str, tier: str) -> list[dict]:
        '''Return tools available for this user tier.'''
        return [t for t in self._tools
                if tier in t.get("available_tiers", ["free", "pro", "enterprise"])]

    def get_tools_for_context(self, context: str) -> list[dict]:
        '''Return tools tagged for a specific context.'''
        return [t for t in self._tools if context in t.get("contexts", [])]

registry = ToolRegistry("tools_registry.json")
tools = registry.get_tools_for_user(user_id="user_123", tier="pro")

Store tool descriptions in a database with versioning. When descriptions are improved, update the registry and all agents automatically benefit — no code deploy required.

Gotchas

Similar tools confuse the model. If you have search_products and search_catalogue that do the same thing, the model will pick randomly between them. Audit your tool set for redundancy — merge or rename overlapping tools.

Tool count degrades selection. Empirically, model tool selection accuracy drops meaningfully above ~20 tools in a single prompt. If you have more tools, use RAG retrieval or hierarchical routing — don't just stuff all tools into the prompt and hope the model handles it.

The model can call no tool when it should. If the model can't find a good tool match, it will sometimes answer from its parametric knowledge instead of calling a tool. Add explicit instructions: "Always use the available tools to answer user questions. Do not use general knowledge for questions about orders, products, or account information."