Add tools to a Speech to Speech agent

Put tools on session.update: server-side web_search, x_search, file_search, and mcp run automatically; custom function tools need your client to return function_call_output then response.create.

Configure tools on the session

import json

await ws.send(json.dumps({
    "type": "session.update",
    "session": {
        "voice": "eve",
        "instructions": "You are a helpful voice agent.",
        "turn_detection": {"type": "server_vad"},
        "tools": [
            {
                "type": "file_search",
                "vector_store_ids": ["your-collection-id"],
                "max_num_results": 10,
            },
            {
                "type": "web_search",
                "allowed_domains": ["docs.x.ai"],
                "location": {"country": "US", "city": "San Francisco"},
            },
            {
                "type": "x_search",
                "allowed_x_handles": ["xai"],
                "from_date": "2025-01-01",
                "to_date": "2025-06-01",
            },
            {
                "type": "mcp",
                "server_url": "https://mcp.example.com/mcp",
                "server_label": "my-tools",
            },
            {
                "type": "function",
                "name": "get_weather",
                "description": "Get current weather for a location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {"type": "string"},
                        "units": {"type": "string"},
                    },
                    "required": ["location"],
                },
            },
        ],
    },
}))

Server-side tools (web_search, x_search, file_search, mcp) execute on the platform — no client handler. Invalid tool config (too many domains/handles, both allowed_* and excluded_*, bad dates) returns an error event; the session stays up on the previous config.

Custom function flow

1. Server → response.function_call_arguments.done (name, call_id, arguments)
2. Run your function
3. Client → conversation.item.create with item.type: "function_call_output"
4. Wait until current turn audio has finished playing (avoids overlap)
5. Client → response.create

async def handle_function_call(ws, event, wait_for_playback):
    name = event["name"]
    call_id = event["call_id"]
    args = json.loads(event["arguments"])
    result = FUNCTION_HANDLERS[name](**args)

    await ws.send(json.dumps({
        "type": "conversation.item.create",
        "item": {
            "type": "function_call_output",
            "call_id": call_id,
            "output": json.dumps(result),
        },
    }))
    await wait_for_playback()
    await ws.send(json.dumps({"type": "response.create"}))

Parallel tool calling

When several functions are needed, the model emits multiple response.function_call_arguments.done events before any audio. Execute them (in parallel if you want), send every function_call_output, then emit a single response.create. Sending response.create early makes the model continue without the missing results.

Pitfalls

• MCP needs server_url and server_label; optional allowed_tools, authorization, and headers restrict or authenticate.
• file_search needs real vector_store_ids from the Collections API.
• Never put the long-lived API key in the browser — use ephemeral tokens for client sockets.
• Console API credits are separate from SuperGrok's weekly pool on grok.com.