Include images in agentic tool conversations

Pass an image in the same user turn as a tool-enabled request so Grok can see the picture, then call web search / X search (or other server tools) with that visual context. Official Advanced Usage shows this with image(...) on the xAI SDK user message. Plain image understanding without tools: Analyze an image with the Grok API.

Advanced agentic patterns are not supported in the Vercel AI SDK yet; use the xAI SDK or OpenAI SDK.

xAI SDK

import os
from xai_sdk import Client
from xai_sdk.chat import image, user
from xai_sdk.tools import web_search, x_search

client = Client(api_key=os.getenv("XAI_API_KEY"))
chat = client.chat.create(
    model="grok-4.6",
    tools=[web_search(), x_search()],
    include=["verbose_streaming"],
)

chat.append(
    user(
        "Search the internet and tell me what kind of dog is in the image below.",
        "And what is the typical lifespan of this dog breed?",
        image(
            "https://pbs.twimg.com/media/G3B7SweXsAAgv5N?format=jpg&name=900x900"
        ),
    )
)

is_thinking = True
for response, chunk in chat.stream():
    for tool_call in chunk.tool_calls:
        print(
            f"\nCalling tool: {tool_call.function.name} "
            f"with arguments: {tool_call.function.arguments}"
        )
    if response.usage.reasoning_tokens and is_thinking:
        print(
            f"\rThinking... ({response.usage.reasoning_tokens} tokens)",
            end="",
            flush=True,
        )
    if chunk.content and is_thinking:
        print("\n\nFinal Response:")
        is_thinking = False
    if chunk.content and not is_thinking:
        print(chunk.content, end="", flush=True)

print("\n\nCitations:", response.citations)
print("Usage:", response.usage)
print("Tool usage:", response.server_side_tool_usage)

include=["verbose_streaming"] surfaces tool calls while they run. Swap the sample URL for your own HTTPS image, or use a Files API file_id path documented for chat attachments when the asset is private.

What you get back

Field	Use
chunk.tool_calls	Live server-side tool invocations during the stream
response.citations	URLs gathered by search tools
response.server_side_tool_usage	Counts of server tool calls for this turn
response.tool_calls	Full server-side tool call list on the finished response

Pitfalls

• Image-only chat without tools will describe the picture but will not open web/X search for breed facts or news.
• Broken or auth-walled image URLs fail before tools run — confirm the URL loads anonymously, or upload via Files API first.
• Vercel AI SDK skips these advanced agentic patterns; stay on xAI SDK / OpenAI SDK for this flow.