Stream binary audio in Speech to Speech

Set audio.input.transport / audio.output.transport to "binary" on session.update, then send and receive raw PCM or Opus as WebSocket binary frames while lifecycle events stay JSON.

Codec vs transport

	Input	Output
json (default)	Base64 in input_audio_buffer.append	Base64 in response.output_audio.delta / response.audio.delta
binary	Raw codec bytes as binary frames (no protocol header)	Same binary frames; response.created, response.done, transcripts stay JSON

format picks the codec (audio/pcm, audio/pcmu, audio/pcma, audio/opus). transport picks the wire path. They are independent.

Enable binary both ways

import json

await ws.send(json.dumps({
    "type": "session.update",
    "session": {
        "audio": {
            "input": {
                "format": {"type": "audio/pcm", "rate": 24000},
                "transport": "binary",
            },
            "output": {
                "format": {"type": "audio/pcm", "rate": 24000},
                "transport": "binary",
            },
        },
    },
}))

# Mic: raw PCM16 little-endian (not base64 JSON)
await ws.send(pcm16_bytes)

async for message in ws:
    if isinstance(message, bytes):
        play(message)  # raw PCM16, or one Opus packet if format is audio/opus
    else:
        event = json.loads(message)
        # response.done, transcripts, tool events, …

Opus: each binary frame (or JSON delta/audio field) is one raw Opus packet at 24 kHz mono — no extra framing header.

Dual-accept input, strict output

• Input: with a configured input format, the server accepts both JSON append and binary frames for that codec. Prefer the transport you set; you do not need to drain one channel before the other mid-session.
• Output: assistant audio leaves only on output.transport. Changing output.transport mid-session applies at the next response boundary so one utterance never mixes JSON deltas and binary frames.

Pitfalls

• Omit transport (or set "json") to keep older clients on base64 deltas.
• Match playback sample rate to audio.output.format.rate (PCM).
• Treat text frames as events and binary frames as audio — do not json.loads binary payloads.
• Console API credits are separate from SuperGrok's weekly pool on grok.com.