Gemini Live WebSocket API Documentation

1. Model Description

ability	Gemini Live Full-Duplex Real-Time Multimodal
Low-latency two-way real-time dialogue (full-duplex, with voice interrupt support at any time)	✅
Multimodal input (text/audio/video) and output (audio/text)	✅
Server-side automatic VAD (can be disabled to switch to client-side activity boundary reporting)	✅
Function Calling / Code Execution / Search as a Tool	✅
Session Memory (retains the full interaction context within a single session)	✅
Session Resumption (sessionResumption, Client-hosted Resumption)	✅
Context Window Compression (Long Sessions)	✅
Input / Output Audio Transcription	✅

2. Billing

2.1 Official Billing

• Live API bills for all context tokens in each turn, including conversation history and system instructions;

• The usage is reported via the usageMetadatafield in the server response, and the Client can monitor and verify the usage accordingly.

2.2 Billing Items and Usage of the Platform

Platform settles payments on a per-frame basis:

• Each server frame containing usageMetadatacorresponds to independent consumption for that round, with settlement conducted on a per-round basis (the sum of all rounds equals the total bill);

Platform billing items (split by modality in the usageMetadatadetails, only items with count > 0 are counted):

chargeItemCode	Corresponding to the usage dimension (usageMetadata field)	Chinese Name
in_token	promptTokensDetails[modality=TEXT]	Input Text Token
in_token_audio	promptTokensDetails[modality=AUDIO]	Input Audio Token
in_token_image	promptTokensDetails[modality=IMAGE]	Input Image Token
in_token_video	promptTokensDetails[modality=VIDEO]	Input Video Tokens
out_token	candidatesTokensDetails[modality=TEXT] + thoughtsTokenCount	Output Text Token (Reasoning Tokens are incorporated into this item)
out_token_audio	candidatesTokensDetails[modality=AUDIO]	Output Audio Token
out_token_video	candidatesTokensDetails[modality=VIDEO]	Output Video Tokens

Description of Breakdown Caliber:

• Input is split by the modality of promptTokensDetails (TEXT/ AUDIO/ IMAGE/ VIDEO), and output is split by the modality of candidatesTokensDetails;

• Thinking tokens (thoughtsTokenCount) are incorporated into out_tokenand not listed separately;

• When the output modality is IMAGE, it will be folded into out_token (there is no independent billing item for output images);

• cachedContentTokenCount (cache hit) will not be listed as a separate billing item for the time being;

• The accessing party does not need to report billing on its own, and usageMetadata (including totalTokenCount) can be used as a reference for reconciliation.

3. Details of Functional Interfaces

3.1 Full-Duplex Real-Time Multimodal WebSocket

3.1.1 Request URL

wss://{YOUR_DOMAIN}/ws/api/v1/ai/{YOUR_ENDPOINT}/gemini/live

• {YOUR_DOMAIN}: The access domain name assigned by the platform.

• {YOUR_ENDPOINT}: The access endpoint identifier (path segment) assigned by the platform.

3.1.2 Handshake Request Header Parameters

Parameter Name	Field Type	Required or not	default value	Description
Authorization	string	either one of the two	-	Bearer {{YOUR_AK}}

3.1.3 Message Frames and Common Fields

Note: For the complete field structure, please refer to the official documentation

• The frame format is a plain JSON text frame: each WebSocket Text Message carries one complete event JSON; audio data is embedded in the JSON field after Base64 encoding (not a binary frame).

• At the top level of a Client event JSON, there is exactly one protocol field (setup/ clientContent/ realtimeInput/ toolResponse), and at the top level of a server event JSON, there is exactly one protocol field (setupComplete/ serverContent/ toolCall/ toolCallCancellation/ usageMetadata/ goAway/ sessionResumptionUpdate/ inputTranscription/ outputTranscription).The communicating parties distribute and process according to this top-level field name (Note: This protocol does not use the typefield).

• Platform synthetic events use the top-level eventfield to distinguish them from the protocol events mentioned above; see §3.3.10.

• Field names use lowerCamelCase as specified in official JSON serialization (e. g., generationConfig, turnComplete, sessionResumption); some tables in the vendor's reference documentation use snake_case (e. g., generation_config, turn_complete, session_resumption), both are equivalent names for the same field, and camelCase will be used uniformly hereinafter.

3.1.4 Session Lifecycle (Recommended Sequence)

客户端                                       平台
  │── WebSocket 建连（鉴权） ───────────────▶│
  │── setup（必须是建连后首帧） ────────────▶│
  │◀── setupComplete ───────────────────────│
  │── realtimeInput（音频 / 视频流式推流） ─▶│   （循环）
  │◀── serverContent（音频 / 文本增量返回） ─│
  │◀── usageMetadata / 转写 / toolCall 等 ──│
  │◀── goAway（连接即将终止预告） ───────────│
  │◀── session_closing（平台：实例即将下线） │
  │◀── 连接关闭                              │
  │◀── connection_closed_by_model（平台） ──│
  │── 新建连 + setup(sessionResumption.handle=上次newHandle) ▶│
  │◀── setupComplete（会话上下文已恢复） ────│

Key Points:

1. setupmust be the first frame after connection establishment, and shall be sent only once; it is required to wait for setupCompletebefore sending other events.

2. After that, audio and video are continuously streamed and pushed via realtimeInput; the model incrementally returns audio and text via serverContent; at the end of a round, there will be a generationComplete/ turnCompleteflag.

3) The connection expires in approximately 10 minutes: around 30 seconds before expiration, the server sends a goAwayframe; upon receipt, new request frames shall stop being sent and reconnection shall be prepared.

4) In scenarios such as instance maintenance, the Client may first receive the session_closing (see §3.3.10.1) delivered by the platform, and then the connection will be closed. It is recommended to reconnect after receiving it to avoid server interruption.

1. When the upstream connection is closed (including normal expiration and abnormal disconnection), the platform will issue connection_closed_by_model (see §3.3.10.2).

2. Session recoverable after disconnection (Client-hosted recovery): After enabling setup sessionResumption, the server periodically sends sessionResumptionUpdate; the Client saves the newHandlein it, and when reconnecting, postback it in the new setup.sessionResumption.handleto restore the context. Recovery must be completed within 24 hours, and the recovery window is approximately 10 minutes (timeout status will be discarded).

3.1.5 Request Example (send setup after connection establishment)

{
  "setup": {
    "generationConfig": {
      "responseModalities": ["AUDIO"],
      "speechConfig": {
        "voiceConfig": {
          "prebuiltVoiceConfig": {
            "voiceName": "Charon"
          }
        }
      }
    },
    "systemInstruction": {
      "parts": [{ "text": "你是友好的语音助手，请用简体中文回答。" }]
    },
    "inputAudioTranscription": {},
    "outputAudioTranscription": {}
  }
}

3.1.6 Response Example (setupComplete)

{
  "setupComplete": {
    "sessionId": "session-xxxxxxxx"
  }
}

---

3.2 Client Events (Client → Server)

Client Event List:

Event (top-level field)	Protocol Message Name	Usage	direction
setup	BidiGenerateContentSetup	Session configuration, must be the first frame after connection establishment (sent only once)	C → S
clientContent	BidiGenerateContentClientContent	Incremental content update (text input, context injection/resumption)	C → S
realtimeInput	BidiGenerateContentRealtimeInput	Real Time Communication stream input, activity boundary signal	C → S
toolResponse	BidiGenerateContentToolResponse	Postback the execution result of Function Calling	C → S

In the following sections, "request parameters" refer to the fields of the root JSON object of a single frame.

3.2.1 setup（BidiGenerateContentSetup）

Parameter Name	Field Type	Required or not	default value	Description
setup	object	Yes	-	A session configuration object, which takes effect for the entire session and can only be sent once as the first frame
setup.generationConfig	object	No	-	Generate Parameter Configuration
setup.generationConfig.responseModalities[]	string[]	No	-	Output modality, e. g.["AUDIO"]; enable transcription to include "TEXT"
setup.generationConfig.temperature	number	No	-	Sampling Temperature
setup.generationConfig.topP	number	No	-	nucleus sampling parameters
setup.generationConfig.topK	int	No	-	top-K sampling parameter
setup.generationConfig.maxOutputTokens	int	No	-	Maximum number of output tokens
setup.generationConfig.candidateCount	int	No	-	Number of candidate responses
setup.generationConfig.presencePenalty	number	No	-	punishment imposed
setup.generationConfig.frequencyPenalty	number	No	-	Frequency Penalty
setup.generationConfig.speechConfig	object	No	-	Voice (Timbre / Language) Configuration
setup.generationConfig.speechConfig.voiceConfig.prebuiltVoiceConfig.voiceName	string	No	-	Specify the timbre name
setup.systemInstruction	object	No	-	System instructions: only text should be used in parts; this can only be set at the start of a session, and during a session, it can be appended and updated via clientContent (role= system)
setup.tools[]	object[]	No	-	Tool Definition List (function declarations use a subset of OpenAPI Schema: name / description / parameters / required)
setup.sessionResumption	object	No	-	Session resumption configuration; if included, the server will periodically deliver sessionResumptionUpdate
setup.sessionResumption.handle	string	No	-	Resume handle: leave blank (or fill in null) for a new session; pass in the value from the last sessionResumptionUpdate.newHandle when resuming a continuous transmission.
setup.sessionResumption.transparent	bool	No	-	Transparent mode; when enabled, sessionResumptionUpdatereturns lastConsumedClientMessageIndexto facilitate replaying unconsumed messages during reconnection
setup.contextWindowCompression	object	No	-	Context window compression configuration (recommended to enable for long sessions; note that early history that gets compressed will be lost)
setup.contextWindowCompression.triggerTokens	int	No	-	The cumulative token threshold for triggering compression, with a value range of 5,000 – 128,000
setup.contextWindowCompression.slidingWindow.targetTokens	int	No	-	Target window size after compression, value range: 0 – 128,000
setup.realtimeInputConfig	object	No	-	Real-time Input Processing Configuration
setup.realtimeInputConfig.automaticActivityDetection.disabled	bool	No	false	Set to trueto disable server-side automatic VAD and instead have the Client send activityStart/ activityEnd
setup.inputAudioTranscription	object	No	-	Input Audio Transcription Configuration
setup.inputAudioTranscription.languageCodes[]	string[]	No	-	BCP-47 language tags, such as "en-US", can improve the quality of multilingual recognition
setup.inputAudioTranscription.customVocabulary[]	string[]	No	-	Custom word list (up to 1,000 entries) for biased recognition of proper nouns
setup.outputAudioTranscription	object	No	-	Output the audio transcription configuration; the fields are the same as above
setup.proactivity	object	No	-	Model proactive configuration (active response, ignore irrelevant inputs)
setup.safetySettings[]	object[]	No	-	Security Settings List
setup.labels	map\	No	-	User-defined metadata tags, only for billing and reporting (key/value ≤ 63 characters)

Note: generationConfigdoes not support the following fields from the supplier, do not send them: responseLogprobs, responseMimeType, logprobs, responseSchema, stopSequences, routingConfig, audioTimestamp.

BidiGenerateContentSetup/ GenerationConfig/ Contentand other composite types are fully defined in the official documentation: BidiGenerateContentSetup, GenerationConfig, and Content.

3.2.2 clientContent（BidiGenerateContentClientContent）

Parameter Name	Field Type	Required or not	default value	Description
clientContent	object	Yes	-	Incremental content update object, the content will be unconditionally appended to the conversation history and used as a generation prompt
clientContent.turns[]	object[]	No	-	Content appended to the current conversation; a single turn refers to one individual entry, while multiple turns refer to the complete history plus the latest request
clientContent.turns[].role	string	No	-	Dialogue Roles: user/ model; system instructions can be updated mid-conversation via system
clientContent.turns[].parts[].text	string	No	-	Text content
clientContent.turnComplete	bool	No	-	trueindicates that the server should start generation based on the currently accumulated prompt;falseindicates waiting for subsequent messages

Note:

1. This event will interrupt the ongoing generation process; do not use it to postback function results (you should use toolResponse) — it is only intended for context establishment or text input.

2. Update system instructions mid-conversation: send the text content of role="system"and set turnCompleteto true, which will remain effective for the rest of the conversation after the update.

{
  "clientContent": {
    "turns": [
      {
        "role": "user",
        "parts": [{ "text": "你好，请介绍一下你自己。" }]
      }
    ],
    "turnComplete": true
  }
}

3.2.3 realtimeInput（BidiGenerateContentRealtimeInput）

Parameter Name	Field Type	Required or not	default value	Description
realtimeInput	object	Yes	-	Real-time input objects; can be sent continuously without interrupting the model generation
realtimeInput.audio	object	No	-	Real-time audio block (Blob), which constitutes a real-time audio Input Stream
realtimeInput.audio.data	string	No	-	Base64-encoded audio bytes (16kHz / 16bit PCM / little-endian)
realtimeInput.audio.mimeType	string	No	-	Fixed as "audio/pcm;rate= 16000"
realtimeInput.video	object	No	-	Real-time video frames (Blob), constituting the real-time video Input Stream
realtimeInput.video.data	string	No	-	Base64-encoded JPEG image bytes
realtimeInput.video.mimeType	string	No	-	Fixed as "image/jpeg"
realtimeInput.text	string	No	-	Real-time Text Input Stream
realtimeInput.activityStart	object	No	-	User activity start signal (can only be sent when automatic VAD is disabled, no fields included)
realtimeInput.activityEnd	object	No	-	User activity end signal (can only be sent when automatic VAD is disabled, no fields)
realtimeInput.audioStreamEnd	bool	No	-	Audio stream end flag (e. g., muting the microphone); only used when automatic VAD is enabled, stream pushing can be resumed at any time after sending
realtimeInput.mediaChunks[]	object	No	-	Inline Media Block (Blob)

Audio stream pushing example:

{
  "realtimeInput": {
    "audio": {
      "mimeType": "audio/pcm;rate=16000",
      "data": "BASE64_AUDIO_DATA"
    }
  }
}

Video Frame Stream Pushing Example:

{
  "realtimeInput": {
    "video": {
      "mimeType": "image/jpeg",
      "data": "BASE64_JPEG_DATA"
    }
  }
}

3.2.4 toolResponse（BidiGenerateContentToolResponse）

Upon receiving from the server toolCall (see §3.3.3), the Client executes the local function and posts back the result via this event, idto pair with the requesting side.

Parameter Name	Field Type	Required or not	default value	Description
toolResponse	object	Yes	-	function execution result postback object
toolResponse.functionResponses[]	object[]	Yes	-	List of function responses, paired with toolCall.functionCalls[]by id
toolResponse.functionResponses[].id	string	No	-	The unique IDcorresponding to the function call must be consistent with the one delivered by toolCall.
toolResponse.functionResponses[].name	string	No	-	function name
toolResponse.functionResponses[].response	object	No	-	Function execution result (object form)

{
  "toolResponse": {
    "functionResponses": [
      {
        "id": "function-call-001",
        "name": "get_weather",
        "response": {
          "output": { "temperature": 25, "unit": "°C", "condition": "晴" }
        }
      }
    ]
  }
}

3.3 Server Events (Server → Client)

List of server-side events:

Event (top-level field)	Protocol Message Name	Usage	direction
setupComplete	BidiGenerateContentSetupComplete	Setup completion confirmation (other events shall be sent only after this event occurs)	S → C
serverContent	BidiGenerateContentServerContent	Model-generated content (audio / text increments, interruption and turn-end flags, transcription, etc.)	S → C
toolCall	BidiGenerateContentToolCall	Request the Client to perform function calling	S → C
toolCallCancellation	BidiGenerateContentToolCallCancellation	User interruption caused the previously issued function call to be canceled	S → C
usageMetadata	UsageMetadata	Current round of token usage reporting (basis for platform settlement and reconciliation, see §2.2)	S → C
goAway	GoAway	Connection termination notice (received approximately 60 seconds before expiration, prepare to reconnect upon receipt)	S → C
sessionResumptionUpdate	SessionResumptionUpdate	Session resumption checkpoint update (requires sessionResumption to be enabled in setup)	S → C
inputTranscription	BidiGenerateContentTranscription	Input Audio Transcription Text (Incremental)	S → C
outputTranscription	BidiGenerateContentTranscription	Output audio transcription text (incremental)	S → C
event (platform synthesized)	-	Platform synthetic event: session_closing/ connection_closed_by_model, see §3.3.10	S → C

The top level of the server-side event JSON has exactly one protocol field, and distribution processing is performed according to the top-level field name (the typefield is not used).

For the official definitions of all event messages, please refer uniformly to: Multimodal Live API Reference; the composite types such as Contentand GenerationConfigmentioned below will not be elaborated further, and official jump links are provided directly.

3.3.1 setupComplete（BidiGenerateContentSetupComplete）

Parameter Name	Field Type	Mandatory return required or not	Description
setupComplete	object	Yes	setup processing completion confirmation; the Client shall wait for this event before sending other events
setupComplete.sessionId	string	Yes	the session id of this session

3.3.2 serverContent（BidiGenerateContentServerContent）

Incremental response of the model to Client messages; content is generated as soon as possible (non-real-time), and the Client can buffer it before playing at a real-time pace.

Parameter Name	Field Type	Mandatory return required or not	Description
serverContent	object	Yes	Model-generated content delta object
serverContent.modelTurn	Content	No	Content generated by the model in this round; the output audio is located at parts[]. inlineData.data (Base64,24kHz / 16-bit PCM). For the Content structure, please refer to the official documentation
serverContent.interrupted	bool	No	trueindicates that the user's voice has interrupted the current generation; the real-time playback Client shall stop immediately and clear the playback queue
serverContent.generationComplete	bool	No	trueindicates that the current generation round is completed; interrupted rounds do not have this flag (they are finalized in the order of interrupted → turnComplete)
serverContent.turnComplete	bool	No	trueindicates the end of a dialogue turn, and the server will not start the next round until it receives a subsequent Client message; this can be in the same frame as modelTurn (this content is the last segment of the current turn)
serverContent.turnCompleteReason	string	No	For the enumeration of round end reasons (safe refusal / content violation / requiring more input, etc.), see the official enumeration
serverContent.speechState	string	No	Server-side speech detection status change: SPEECH (speech detected) / NON_SPEECH (no speech)
serverContent.interactionStatus	string	No	The status of the session activity is always delivered in the same frame as turnComplete: IN_PROGRESS (still processing) / IDLE (all processing completed)
serverContent.inputTranscription	object	No	Input audio transcription (text/ finished) is independent of model rounds with no guaranteed order
serverContent.outputTranscription	object	No	Output audio transcription (text/ finished), same as above
serverContent.interimInputTranscription	object	No	Low-latency intermediate transcription during user speech
serverContent.groundingMetadata	object	No	For source metadata of Search as a Tool, see official documentation

3.3.3 toolCall（BidiGenerateContentToolCall）

Parameter Name	Field Type	Mandatory Return	Description
toolCall	object	Yes	function call request object
toolCall.functionCalls[]	object[]	Yes	List of function calls to be executed; after the Client executes them, they will be toolResponseposted back by id
toolCall.functionCalls[].id	string	No	Unique ID for function call (paired with toolResponse.functionResponses[]. id)
toolCall.functionCalls[].name	string	No	Function name (corresponding to the declaration in setup.tools[])
toolCall.functionCalls[].args	object	No	Function parameters (in object form)

For the complete definition of FunctionCall, see the official documentation. Calls canceled due to user interruption are notified via toolCallCancellation (see §3.3.4).

3.3.4 toolCallCancellation（BidiGenerateContentToolCallCancellation）

Parameter Name	Field Type	Return Required	Description
toolCallCancellation	object	Yes	Function call cancellation notification (only occurs when the Client interrupts the server-side turn)
toolCallCancellation.ids[]	string[]	Yes	List of IDs of canceled function calls; calling Clients that have already produced side effects may attempt rollback

3.3.5 usageMetadata（UsageMetadata）

Report on token usage for this round. The platform's billing standard is specified in §2.2: Each server-side frame containing usageMetadata counts as a separate consumption for the round, and the sum of all rounds equals the total bill.

Parameter Name	Field Type	Mandatory return required	Description
usageMetadata	object	Yes	Usage Object (the frame corresponding to this field serves as the basis for the current round of settlement)
usageMetadata.promptTokenCount	int	Yes	Total number of tokens in this input round
usageMetadata.candidatesTokenCount	int	No	Total number of tokens output in this round
usageMetadata.totalTokenCount	int	No	Total number of tokens in this round (for reconciliation reference)
usageMetadata.cachedContentTokenCount	int	No	Number of cache hit tokens
usageMetadata.thoughtsTokenCount	int	No	Thought token count (included in billing by the platform out_tokenbilling)
usageMetadata.promptTokensDetails[]	object[]	No	Breakdown of inputs by modality (TEXT/ AUDIO/ IMAGE/ VIDEO)(modality+ tokenCount)
usageMetadata.candidatesTokensDetails[]	object[]	No	Output breakdown by modality (modality+ tokenCount)

For the complete definition of UsageMetadata, please refer to the official documentation.

3.3.6 goAway（GoAway）

Parameter Name	Field Type	Mandatory return required or not	Description
goAway	object	Yes	Connection Termination Notice Coming Soon
goAway.timeLeft	string	No	Remaining connection duration (e. g. "60s"); the platform connection expires in approximately 10 minutes, and the notification will be sent around 60 seconds before expiration. Stop sending new request frames upon receipt, and reconnect per §3.1.4 (context can be restored with sessionResumption.handle)

For the complete definition of GoAway, please refer to the official documentation.

3.3.7 sessionResumptionUpdate（SessionResumptionUpdate）

Parameter Name	Field Type	Mandatory return required or not	Description
sessionResumptionUpdate	object	Yes	Session resumption checkpoint update (sessionResumption needs to be enabled in setup)
sessionResumptionUpdate.newHandle	string	No	New resumption handle; Client stores it and sends postback upon reconnection setup.sessionResumption.handle
sessionResumptionUpdate.resumable	bool	No	Whether the current checkpoint is recoverable;falseindicates that newHandle cannot be used for recovery
sessionResumptionUpdate.lastConsumedClientMessageIndex	string	No	Transparent mode (transparent= true) returns the consumed Client message index, which is used for reconnection and replay of unconsumed messages

For the complete definition of SessionResumptionUpdate, please refer to the official documentation.

3.3.8 inputTranscription（BidiGenerateContentTranscription）

Parameter Name	Field Type	Mandatory Return	Description
inputTranscription	object	Yes	Input Audio Transcription Event (Incremental; Aligns with Input Audio Language)
inputTranscription.text	string	No	Transcribed Text
inputTranscription.finished	bool	No	Is the transcription completed?

3.3.9 outputTranscription（BidiGenerateContentTranscription）

The field structure is consistent with §3.3.8, which is used for outputting audio transcription events (aligned with the language of the output audio). To enable transcription, responseModalitiesmust contain "TEXT".

3.3.10 Platform Synthetic Event (Top-level event Field)

Outside of protocol events, the platform uses the top-level eventfield to deliver platform-synthesized events, which are mutually exclusive with the protocol event field.

3.3.10.1 session_closing (instance is about to go offline)

Parameter Name	Field Type	Mandatory return required	Description
event	string	Yes	Fixed as "session_closing": The platform instance is about to go offline (e. g. instance maintenance, expiration of lifecycle, etc.)

Upon receipt, immediately stop sending new request frames and re-establish the connection in accordance with §3.1.4 (carrying sessionResumption.handleto resume the context) to prevent the server from directly disconnecting. Frame example:

{ "event": "session_closing" }

3.3.10.2 connection_closed_by_model (Upstream connection closed)

Parameter Name	Field Type	Mandatory return required	Description
event	string	Yes	Fixed as "connection_closed_by_model": The connection to the upstream model has been closed (this notification will be issued for both normal expiration and abnormal disconnection).
code	int	Yes	Close code: Expected closure (e. g. upon expiration) is 1000, and unexpected disconnection is 1011
reason	string	Yes	Description of Closure Reason (Transparent Transmission of Upstream Closure Description)

After receipt, reconnect and resume the session in accordance with §3.1.4. Frame example:

{ "event": "connection_closed_by_model", "code": 1000, "reason": "..." }

3.4 Audio/Video Formats and Notes on Stream Pushing

Input Audio

• Format: raw 16-bit PCM, 16000Hz, mono, little-endian; the bytes are Base64 encoded and placed in realtimeInput.audio.data;

• Sharding recommendation: 20 – 40 ms per segment; do not perform noticeable buffering before transmission (e. g., do not buffer for 1 second before sending);

• Microphone acquisition is usually at 44.1kHz / 48kHz, and before transmission it must be resampled to 16kHz;

• Mute or pause for more than 1 second (when automatic VAD is enabled): send realtimeInput.audioStreamEnd= trueto flush the audio cached on the server, after which you can resume stream pushing at any time.

Output Audio

• Format: Raw 16-bit PCM, 24000Hz, mono, little-endian, located at serverContent.modelTurn.parts[]. inlineData.data (Base64);

• The server returns data in a block-based streaming manner, and the Client is responsible for decoding, buffering and real-time playback; upon receiving interrupted= true, it shall stop immediately and clear the playback queue.

video

• Frame-by-frame JPEG, recommended 1 FPS, 768x768 resolution; the model processes at 1 FPS (not suitable for analyzing fast-changing footage);

• After Base64 encoding, place it in realtimeInput.video.data, with the mimeTypeset to "image/jpeg";

• Can be configured in the session settings mediaResolution (low/ medium/ high): Low resolution reduces token consumption and latency, while high resolution improves detail recognition.

VAD and Interruption

• By default, VAD is automatically performed on the server side; when the user interrupts with voice input, the ongoing generation is canceled and discarded, and the server sends interrupted= true (unfinished function calls are canceled via toolCallCancellation);

• Custom activity detection: In setup, set realtimeInputConfig.automaticActivityDetection.disabledto true, and the Client will send activityStart/ activityEndat the appropriate time (under this configuration, audioStreamEndwill not be sent, and stream interruption will be marked with activityEnd);

• You can enable setupin explicitVadSignalto explicitly receive server-side VAD signal events.

Session Duration and Resumption

• WebSocket connections have an upper limit of approximately 10 minutes, and a goAwaywill be delivered about 60 seconds before expiration;

• Without compression: an audio-only session lasts for approximately 15 minutes, and a video-included session lasts for around 2 minutes (subject to the token limit); when contextWindowCompressionis enabled, the session duration can be extended indefinitely (note: compression will result in the loss of early conversation history);

• Session resumption: Enable setupin sessionResumption;within 24 hours, the handle can be carried for resumption, with a resumption window of approximately 10 minutes (timeout status will be discarded); in transparent mode (transparent= true), unconsumed Client messages can be replayed according to lastConsumedClientMessageIndex;

• For scenarios requiring zero data retention, do not enable session resumption (as resumption relies on the server caching session data).

Language and Transcription

• It is recommended to explicitly specify the voice and output language, and emphasize the output language in the system prompt (e. g., RESPOND IN {OUTPUT_LANGUAGE}. YOU MUST RESPOND UNMISTAKABLY IN {OUTPUT_LANGUAGE}.);

• To enable transcription, responseModalitiesmust contain "TEXT";enabling transcription will increase token consumption;

• customVocabularyimproves the recognition accuracy of product names and proper nouns (up to 1,000 entries); listing a word list in system instructions cannot achieve the same effect.

Rate Limiting and Concurrency

• The number of concurrent sessions per project and the number of tokens per minute are subject to the vendor's quota restrictions (the standards specified in different vendors' documents vary slightly, and the requirements in the must-read for platform access shall prevail).

Platform-side disconnection

• Upon receiving session_closing (instance is about to go offline) or connection_closed_by_model (upstream connection closed), stop sending new request frames and reconnect in accordance with §3.1.4;