Gemini Live API reference

The Gemini Live API enables low-latency bidirectional voice and video interactions with Gemini. Using the Gemini Live API, you can provide end users with the experience of natural, human-like voice conversations, and with the ability to interrupt the model's responses using voice commands. The Gemini Live API can process text, audio, and video input, and it can provide text and audio output.

For more information about the Gemini Live API, see Gemini Live API .

Capabilities

Gemini Live API includes the following key capabilities:

• Multimodality: The model can see, hear, and speak.
• Low-latency realtime interaction: The model can provide fast responses.
• Session memory: The model retains memory of all interactions within a single session, recalling previously heard or seen information.
• Support for function calling, code execution, and Search as a Tool: You can integrate the model with external services and data sources.

Gemini Live API is designed for server-to-server communication.

For web and mobile apps, we recommend using the integration from our partners at Daily .

Supported models

Get started

To try the Gemini Live API, go to the Vertex AI Studio , and then click Start Session.

Gemini Live API is a stateful API that uses WebSockets .

This section shows an example of how to use Gemini Live API for text-to-text generation, using Python 3.9+.

pip install --upgrade google-genai

 # Replace the `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION` values 
 # with appropriate values for your project. 
 export 
  
 GOOGLE_CLOUD_PROJECT 
 = 
 GOOGLE_CLOUD_PROJECT 
 export 
  
 GOOGLE_CLOUD_LOCATION 
 = 
 global 
 export 
  
 GOOGLE_GENAI_USE_VERTEXAI 
 = 
True

  from 
  
 google 
  
 import 
 genai 
 from 
  
 google.genai.types 
  
 import 
 ( 
 Content 
 , 
 HttpOptions 
 , 
 LiveConnectConfig 
 , 
 Modality 
 , 
 Part 
 ) 
 client 
 = 
 genai 
 . 
 Client 
 ( 
 http_options 
 = 
 HttpOptions 
 ( 
 api_version 
 = 
 "v1beta1" 
 )) 
 model_id 
 = 
 "gemini-2.0-flash-live-preview-04-09" 
 async 
 with 
 client 
 . 
 aio 
 . 
 live 
 . 
 connect 
 ( 
 model 
 = 
 model_id 
 , 
 config 
 = 
 LiveConnectConfig 
 ( 
 response_modalities 
 = 
 [ 
 Modality 
 . 
 TEXT 
 ]), 
 ) 
 as 
 session 
 : 
 text_input 
 = 
 "Hello? Gemini, are you there?" 
 print 
 ( 
 "> " 
 , 
 text_input 
 , 
 " 
 \n 
 " 
 ) 
 await 
 session 
 . 
 send_client_content 
 ( 
 turns 
 = 
 Content 
 ( 
 role 
 = 
 "user" 
 , 
 parts 
 = 
 [ 
 Part 
 ( 
 text 
 = 
 text_input 
 )]) 
 ) 
 response 
 = 
 [] 
 async 
 for 
 message 
 in 
 session 
 . 
 receive 
 (): 
 if 
 message 
 . 
 text 
 : 
 response 
 . 
 append 
 ( 
 message 
 . 
 text 
 ) 
 print 
 ( 
 "" 
 . 
 join 
 ( 
 response 
 )) 
 # Example output: 
 # >  Hello? Gemini, are you there? 
 # Yes, I'm here. What would you like to talk about? 
 

Integration guide

This section describes how integration works with Gemini Live API.

Sessions

A WebSocket connection establishes a session between the client and the Gemini server.

After a client initiates a new connection the session can exchange messages with the server to:

• Send text, audio, or video to the Gemini server.
• Receive audio, text, or function call requests from the Gemini server.

The session configuration is sent in the first message after connection. A session configuration includes the model, generation parameters, system instructions, and tools.

See the following example configuration:

 {
  "model": string,
  "generationConfig": {
    "candidateCount": integer,
    "maxOutputTokens": integer,
    "temperature": number,
    "topP": number,
    "topK": integer,
    "presencePenalty": number,
    "frequencyPenalty": number,
    "responseModalities": [string],
    "speechConfig": object
  },

  "systemInstruction": string,
  "tools": [object]
} 

For more information, see BidiGenerateContentSetup .

Send messages

Messages are JSON-formatted objects exchanged over the WebSocket connection.

To send a message the client must send a JSON object over an open WebSocket connection. The JSON object must have exactly one of the fields from the following object set:

 {
  "setup": BidiGenerateContentSetup,
  "clientContent": BidiGenerateContentClientContent,
  "realtimeInput": BidiGenerateContentRealtimeInput,
  "toolResponse": BidiGenerateContentToolResponse
} 

Supported client messages

See the supported client messages in the following table:

Receive messages

To receive messages from Gemini, listen for the WebSocket 'message' event, and then parse the result according to the definition of the supported server messages.

See the following:

 ws 
 . 
 addEventListener 
 ( 
 "message" 
 , 
 async 
 ( 
 evt 
 ) 
 => 
 { 
 if 
 ( 
 evt 
 . 
 data 
 instanceof 
 Blob 
 ) 
 { 
 // 
 Process 
 the 
 received 
 data 
 ( 
 audio 
 , 
 video 
 , 
 etc 
 . 
 ) 
 } 
 else 
 { 
 // 
 Process 
 JSON 
 response 
 } 
 }); 

Server messages will have exactly one of the fields from the following object set:

 {
  "setupComplete": BidiGenerateContentSetupComplete,
  "serverContent": BidiGenerateContentServerContent,
  "toolCall": BidiGenerateContentToolCall,
  "toolCallCancellation": BidiGenerateContentToolCallCancellation
  "usageMetadata": UsageMetadata
  "goAway": GoAway
  "sessionResumptionUpdate": SessionResumptionUpdate
  "inputTranscription": BidiGenerateContentTranscription
  "outputTranscription": BidiGenerateContentTranscription
} 

Supported server messages

See the supported server messages in the following table:

Incremental content updates

Use incremental updates to send text input, establish session context, or restore session context. For short contexts you can send turn-by-turn interactions to represent the exact sequence of events. For longer contexts it's recommended to provide a single message summary to free up the context window for the follow up interactions.

See the following example context message:

 { 
  
 "clientContent" 
 : 
  
 { 
  
 "turns" 
 : 
  
 [ 
  
 { 
  
 "parts" 
 :[ 
  
 { 
  
 "text" 
 : 
  
 "" 
  
 } 
  
 ], 
  
 "role" 
 : 
 "user" 
  
 }, 
  
 { 
  
 "parts" 
 :[ 
  
 { 
  
 "text" 
 : 
  
 "" 
  
 } 
  
 ], 
  
 "role" 
 : 
 "model" 
  
 } 
  
 ], 
  
 "turnComplete" 
 : 
  
 true 
  
 } 
 } 

Note that while content parts can be of a functionResponse type, BidiGenerateContentClientContent shouldn't be used to provide a response to the function calls issued by the model. BidiGenerateContentToolResponse should be used instead. BidiGenerateContentClientContent should only be used to establish previous context or provide text input to the conversation.

Streaming audio and video

Code execution

To learn more about code execution, see Code execution .

Function calling

All functions must be declared at the start of the session by sending tool definitions as part of the BidiGenerateContentSetup message.

You define functions by using JSON, specifically with a select subset of the OpenAPI schema format . A single function declaration can include the following parameters:

• name(string): The unique identifier for the function within the API call.

• description(string): A comprehensive explanation of the function's purpose and capabilities.

• parameters(object): Defines the input data required by the function.

  • type(string): Specifies the overall data type, such as object.

  • properties(object): Lists individual parameters, each with:

    • type(string): The data type of the parameter, such as string, integer, boolean.
    • description(string): A clear explanation of the parameter's purpose and expected format.

  • required(array): An array of strings listing the parameter names that are mandatory for the function to operate.

For code examples of a function declaration using curl commands, see Function calling with the Gemini API . For examples of how to create function declarations using the Gemini API SDKs, see the Function calling tutorial .

From a single prompt, the model can generate multiple function calls and the code necessary to chain their outputs. This code executes in a sandbox environment, generating subsequent BidiGenerateContentToolCall messages. The execution pauses until the results of each function call are available, which ensures sequential processing.

The client should respond with BidiGenerateContentToolResponse .

To learn more, see Introduction to function calling .

Audio formats

See the list of supported audio formats .

System instructions

You can provide system instructions to better control the model's output and specify the tone and sentiment of audio responses.

System instructions are added to the prompt before the interaction begins and remain in effect for the entire session.

System instructions can only be set at the beginning of a session, immediately following the initial connection. To provide further input to the model during the session, use incremental content updates.

Interruptions

Users can interrupt the model's output at any time. When Voice activity detection (VAD) detects an interruption, the ongoing generation is canceled and discarded. Only the information already sent to the client is retained in the session history. The server then sends a BidiGenerateContentServerContent message to report the interruption.

In addition, the Gemini server discards any pending function calls and sends a BidiGenerateContentServerContent message with the IDs of the canceled calls.

Voices

To specify a voice, set the voiceName within the speechConfig object, as part of your session configuration .

See the following JSON representation of a speechConfig object:

 { 
  
 "voiceConfig" 
 : 
  
 { 
  
 "prebuiltVoiceConfig" 
 : 
  
 { 
  
 "voiceName" 
 : 
  
 " VOICE_NAME 
" 
  
 } 
  
 } 
 } 

To see the list of supported voices, see Change voice and language settings .

Limitations

Consider the following limitations of Gemini Live API and Gemini 2.0 when you plan your project.

Client authentication

Gemini Live API only provides server to server authentication and isn't recommended for direct client use. Client input should be routed through an intermediate application server for secure authentication with the Gemini Live API.

Maximum session duration

The default maximum length of a conversation session is 10 minutes. For more information, see Session length .

Video frame rate

When you stream video to the model, it is processed at 1 frame per second (FPS). This makes the API unsuitable for use cases that require analyzing fast-changing video, such as play-by-play in high-speed sports.

Voice activity detection (VAD)

By default, the model automatically performs voice activity detection (VAD) on a continuous audio input stream. VAD can be configured with the RealtimeInputConfig.AutomaticActivityDetection field of the setup message .

When the audio stream is paused for more than a second (for example, when the user switches off the microphone), an AudioStreamEnd event is sent to flush any cached audio. The client can resume sending audio data at any time.

Alternatively, the automatic VAD can be turned off by setting RealtimeInputConfig.AutomaticActivityDetection.disabled to true in the setup message. In this configuration the client is responsible for detecting user speech and sending ActivityStart and ActivityEnd messages at the appropriate times. An AudioStreamEnd isn't sent in this configuration. Instead, any interruption of the stream is marked by an ActivityEnd message.

Additional limitations

Manual endpointing isn't supported.

Audio inputs and audio outputs negatively impact the model's ability to use function calling.

Token count

Token count isn't supported.

Rate limits

The following rate limits apply:

• 5,000 concurrent sessions per project
• 4M tokens per minute

Messages and events

What's next

• Learn more about function calling .
• See the Function calling reference for examples.