This document describes how to send audio and video streams to the Live API for real-time, bidirectional communication with Gemini models. Learn how to configure and transmit audio and video data to build dynamic and interactive applications.
Send audio streams
Implementing real-time audio requires strict adherence to sample rate specifications and careful buffer management to ensure low latency and natural interruptibility.
The Live API supports the following audio formats:
- Input audio: Raw 16-bit PCM audio at 16 kHz, little-endian
- Output audio: Raw 16-bit PCM audio at 24 kHz, little-endian
The following code sample shows you how to send streaming audio data:
import
asyncio
# Assumes session is an active Live API session
# and chunk_data contains bytes of raw 16-bit PCM audio at 16 kHz.
from
google.genai
import
types
# Send audio input data in chunks
await
session
.
send_realtime_input
(
audio
=
types
.
Blob
(
data
=
chunk_data
,
mime_type
=
"audio/pcm;rate=16000"
)
)
The client must maintain a playback buffer. The server streams audio in chunks
within server_content
messages. The client's responsibility is to decode,
buffer, and play the data.
The following code sample shows you how to process streaming audio data:
import
asyncio
# Assumes session is an active Live API session
# and audio_queue is an asyncio.Queue for buffering audio for playback.
import
numpy
as
np
async
for
msg
in
session
.
receive
():
server_content
=
msg
.
server_content
if
server_content
:
# 1. Handle Interruption
if
server_content
.
interrupted
:
print
(
"
\n
[Interrupted] Flushing buffer..."
)
# Clear the Python queue
while
not
audio_queue
.
empty
():
try
:
audio_queue
.
get_nowait
()
except
asyncio
.
QueueEmpty
:
break
# Send signal to worker to reset hardware buffers if needed
await
audio_queue
.
put
(
None
)
continue
# 2. Process Audio chunks
if
server_content
.
model_turn
:
for
part
in
server_content
.
model_turn
.
parts
:
if
part
.
inline_data
:
# Add PCM data to playback queue
await
audio_queue
.
put
(
np
.
frombuffer
(
part
.
inline_data
.
data
,
dtype
=
'int16'
))
Send video streams
Video streaming provides visual context. The Live API expects a sequence of discrete image frames and supports video frames input at 1 FPS. For best results, use native 768x768 resolution at 1 FPS.
The following code sample shows you how to send streaming video data:
import
asyncio
# Assumes session is an active Live API session
# and chunk_data contains bytes of a JPEG image.
from
google.genai
import
types
# Send video input data in chunks
await
session
.
send_realtime_input
(
media
=
types
.
Blob
(
data
=
chunk_data
,
mime_type
=
"image/jpeg"
)
)
The client implementation captures a frame from the video feed, encodes it as a
JPEG blob, and transmits it using the realtime_input
message structure.
import
cv2
import
asyncio
from
google.genai
import
types
async
def
send_video_stream
(
session
):
# Open webcam
cap
=
cv2
.
VideoCapture
(
0
)
while
True
:
ret
,
frame
=
cap
.
read
()
if
not
ret
:
break
# 1. Resize to optimal resolution (768x768 max)
frame
=
cv2
.
resize
(
frame
,
(
768
,
768
))
# 2. Encode as JPEG
_
,
buffer
=
cv2
.
imencode
(
'.jpg'
,
frame
,)
# 3. Send as realtime input
await
session
.
send_realtime_input
(
media
=
types
.
Blob
(
data
=
buffer
.
tobytes
(),
mime_type
=
"image/jpeg"
)
)
# 4. Wait 1 second (1 FPS)
await
asyncio
.
sleep
(
1.0
)
cap
.
release
()
Configure media resolution
You can specify the resolution for input media by setting the media_resolution
field in the session configuration. Lower resolution
reduces token usage and latency, while higher resolution improves detail
recognition. Supported values include low
, medium
, and high
.
config
=
{
"response_modalities"
:
[
"audio"
],
"media_resolution"
:
"low"
,
}
