mp.tasks.audio.AudioEmbedder

View source on GitHub

Class that performs embedding extraction on audio clips or audio stream.

  mp 
 . 
 tasks 
 . 
 audio 
 . 
 AudioEmbedder 
 ( 
 graph_config 
 : 
   mp 
 . 
 calculators 
 . 
 core 
 . 
 constant_side_packet_calculator_pb2 
 . 
 mediapipe_dot_framework_dot_calculator__pb2 
 . 
 CalculatorGraphConfig 
 
 
 , 
 running_mode 
 : 
   mp 
 . 
 tasks 
 . 
 audio 
 . 
 RunningMode 
 
 
 , 
 packet_callback 
 : 
 Optional 
 [ 
 Callable 
 [[ 
 Mapping 
 [ 
 str 
 , 
 packet_module 
 . 
 Packet 
 ]], 
 None 
 ]] 
 = 
 None 
 ) 
 -> 
 None 
 

This API expects a TFLite model with mandatory TFLite Model Metadata that contains the mandatory AudioProperties of the solo input audio tensor and the optional (but recommended) label items as AssociatedFiles with type TENSOR_AXIS_LABELS per output embedding tensor.

At least one output tensor with: (kTfLiteUInt8/kTfLiteFloat32)

• N components corresponding to the N dimensions of the returned feature vector for this output layer.
• Either 2 or 4 dimensions, i.e. [1 x N] or [1 x 1 x 1 x N] .

Methods

close

View source

  close 
 () 
 -> 
 None 
 

Shuts down the mediapipe audio task instance.

create_audio_record

View source

  create_audio_record 
 ( 
 num_channels 
 : 
 int 
 , 
 sample_rate 
 : 
 int 
 , 
 required_input_buffer_size 
 : 
 int 
 ) 
 -> 
 audio_record 
 . 
 AudioRecord 
 

Creates an AudioRecord instance to record audio stream.

The returned AudioRecord instance is initialized and client needs to call the appropriate method to start recording.

Note that MediaPipe Audio tasks will up/down sample automatically to fit the sample rate required by the model. The default sample rate of the MediaPipe pretrained audio model, Yamnet is 16kHz.

create_from_model_path

View source

  @classmethod 
 
  create_from_model_path 
 ( 
 model_path 
 : 
 str 
 ) 
 -> 
 'AudioEmbedder' 
 

Creates an AudioEmbedder object from a TensorFlow Lite model and the default AudioEmbedderOptions .

Note that the created AudioEmbedder instance is in audio clips mode, for embedding extraction on the independent audio clips.

create_from_options

View source

  @classmethod 
 
  create_from_options 
 ( 
 options 
 : 
   mp 
 . 
 tasks 
 . 
 audio 
 . 
 AudioEmbedderOptions 
 
 
 ) 
 -> 
 'AudioEmbedder' 
 

Creates the AudioEmbedder object from audio embedder options.

embed

View source

  embed 
 ( 
 audio_clip 
 : 
   mp 
 . 
 tasks 
 . 
 components 
 . 
 containers 
 . 
 AudioData 
 
 
 ) 
 -> 
 List 
 [ 
   mp 
 . 
 tasks 
 . 
 audio 
 . 
 AudioEmbedderResult 
 
 
 ] 
 

Performs embedding extraction on the provided audio clips.

The audio clip is represented as a MediaPipe AudioData. The method accepts audio clips with various length and audio sample rate. It's required to provide the corresponding audio sample rate within the AudioData object.

The input audio clip may be longer than what the model is able to process in a single inference. When this occurs, the input audio clip is split into multiple chunks starting at different timestamps. For this reason, this function returns a vector of EmbeddingResult objects, each associated ith a timestamp corresponding to the start (in milliseconds) of the chunk data on which embedding extraction was carried out.

embed_async

View source

  embed_async 
 ( 
 audio_block 
 : 
   mp 
 . 
 tasks 
 . 
 components 
 . 
 containers 
 . 
 AudioData 
 
 
 , 
 timestamp_ms 
 : 
 int 
 ) 
 -> 
 None 
 

Sends audio data (a block in a continuous audio stream) to perform audio embedding extraction.

Only use this method when the AudioEmbedder is created with the audio stream running mode. The input timestamps should be monotonically increasing for adjacent calls of this method. This method will return immediately after the input audio data is accepted. The results will be available via the result_callback provided in the AudioEmbedderOptions . The embed_async method is designed to process auido stream data such as microphone input.

The input audio data may be longer than what the model is able to process in a single inference. When this occurs, the input audio block is split into multiple chunks. For this reason, the callback may be called multiple times (once per chunk) for each call to this function.

The result_callback provides:

• An AudioEmbedderResult object that contains a list of embeddings.
• The input timestamp in milliseconds.

__enter__

View source

  __enter__ 
 () 
 

Return self upon entering the runtime context.

__exit__

View source

  __exit__ 
 ( 
 unused_exc_type 
 , 
 unused_exc_value 
 , 
 unused_traceback 
 ) 
 

Shuts down the mediapipe audio task instance on exit of the context manager.