mp.tasks.audio.AudioClassifier

View source on GitHub

Class that performs audio classification on audio data.

  mp 
 . 
 tasks 
 . 
 audio 
 . 
 AudioClassifier 
 ( 
 graph_config 
 : 
   mp 
 . 
 calculators 
 . 
 core 
 . 
 constant_side_packet_calculator_pb2 
 . 
 mediapipe_dot_framework_dot_calculator__pb2 
 . 
 CalculatorGraphConfig 
 
 
 , 
 running_mode 
 : 
   mp 
 . 
 tasks 
 . 
 audio 
 . 
 RunningMode 
 
 
 , 
 packet_callback 
 : 
 Optional 
 [ 
 Callable 
 [[ 
 Mapping 
 [ 
 str 
 , 
 packet_module 
 . 
 Packet 
 ]], 
 None 
 ]] 
 = 
 None 
 ) 
 -> 
 None 
 

This API expects a TFLite model with mandatory TFLite Model Metadata that contains the mandatory AudioProperties of the solo input audio tensor and the optional (but recommended) category labels as AssociatedFiles with type TENSOR_AXIS_LABELS per output classification tensor.

At least one output tensor with: (kTfLiteFloat32)

• [1 x N] array with N represents the number of categories.
• optional (but recommended) category labels as AssociatedFiles with type TENSOR_AXIS_LABELS, containing one label per line. The first such AssociatedFile (if any) is used to fill the category_name field of the results. The display_name field is filled from the AssociatedFile (if any) whose locale matches the display_names_locale field of the AudioClassifierOptions used at creation time ("en" by default, i.e. English). If none of these are available, only the index field of the results will be filled.

Methods

classify

View source

  classify 
 ( 
 audio_clip 
 : 
   mp 
 . 
 tasks 
 . 
 components 
 . 
 containers 
 . 
 AudioData 
 
 
 ) 
 -> 
 List 
 [ 
   mp 
 . 
 tasks 
 . 
 audio 
 . 
 AudioClassifierResult 
 
 
 ] 
 

Performs audio classification on the provided audio clip.

The audio clip is represented as a MediaPipe AudioData. The method accepts audio clips with various length and audio sample rate. It's required to provide the corresponding audio sample rate within the AudioData object.

The input audio clip may be longer than what the model is able to process in a single inference. When this occurs, the input audio clip is split into multiple chunks starting at different timestamps. For this reason, this function returns a vector of ClassificationResult objects, each associated ith a timestamp corresponding to the start (in milliseconds) of the chunk data that was classified, e.g:

ClassificationResult #0 (first chunk of data): timestamp_ms: 0 (starts at 0ms) classifications #0 (single head model): category #0: category_name: "Speech" score: 0.6 category #1: category_name: "Music" score: 0.2 ClassificationResult #1 (second chunk of data): timestamp_ms: 800 (starts at 800ms) classifications #0 (single head model): category #0: category_name: "Speech" score: 0.5 category #1: category_name: "Silence" score: 0.1

classify_async

View source

  classify_async 
 ( 
 audio_block 
 : 
   mp 
 . 
 tasks 
 . 
 components 
 . 
 containers 
 . 
 AudioData 
 
 
 , 
 timestamp_ms 
 : 
 int 
 ) 
 -> 
 None 
 

Sends audio data (a block in a continuous audio stream) to perform audio classification.

Only use this method when the AudioClassifier is created with the audio stream running mode. The input timestamps should be monotonically increasing for adjacent calls of this method. This method will return immediately after the input audio data is accepted. The results will be available via the result_callback provided in the AudioClassifierOptions . The classify_async method is designed to process auido stream data such as microphone input.

The input audio data may be longer than what the model is able to process in a single inference. When this occurs, the input audio block is split into multiple chunks. For this reason, the callback may be called multiple times (once per chunk) for each call to this function.

The result_callback provides:

• An AudioClassifierResult object that contains a list of classifications.
• The input timestamp in milliseconds.

close

View source

  close 
 () 
 -> 
 None 
 

Shuts down the mediapipe audio task instance.

create_audio_record

View source

  create_audio_record 
 ( 
 num_channels 
 : 
 int 
 , 
 sample_rate 
 : 
 int 
 , 
 required_input_buffer_size 
 : 
 int 
 ) 
 -> 
 audio_record 
 . 
 AudioRecord 
 

Creates an AudioRecord instance to record audio stream.

The returned AudioRecord instance is initialized and client needs to call the appropriate method to start recording.

Note that MediaPipe Audio tasks will up/down sample automatically to fit the sample rate required by the model. The default sample rate of the MediaPipe pretrained audio model, Yamnet is 16kHz.

create_from_model_path

View source

  @classmethod 
 
  create_from_model_path 
 ( 
 model_path 
 : 
 str 
 ) 
 -> 
 'AudioClassifier' 
 

Creates an AudioClassifier object from a TensorFlow Lite model and the default AudioClassifierOptions .

Note that the created AudioClassifier instance is in audio clips mode, for classifying on independent audio clips.

create_from_options

View source

  @classmethod 
 
  create_from_options 
 ( 
 options 
 : 
   mp 
 . 
 tasks 
 . 
 audio 
 . 
 AudioClassifierOptions 
 
 
 ) 
 -> 
 'AudioClassifier' 
 

Creates the AudioClassifier object from audio classifier options.

__enter__

View source

  __enter__ 
 () 
 

Return self upon entering the runtime context.

__exit__

View source

  __exit__ 
 ( 
 unused_exc_type 
 , 
 unused_exc_value 
 , 
 unused_traceback 
 ) 
 

Shuts down the mediapipe audio task instance on exit of the context manager.