You can use ML Kit to detect and track objects across frames of video.
When you pass ML Kit images, ML Kit returns, for each image, a list of up to five detected objects and their position in the image. When detecting objects in video streams, every object has an ID that you can use to track the object across images. You can also optionally enable coarse object classification, which labels objects with broad category descriptions.
Before you begin
- If you haven't already, add Firebase to your Android project .
- Add the dependencies for the ML Kit Android libraries to your module
(app-level) Gradle file (usually
app/build.gradle):apply plugin : ' com . android . application ' apply plugin : ' com . google . gms . google - services ' dependencies { // ... implementation ' com . google . firebase : firebase - ml - vision : 24.0.3 ' implementation ' com . google . firebase : firebase - ml - vision - object - detection - model : 19.0.6 ' }
1. Configure the object detector
To start detecting and tracking objects, first create an instance of FirebaseVisionObjectDetector
, optionally specifying any detector settings you
want to change from the default.
-
Configure the object detector for your use case with a
FirebaseVisionObjectDetectorOptionsobject. You can change the following settings:Object Detector SettingsDetection modeSTREAM_MODE(default) |SINGLE_IMAGE_MODEIn
STREAM_MODE(default), the object detector runs with low latency, but might produce incomplete results (such as unspecified bounding boxes or category labels) on the first few invocations of the detector. Also, inSTREAM_MODE, the detector assigns tracking IDs to objects, which you can use to track objects across frames. Use this mode when you want to track objects, or when low latency is important, such as when processing video streams in real time.In
SINGLE_IMAGE_MODE, the object detector waits until a detected object's bounding box and (if you enabled classification) category label are available before returning a result. As a consequence, detection latency is potentially higher. Also, inSINGLE_IMAGE_MODE, tracking IDs are not assigned. Use this mode if latency isn't critical and you don't want to deal with partial results.Detect and track multiple objectsfalse(default) |trueWhether to detect and track up to five objects or only the most prominent object (default).
Classify objectsfalse(default) |trueWhether or not to classify detected objects into coarse categories. When enabled, the object detector classifies objects into the following categories: fashion goods, food, home goods, places, plants, and unknown.
The object detection and tracking API is optimized for these two core use cases:
- Live detection and tracking of the most prominent object in the camera viewfinder
- Detection of multiple objects from a static image
To configure the API for these use cases:
Java
// Live detection and tracking FirebaseVisionObjectDetectorOptions options = new FirebaseVisionObjectDetectorOptions . Builder () . setDetectorMode ( FirebaseVisionObjectDetectorOptions . STREAM_MODE ) . enableClassification () // Optional . build (); // Multiple object detection in static images FirebaseVisionObjectDetectorOptions options = new FirebaseVisionObjectDetectorOptions . Builder () . setDetectorMode ( FirebaseVisionObjectDetectorOptions . SINGLE_IMAGE_MODE ) . enableMultipleObjects () . enableClassification () // Optional . build ();Kotlin
// Live detection and tracking val options = FirebaseVisionObjectDetectorOptions . Builder () . setDetectorMode ( FirebaseVisionObjectDetectorOptions . STREAM_MODE ) . enableClassification () // Optional . build () // Multiple object detection in static images val options = FirebaseVisionObjectDetectorOptions . Builder () . setDetectorMode ( FirebaseVisionObjectDetectorOptions . SINGLE_IMAGE_MODE ) . enableMultipleObjects () . enableClassification () // Optional . build () -
Get an instance of
FirebaseVisionObjectDetector:Java
FirebaseVisionObjectDetector objectDetector = FirebaseVision . getInstance (). getOnDeviceObjectDetector (); // Or, to change the default settings: FirebaseVisionObjectDetector objectDetector = FirebaseVision . getInstance (). getOnDeviceObjectDetector ( options );Kotlin
val objectDetector = FirebaseVision . getInstance (). getOnDeviceObjectDetector () // Or, to change the default settings: val objectDetector = FirebaseVision . getInstance (). getOnDeviceObjectDetector ( options )
2. Run the object detector
To detect and track objects, pass images to the FirebaseVisionObjectDetector
instance's processImage()
method.
For each frame of video or image in a sequence, do the following:
-
Create a
FirebaseVisionImageobject from your image.-
To create a
FirebaseVisionImageobject from amedia.Imageobject, such as when capturing an image from a device's camera, pass themedia.Imageobject and the image's rotation toFirebaseVisionImage.fromMediaImage().If you use the CameraX library, the
OnImageCapturedListenerandImageAnalysis.Analyzerclasses calculate the rotation value for you, so you just need to convert the rotation to one of ML Kit'sROTATION_constants before callingFirebaseVisionImage.fromMediaImage():Java
private class YourAnalyzer implements ImageAnalysis . Analyzer { private int degreesToFirebaseRotation ( int degrees ) { switch ( degrees ) { case 0 : return FirebaseVisionImageMetadata . ROTATION_0 ; case 90 : return FirebaseVisionImageMetadata . ROTATION_90 ; case 180 : return FirebaseVisionImageMetadata . ROTATION_180 ; case 270 : return FirebaseVisionImageMetadata . ROTATION_270 ; default : throw new IllegalArgumentException ( "Rotation must be 0, 90, 180, or 270." ); } } @Override public void analyze ( ImageProxy imageProxy , int degrees ) { if ( imageProxy == null || imageProxy . getImage () == null ) { return ; } Image mediaImage = imageProxy . getImage (); int rotation = degreesToFirebaseRotation ( degrees ); FirebaseVisionImage image = FirebaseVisionImage . fromMediaImage ( mediaImage , rotation ); // Pass image to an ML Kit Vision API // ... } }
Kotlin
private class YourImageAnalyzer : ImageAnalysis . Analyzer { private fun degreesToFirebaseRotation ( degrees : Int ): Int = when ( degrees ) { 0 -> FirebaseVisionImageMetadata . ROTATION_0 90 -> FirebaseVisionImageMetadata . ROTATION_90 180 -> FirebaseVisionImageMetadata . ROTATION_180 270 -> FirebaseVisionImageMetadata . ROTATION_270 else -> throw Exception ( "Rotation must be 0, 90, 180, or 270." ) } override fun analyze ( imageProxy : ImageProxy?, degrees : Int ) { val mediaImage = imageProxy ?. image val imageRotation = degreesToFirebaseRotation ( degrees ) if ( mediaImage != null ) { val image = FirebaseVisionImage . fromMediaImage ( mediaImage , imageRotation ) // Pass image to an ML Kit Vision API // ... } } }
If you don't use a camera library that gives you the image's rotation, you can calculate it from the device's rotation and the orientation of camera sensor in the device:
Java
private static final SparseIntArray ORIENTATIONS = new SparseIntArray (); static { ORIENTATIONS . append ( Surface . ROTATION_0 , 90 ); ORIENTATIONS . append ( Surface . ROTATION_90 , 0 ); ORIENTATIONS . append ( Surface . ROTATION_180 , 270 ); ORIENTATIONS . append ( Surface . ROTATION_270 , 180 ); } /** * Get the angle by which an image must be rotated given the device's current * orientation. */ @RequiresApi ( api = Build . VERSION_CODES . LOLLIPOP ) private int getRotationCompensation ( String cameraId , Activity activity , Context context ) throws CameraAccessException { // Get the device's current rotation relative to its "native" orientation. // Then, from the ORIENTATIONS table, look up the angle the image must be // rotated to compensate for the device's rotation. int deviceRotation = activity . getWindowManager (). getDefaultDisplay (). getRotation (); int rotationCompensation = ORIENTATIONS . get ( deviceRotation ); // On most devices, the sensor orientation is 90 degrees, but for some // devices it is 270 degrees. For devices with a sensor orientation of // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees. CameraManager cameraManager = ( CameraManager ) context . getSystemService ( CAMERA_SERVICE ); int sensorOrientation = cameraManager . getCameraCharacteristics ( cameraId ) . get ( CameraCharacteristics . SENSOR_ORIENTATION ); rotationCompensation = ( rotationCompensation + sensorOrientation + 270 ) % 360 ; // Return the corresponding FirebaseVisionImageMetadata rotation value. int result ; switch ( rotationCompensation ) { case 0 : result = FirebaseVisionImageMetadata . ROTATION_0 ; break ; case 90 : result = FirebaseVisionImageMetadata . ROTATION_90 ; break ; case 180 : result = FirebaseVisionImageMetadata . ROTATION_180 ; break ; case 270 : result = FirebaseVisionImageMetadata . ROTATION_270 ; break ; default : result = FirebaseVisionImageMetadata . ROTATION_0 ; Log . e ( TAG , "Bad rotation value: " + rotationCompensation ); } return result ; }
Kotlin
private val ORIENTATIONS = SparseIntArray () init { ORIENTATIONS . append ( Surface . ROTATION_0 , 90 ) ORIENTATIONS . append ( Surface . ROTATION_90 , 0 ) ORIENTATIONS . append ( Surface . ROTATION_180 , 270 ) ORIENTATIONS . append ( Surface . ROTATION_270 , 180 ) } /** * Get the angle by which an image must be rotated given the device's current * orientation. */ @RequiresApi ( api = Build . VERSION_CODES . LOLLIPOP ) @Throws ( CameraAccessException :: class ) private fun getRotationCompensation ( cameraId : String , activity : Activity , context : Context ): Int { // Get the device's current rotation relative to its "native" orientation. // Then, from the ORIENTATIONS table, look up the angle the image must be // rotated to compensate for the device's rotation. val deviceRotation = activity . windowManager . defaultDisplay . rotation var rotationCompensation = ORIENTATIONS . get ( deviceRotation ) // On most devices, the sensor orientation is 90 degrees, but for some // devices it is 270 degrees. For devices with a sensor orientation of // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees. val cameraManager = context . getSystemService ( CAMERA_SERVICE ) as CameraManager val sensorOrientation = cameraManager . getCameraCharacteristics ( cameraId ) . get ( CameraCharacteristics . SENSOR_ORIENTATION ) !! rotationCompensation = ( rotationCompensation + sensorOrientation + 270 ) % 360 // Return the corresponding FirebaseVisionImageMetadata rotation value. val result : Int when ( rotationCompensation ) { 0 - > result = FirebaseVisionImageMetadata . ROTATION_0 90 - > result = FirebaseVisionImageMetadata . ROTATION_90 180 - > result = FirebaseVisionImageMetadata . ROTATION_180 270 - > result = FirebaseVisionImageMetadata . ROTATION_270 else - > { result = FirebaseVisionImageMetadata . ROTATION_0 Log . e ( TAG , "Bad rotation value: $ rotationCompensation " ) } } return result }
Then, pass the
media.Imageobject and the rotation value toFirebaseVisionImage.fromMediaImage():Java
FirebaseVisionImage image = FirebaseVisionImage . fromMediaImage ( mediaImage , rotation );
Kotlin
val image = FirebaseVisionImage . fromMediaImage ( mediaImage , rotation )
- To create a
FirebaseVisionImageobject from a file URI, pass the app context and file URI toFirebaseVisionImage.fromFilePath(). This is useful when you use anACTION_GET_CONTENTintent to prompt the user to select an image from their gallery app.Java
FirebaseVisionImage image ; try { image = FirebaseVisionImage . fromFilePath ( context , uri ); } catch ( IOException e ) { e . printStackTrace (); }
Kotlin
val image : FirebaseVisionImage try { image = FirebaseVisionImage . fromFilePath ( context , uri ) } catch ( e : IOException ) { e . printStackTrace () }
- To create a
FirebaseVisionImageobject from aByteBufferor a byte array, first calculate the image rotation as described above formedia.Imageinput.Then, create a
FirebaseVisionImageMetadataobject that contains the image's height, width, color encoding format, and rotation:Java
FirebaseVisionImageMetadata metadata = new FirebaseVisionImageMetadata . Builder () . setWidth ( 480 ) // 480x360 is typically sufficient for . setHeight ( 360 ) // image recognition . setFormat ( FirebaseVisionImageMetadata . IMAGE_FORMAT_NV21 ) . setRotation ( rotation ) . build ();
Kotlin
val metadata = FirebaseVisionImageMetadata . Builder () . setWidth ( 480 ) // 480x360 is typically sufficient for . setHeight ( 360 ) // image recognition . setFormat ( FirebaseVisionImageMetadata . IMAGE_FORMAT_NV21 ) . setRotation ( rotation ) . build ()
Use the buffer or array, and the metadata object, to create a
FirebaseVisionImageobject:Java
FirebaseVisionImage image = FirebaseVisionImage . fromByteBuffer ( buffer , metadata ); // Or: FirebaseVisionImage image = FirebaseVisionImage.fromByteArray(byteArray, metadata);
-
