Image labeling

Key capabilities

• A powerful general-purpose base classifierRecognizes more than 400 categories that describe the most commonly found objects in photos.
• Tailor to your use case with custom modelsUse other pre-trained models from TensorFlow Hub or your own custom model trained with TensorFlow, AutoML Vision Edge or TensorFlow Lite Model maker.
• Easy-to-use high-level APIsNo need to deal with low-level model input/output, image pre- and post-processing, or building a processing pipeline. ML Kit extracts the labels from the TensorFlow Lite model and provides them as a text description.

Note that this API is intended for image classification models that describe the full image. For classifying one or more objects in an image, such as shoes or pieces of furniture, the Object Detection & Tracking API may be a better fit.

Supported image classification models

The Image Labeling APIs support different image classification models:

Supported image classification models
Base model	By default the API uses a powerful general-purpose image labeling model that recognizes more than 400 entities that cover the most commonly-found concepts in photos.
Custom TensorFlow Lite models	To target application-specific concepts, the API accepts custom image classification models from a wide range of sources. These can be pre-trained models downloaded from TensorFlow Hub or your own models trained with AutoML Vision Edge, TensorFlow Lite Model Maker or TensorFlow itself. Models can be bundled with your app or hosted with Firebase Machine Learning and downloaded at run-time.

Using the base model

ML Kit’s base model returns a list of entities that identify people, things, places, activities, and so on. Each entity comes with a score that indicates the confidence the ML model has in its relevance. With this information, you can perform tasks such as automatic metadata generation and content moderation. The default model provided with ML Kit recognizes more than 400 different entities .

iOS Android

Example labels

The base model in the image labeling API supports 400+ labels, such as the following examples:

Category	Example labels
People	Crowd Selfie Smile
Activities	Dancing Eating Surfing
Things	Car Piano Receipt
Animals	Bird Cat Dog
Plants	Flower Fruit Vegetable
Places	Beach Lake Mountain

Example results

Here is an example of the entities that were recognized in the accompanying photo.

Using a custom TensorFlow Lite model

ML Kit's base image labeling model is built for general-purpose use. It's trained to recognize 400 categories that describe the most commonly-found objects in photos. Your app might need a specialized image classification model that recognizes a narrower number of categories in more detail, such as a model that distinguishes between species of flowers or types of food.

This API lets you tailor to a particular use case by supporting custom image classification models from a wide range of sources. Please refer to Custom models with ML Kit to learn more. Custom models can be bundled with your app or dynamically downloaded from the cloud using Firebase Machine Learning's Model deployment service.

iOS Android

Input image preprocessing

If needed, Image Labeling uses bilinear image scaling and stretching to adjust the input image size and aspect ratio so that they fit the requirements of the underlying model.