Image search allows searching for similar images in a database of images. It works by embedding the search query into a high-dimensional vector representing the semantic meaning of the query, followed by similarity search in a predefined, custom index using ScaNN (Scalable Nearest Neighbors).
As opposed to image classification , expanding the number of items that can be recognized doesn't require re-training the entire model. New items can be added simply re-building the index. This also enables working with larger (100k+ items) databases of images.
Use the Task Library ImageSearcher
API to deploy your custom image searcher
into your mobile apps.
Key features of the ImageSearcher API
-
Takes a single image as input, performs embedding extraction and nearest-neighbor search in the index.
-
Input image processing, including rotation, resizing, and color space conversion.
-
Region of interest of the input image.
Prerequisites
Before using the ImageSearcher
API, an index needs to be built based on the
custom corpus of images to search into. This can be achieved using Model Maker
Searcher
API
by following and adapting the tutorial
.
For this you will need:
- a TFLite image embedder model such as mobilenet
v3
.
See more pretrained embedder models (a.k.a feature vector models) from the Google Image Modules collection on Kaggle
Models
.
- your corpus of images.
After this step, you should have a standalone TFLite searcher model (e.g. mobilenet_v3_searcher.tflite
), which is the original image embedder model with
the index attached into the TFLite Model
Metadata
.
Run inference in Java
Step 1: Import Gradle dependency and other settings
Copy the .tflite
searcher model file to the assets directory of the Android
module where the model will be run. Specify that the file should not be
compressed, and add the TensorFlow Lite library to the module’s build.gradle
file:
android
{
// Other settings
// Specify tflite index file should not be compressed for the app apk
aaptOptions
{
noCompress
"tflite"
}
}
dependencies
{
// Other dependencies
// Import the Task Vision Library dependency
implementation
'
org
.
tensorflow
:
tensorflow
-
lite
-
task
-
vision
:
0.4.4
'
// Import the GPU delegate plugin Library for GPU inference
implementation
'
org
.
tensorflow
:
tensorflow
-
lite
-
gpu
-
delegate
-
plugin
:
0.4.4
'
}
Step 2: Using the model
// Initialization
ImageSearcherOptions
options
=
ImageSearcherOptions
.
builder
()
.
setBaseOptions
(
BaseOptions
.
builder
().
useGpu
().
build
())
.
setSearcherOptions
(
SearcherOptions
.
builder
().
setL2Normalize
(
true
).
build
())
.
build
();
ImageSearcher
imageSearcher
=
ImageSearcher
.
createFromFileAndOptions
(
context
,
modelFile
,
options
);
// Run inference
List<NearestNeighbor>
results
=
imageSearcher
.
search
(
image
);
See the source code and
javadoc
for more options to configure the ImageSearcher
.
Run inference in C++
// Initialization
ImageSearcherOptions
options
;
options
.
mutable_base_options
()
-
> mutable_model_file
()
-
> set_file_name
(
model_path
);
options
.
mutable_embedding_options
()
-
> set_l2_normalize
(
true
);
std
::
unique_ptr<ImageSearcher>
image_searcher
=
ImageSearcher
::
CreateFromOptions
(
options
).
value
();
// Create input frame_buffer from your inputs, `image_data` and `image_dimension`.
// See more information here: tensorflow_lite_support/cc/task/vision/utils/frame_buffer_common_utils.h
std
::
unique_ptr<FrameBuffer>
frame_buffer
=
CreateFromRgbRawBuffer
(
image_data
,
image_dimension
);
// Run inference
const
SearchResult
result
=
image_searcher
-
> Search
(
*
frame_buffer
).
value
();
See the source
code
for more options to configure ImageSearcher
.
Run inference in Python
Step 1: Install TensorFlow Lite Support Pypi package.
You can install the TensorFlow Lite Support Pypi package using the following command:
pip
install
tflite-support
Step 2: Using the model
from
tflite_support.task
import
vision
# Initialization
image_searcher
=
vision
.
ImageSearcher
.
create_from_file
(
model_path
)
# Run inference
image
=
vision
.
TensorImage
.
create_from_file
(
image_file
)
result
=
image_searcher
.
search
(
image
)
See the source
code
for more options to configure ImageSearcher
.
Example results
Results:
Rank#0:
metadata: burger
distance: 0.13452
Rank#1:
metadata: car
distance: 1.81935
Rank#2:
metadata: bird
distance: 1.96617
Rank#3:
metadata: dog
distance: 2.05610
Rank#4:
metadata: cat
distance: 2.06347
Try out the simple CLI demo tool for ImageSearcher with your own model and test data.
