Input data format and structure

To build a new index or update an existing index, provide vectors to Vector Search in the format and structure described in the following sections.

Prerequisites

Store your input data in a Cloud Storage bucket , in your Google Cloud project.

Input data files should be organized as follows:

• Each batch of input data files should be under a single Cloud Storage directory.
• Data files should be placed directly under batch_root and named with the following suffixes: .csv , .json , and .avro .
• There is a limit of 5000 objects (files) in the batch root directory.
• Each data file is interpreted as a set of records. The format of the record is determined by the suffix of the filename and those format requirements are described. See Data file formats .
• Each record should have an id , a feature vector, and your optional fields supported by Vertex AI Feature Store, like restricts and crowding.
• A subdirectory named delete may be present. Each file directly under batch_root / delete is taken as a text file of id records with one id in each line.
• All other subdirectories are not allowed.
• Transcoding of gzip-compressed files isn't supported as input data.

Input data processing

• All records from all data files, including those under delete , consist of a single batch of input.
• The relative ordering of records within a data file is not important.
• A single ID should only appear once in a batch. If there is a duplicate with the same ID, it displays as one vector count.
• An ID cannot appear both in a regular data file and a delete data file.
• All IDs from a data file under delete causes it to be removed from the next index version.
• Records from regular data files is included in the next version, overwriting a value in an older index version.

The following are examples of dense, sparse, and hybrid embeddings:

• Dense embeddings:

    { 
   "id" 
   : 
    
   "1" 
   , 
    
   "embedding" 
   : 
    
   [ 
   1 
   , 
   1 
   , 
   1 
   ]} 
   { 
   "id" 
   : 
    
   "2" 
   , 
    
   "embedding" 
   : 
    
   [ 
   2 
   , 
   2 
   , 
   2 
   ]} 
   
  

• Sparse embeddings:

    { 
   "id" 
   : 
    
   "3" 
   , 
    
   "sparse_embedding" 
   : 
    
   { 
   "values" 
   : 
    
   [ 
   0.1 
   , 
    
   0.2 
   ], 
    
   "dimensions" 
   : 
    
   [ 
   1 
   , 
    
   4 
   ]}} 
   { 
   "id" 
   : 
    
   "4" 
   , 
    
   "sparse_embedding" 
   : 
    
   { 
   "values" 
   : 
    
   [ 
   -0.4 
   , 
    
   0.2 
   , 
    
   -1.3 
   ], 
    
   "dimensions" 
   : 
    
   [ 
   10 
   , 
    
   20 
   , 
    
   20 
   ]}} 
   
  

• Hybrid embeddings:

    { 
   "id" 
   : 
    
   "5" 
   , 
    
   "embedding" 
   : 
    
   [ 
   5 
   , 
    
   5 
   , 
    
   -5 
   ], 
    
   "sparse_embedding" 
   : 
    
   { 
   "values" 
   : 
    
   [ 
   0.1 
   ], 
    
   "dimensions" 
   : 
    
   [ 
   500 
   ]}} 
   { 
   "id" 
   : 
    
   "6" 
   , 
    
   "embedding" 
   : 
    
   [ 
   6 
   , 
    
   7 
   , 
    
   -8.1 
   ], 
    
   "sparse_embedding" 
   : 
    
   { 
   "values" 
   : 
    
   [ 
   0.1 
   , 
    
   -0.2 
   ], 
    
   "dimensions" 
   : 
    
   [ 
   40 
   , 
    
   901 
   ]}} 
   
  

The following is an example of a valid input data file organization:

  ba 
 t 
 ch_roo 
 t 
 / 
  
 feature 
 _ 
 f 
 ile_ 
 1. 
 csv 
  
 feature 
 _ 
 f 
 ile_ 
 2. 
 csv 
  
 dele 
 te 
 / 
  
 dele 
 te 
 _ 
 f 
 ile. 
 t 
 x 
 t 
 

The feature_file_1.csv and feature_file_2.csv files contain records in CSV format. The delete_file.txt file contains a list of record IDs to be deleted from the next index version.

Data file formats

What's next

• Learn how to Create and manage your index