Configure field settings

This page shows you how to configure the schema fields to set up an app for structured data, for unstructured data with metadata, or for website data with custom, structured attributes.

Field settings help determine how Agent Search uses fields in its results. You can use the Schematab in the Google Cloud console to configure field settings.

Configuring field settings is available only for apps with data stores containing either structured data or unstructured data with metadata.

Field settings

The following field settings are available for many field types in your search or recommendations data, but not for all data types. A schema contains multiple field settings for individual fields, and the next table contains settings which can be applied to a field within a schema. Using structured data is highly recommended for these field settings:

Differences between commonly used settings

There are key differences between the indexable, searchable, and retrievable field settings. The table summarizes these differences.

In practice, many fields that are crucial for user experience (such as titles, descriptions, and identifying information) are often set to be indexable , searchable , and retrievable .

Limitations

Field settings have the following limitations:

• You can configure up to 50 fields as indexable, searchable, retrievable, or dynamic facetable.
• To configure a field as dynamic facetable, it must first be configured as indexable.
• Changing the indexable setting requires re-indexing the data, which can take hours, especially for large data stores.

If you are configuring fields for a media search app and want detailed information about the fields in the schema, see About media documents and data stores .

Update field settings

To update field settings:

1. In the Google Cloud console, go to the AI Applicationspage.

   AI Applications

2. Click the name of the app that you want to edit.

3. Click Data.

4. Click the Schematab. This tab shows current field settings.

   You won't see the Schematab if your data store contains basic website data or unstructured data without metadata .

5. Click Edit.

6. Select or clear field settings that you need to update. Some field settings are not supported. For example, numerical fields cannot be set to Searchable.

7. Click Saveto apply your changes.

Weight searchable fields (Preview)

If you mark a field searchable, you can specify a weight to indicate its relative importance in search results. Most situations don't require you to specify weights for individual fields because the default weights work well.

However, adjusting weights can be necessary in a few situations, for example:

• You're migrating data from an existing search platform that already uses weighted fields.

• When default weights aren't providing satisfactory search results. Specifically, this can happen when you have many searchable fields and some are markedly more important than others.

  Perhaps, the summary is the most important field for searches and so you want to prioritize that text.

  Or, the schema has a field containing highly relevant keywords that are excellent predictors for search results, but, because this field is much shorter than others, its influence is often overshadowed by longer fields. Increasing its weight ensures it has the intended impact.

Weight levels

Weights are banded into the following levels:

Schema update and reindexing

Adding weights to searchable fields requires a schema update and subsequent re-indexing of the data in the data store. Updating the schema takes hours, and there isn't a reliable indicator to tell you when indexing is completed, so you need to overestimate the indexing time.

Set weight levels on fields

The task of setting weight levels for fields can be tedious because you should make only small changes and carefully review search results afterward to check for unintended consequences. After each change, you must wait for re-indexing to complete before you can evaluate the impact of the change.

You can configure search field weighting only through the API. This feature is not available in the Google Cloud console.

To set weights, you need to update the schema for the data store through the API projects.locations.dataStores.schemas.patch method.

1. If you don't have your schema already, follow the instructions to get your schema in View a schema definition .

2. Follow the instructions to update the schema programmatically . Add weights to one or more searchable fields, as in these examples:

    "summary": {
      "type": "string",
      "searchable": true,
      "weight": "high"
    },
    "uri": {
      "type": "string",
      "searchable": true,
      "weight": "low"
    }, 
   

   In this example, the summary field is set to a higher weight than normal and the uri field to a lower weight. If you want to return a weight to the default value, set it to default .

   Allowed values for the weight parameter are:

   • very_low
   • low
   • default
   • high
   • very_high

   Example command and result

   curl -X PATCH \
   -H "Authorization: Bearer $(gcloud auth print-access-token)" \
   -H "Content-Type: application/json" \
   "https://discoveryengine.googleapis.com/v1beta/projects/my-project-123/locations/global/collections/default_collection/dataStores/my-data-store/schemas/default_schema" \
   -d '{
   "structSchema": { "$schema": "https://json-schema.org/draft/2020-12/schema", "properties": { "amenities": { "items": { "completable": true, "dynamicFacetable": true, "indexable": true, "retrievable": true, "searchable": true, "type": "string", "weight": "high" }, "type": "array" }, "available_date": { "indexable": true, "retrievable": true, "type": "datetime" }, "id": { "dynamicFacetable": false, "indexable": true, "retrievable": true, "type": "number" }, "location": { "indexable": true, "retrievable": true, "type": "geolocation" }, "non_smoking": { "dynamicFacetable": true, "indexable": true, "type": "boolean" }, "price_per_night": { "indexable": true, "retrievable": true, "type": "number" }, "rating": { "dynamicFacetable": true, "indexable": true, "retrievable": true, "type": "number" }, "room_types": { "items": { "dynamicFacetable": true, "indexable": true, "retrievable": true, "searchable": true, "type": "string", "weight": "low" }, "type": "array" }, "title": { "keyPropertyMapping": "title", "retrievable": true, "type": "string" } }, "type": "object" }
   }'
   {
   "name": "projects/123456/locations/global/collections/default_collection/dataStores/my-data-store/schemas/default_schema/operations/update-schema-123456",
   "metadata": {
   "@type": "type.googleapis.com/google.cloud.discoveryengine.v1beta.UpdateSchemaMetadata"
   }
   }

   In this example, the amentities field is given a higher than normal weighting and the room_types field is given a lower weighting.

3. Wait for reindexing to complete and test the search behavior.

Make fields available for prefix and partial matching (Preview)

For fields of type string , you can edit the schema to make the fields available for prefix matching or partial matching. This lets you use STARTS_WITH or CONTAINS in filter expressions.

Schema update and reindexing

Making fields available for prefix or partial matching requires a schema update and subsequent re-indexing of the data in the data store. Updating the schema takes hours, and there isn't a reliable indicator to tell you when indexing is completed, so you need to overestimate the indexing time.

Update schema for prefix and partial matching

To specify fields as available for prefix matching or partial matching, you need to update the schema for the data store through the API projects.locations.dataStores.schemas.patch method.

1. If you don't have your schema already, follow the instructions to get your schema in View a schema definition .

2. Follow the instructions to update the schema programmatically . Set the matchable parameters to true in the schema, as in these examples:

    "zone": {
      "type": "string",
      "searchable": true,
      "prefixMatchable": true
    },
    "region": {
      "type": "string",
      "searchable": true,
      "partialMatchable": true
    },
    "model": {
      "type": "string",
      "searchable": true,
      "prefixMatchable": true,
      "partialMatchable": true
    }, 
   

   In this example, the zone field is set to be prefix matchable. Up to 12 characters, case insensitive, can be used in the filter expression to match the field value. The region field is set to be partially matchable.

   Example command and result

   curl -X PATCH \
   -H "Authorization: Bearer $(gcloud auth print-access-token)" \
   -H "Content-Type: application/json" \
   "https://discoveryengine.googleapis.com/v1beta/projects/my-project-123/locations/global/collections/default_collection/dataStores/my-data-store/schemas/default_schema" \
   -d '{
     "structSchema": { "$schema": "https://json-schema.org/draft/2020-12/schema",
     "properties": { "amenities": { "items": { "completable":
     true,"partialMatchable": true, "dynamicFacetable": true, "indexable": true,
     "retrievable": true, "searchable": true, "type": "string" },
     "type": "array" }, "available_date": { "indexable": true, "retrievable": true,
     "type": "datetime" }, "id": { "dynamicFacetable": false, "indexable": true,
     "retrievable": true, "type": "number" }, "location": { "indexable": true,
     "retrievable": true, "type": "geolocation" }, "non_smoking": {
     "dynamicFacetable": true, "indexable": true, "type": "boolean" },
     "price_per_night": { "indexable": true, "retrievable": true, "type": "number"
     }, "rating": { "dynamicFacetable": true, "indexable": true, "retrievable":
     true, "type": "number" }, "room_types": { "items": { "dynamicFacetable": true,
     "prefixMatchable": true, "indexable": true, "retrievable": true,
     "searchable": true, "type": "string" }, "type": "array" }, "title": {
     "keyPropertyMapping": "title", "retrievable": true, "type": "string" } },
     "type": "object" } }'
   {
   "name": "projects/123456/locations/global/collections/default_collection/dataStores/my-data-store/schemas/default_schema/operations/update-schema-123456",
     "metadata": {
       "@type": "type.googleapis.com/google.cloud.discoveryengine.v1beta.UpdateSchemaMetadata"
     }
   }

   In this example, the amentities field is partially matchable. This means that CONTAINS("outdoor pool") matches an amenity with a value of outdoor swimming pool . The room_type field is prefix matchable, which means that a filter of STARTS_WITH("lux") matches with a room type of Luxury Room but not Deluxe ).

3. Wait for reindexing to complete.

Details about prefix matching

Prefix matching in Agent Search lets you filter results based on whether a field's value starts with a specific string. This capability is powered by the STARTS_WITH operator and requires that the target text field be configured as prefixMatchable in the schema.

Normalization and matching logic

To optimize performance and ensure consistency, the system applies a specific normalization process to both the field value(during indexing) and the query string(during search).

1. Lowercasing: All characters are converted to lowercase. This makes the matching case-insensitive.

2. 12-character truncation: The system only considers the first 12 charactersof the string. Any characters beyond this limit are ignored for prefix matching purposes.

How it works

At indexing time, for a field marked as prefixMatchable , the system generates prefix tokens for the first 12 characters. For a value like asia-south1-c , the index stores tokens for a , as , asi , asia , asia- , and so on, up to the 12th character ( asia-south1- ).

At query time, the string provided to the STARTS_WITH operator is also lowercased and truncated to 12 characters. The query is then matched against the stored prefix tokens.

Examples

The following examples demonstrate how the 12-character normalization affects search results:

• Broad matches: STARTS_WITH("A") matches any value starting with "a" (case-insensitive), such as asia , australia , or africa .

• Partial prefixes: STARTS_WITH("asia-south") matches both asia-south1-a and asia-southeast1-b because both start with the specified 10-character string.

• Truncation behavior: Because only the first 12 characters are matched, STARTS_WITH("asia-south1-a") matches a field value of asia-south1-c . This occurs because both strings are normalized to the same 12-character prefix: asia-south1- .

Details about partial matching

Partial matching in Agent Search lets you filter results based on whether a field's value contains specific words or tokens. This capability is powered by the CONTAINS operator and requires that the target text field be configured as partialMatchable in the schema.

Normalization and tokenization logic

The Agent Search normalizes and tokenizes both the field value(during indexing) and the query string(during search):

• Lowercasing: Agent Search converts the characters to lowercase. This makes the matching case-insensitive.

• Tokenization: Agent Search splits the string into individual tokens (words) using spaces and other characters as delimiters.

  • Standard delimiters: Spaces and common punctuation act as delimiters; these include hyphens ( - ), slashes ( / ), commas ( , ), periods ( . ), asterisks ( * ), curly brackets ( { } ), square brackets ( [ ] ), round brackets ( ( ) ), and apostrophes ( ' ).

  • Non-delimiters: Ampersand ( & ) and underscore ( _ ) don't act as delimiters. The system treats characters joined by these symbols as a single token.

  • Email delimiter ( @ ): The @ symbol acts as a special delimiter to assist in recognizing email addresses. The system generates tokens for the individual components as well as combined forms. For example, it tokenizes support+tier1@example.com to support , tier1 , example , com , support+tier1@example.com , and support@example.com .

How it works

At indexing time, for a field marked as partialMatchable , the system normalizes and tokenizes the field value and stores the resulting tokens in the index. For example, it tokenizes a field value of 25-meter, outdoor, swimming pool to [25, meter, outdoor, swimming, pool] .

At query time, the string provided to the CONTAINS operator undergoes the same normalization and tokenization process. The search engine then verifies that all resulting query tokens match the stored tokens for the field. The order of the tokens has no effect.

Examples

The following examples demonstrate how tokenization affects partial matching results:

• Basic token matching: If the field value is 25-meter, outdoor, swimming pool , a filter of CONTAINS("Outdoor pool") matches. The system tokenizes the query to [outdoor, pool] , both of which exist in the field's tokens.

• Order independence: A filter of CONTAINS("pool outdoor") also matches the field value 25-meter, outdoor, swimming pool because the system checks for the presence of each token independently of their order.

• Email address matching: If a field stores the email support+tier1@example.com , filters like CONTAINS("support") , CONTAINS("example.com") , or CONTAINS("support@example.com") all successfully match due to the special tokenization of the @ symbol.

What's next

• Update a schema for structured data
• Configure search results
• Preview search results