Creating and using glossaries (Advanced)
A glossary is a custom dictionary the Cloud Translation API uses to consistently translate the customer's domain-specific terminology. This typically involves specifying how to translate a named entity.
You might use a glossary for the following use cases:
- Product names: For example, "Google Home" must translate to "Google Home".
- Ambiguous words: For example, the word "bat" can mean a piece of sports equipment or an animal. If you know that you are translating words about sports, you might want to use a glossary to feed the Cloud Translation API the sports translation of "bat", not the translation for the animal.
- Borrowed words: For example, "bouillabaisse" in French translates to "bouillabaisse" in English. English borrowed the word "bouillabaisse" from French in the 19th century. An English speaker lacking French cultural context might not know that bouillabaisse is a fish stew dish. Glossaries can override a translation so that "bouillabaisse" in French translates to "fish stew" in English.
Before you begin
Before you can start using the Cloud Translation API, you must have a project that has the Cloud Translation API enabled, and you must have the appropriate credentials. You can also install client libraries for common programming languages to help you make calls to the API. For more information, see the Setup page.
Required permissions
To work with glossaries, your service account requires glossary-specific
permissions. You can grant a role to your service account by using one of the pre-defined IAM roles
, such as
Cloud Translation API Editor ( roles/cloudtranslate.editor
), or you can create a custom
role
that grants the necessary
permissions. You can view all of the Cloud Translation API permissions in the IAM permissions reference
.
Cloud Translation permissions begin with cloudtranslate
.
For creating glossaries, you also need permissions to read objects in the
Cloud Storage bucket where your glossary file is located. You can grant
a role to your service account by using one of the pre-defined
IAM roles
, such as
Storage Object Viewer ( roles/storage.objectViewer
), or you can create a
custom role
that grants permissions to
read objects.
For information on adding an account to a role, see Granting, changing, and revoking access to resources .
Create a glossary
The terms in a glossary can be single tokens (words) or short phrases (usually fewer than five words). The main steps for using a glossary are:
- Create a glossary file
- Create the glossary resource with our Cloud Translation API
- Specify which glossary to use when you request a translation
A project can have multiple glossaries. You can get a list of the available glossaries and can delete glossaries that you no longer need.
Stopwords
Cloud Translation ignores some terms that are included in a glossary; these terms are known as stopwords . When translating stopwords, Cloud Translation ignores any matching glossary entries. For a list of all stopwords, see Glossary stopwords .
Creating a glossary file
Fundamentally, a glossary is a text file in which each line contains corresponding terms in multiple languages. The Cloud Translation API supports both unidirectional glossaries, which specify the desired translation for a single pair of source and target languages, and equivalent term sets , which identify the equivalent terms in multiple languages.
The total number of terms in a glossary input file can't exceed 10.4 million (10,485,760) UTF-8 bytes for all terms in all the languages combined. Any single glossary term must be less than 1024 UTF-8 bytes. Terms longer than 1024 bytes are ignored.
By default, glossary matches are case sensitive. When you apply a glossary, you can ignore case for all entries. If you have a mix of case-sensitive and case-insensitive terms, use the default behavior and, for case-insensitive terms, include both forms in your glossary.
Unidirectional glossaries
The Cloud Translation API accepts TSV, CSV or TMX files.
TSV and CSV
For tab-separated values (TSV) and comma separated values (CSV), each row
contains a pair of terms separated by a tab ( \t
) or comma ( ,
). The first
column includes the term in the source language and the second column includes
the term in the target language, as shown in the following example:
Translation Memory eXchange (TMX)
Translation Memory eXchange (TMX) is a standard XML format for providing source and target translations. The Cloud Translation API supports input files in a format based on TMX version 1.4 . This example illustrates the required structure:
<?xml
version='1.0'
encoding='utf-8'?>
<!DOCTYPE
tmx
SYSTEM
"tmx14.dtd">
<tmx
version="1.4">
<header
segtype="sentence"
o-tmf="UTF-8"
adminlang="en"
srclang="en"
datatype="PlainText"/>
<body>
<tu>
<tuv
xml:lang="en">
<seg>account</seg>
</tuv>
<tuv
xml:lang="es">
<seg>cuenta</seg>
</tuv>
</tu>
<tu>
<tuv
xml:lang="en">
<seg>directions</seg>
</tuv>
<tuv
xml:lang="es">
<seg>indicaciones</seg>
</tuv>
</tu>
</body>
</tmx>
The <header>
element of a well-formed TMX file must identify the source
language using the srclang
attribute, and every <tuv>
element must identify
the language of the contained text using the xml:lang
attribute. You identify
the source and target languages using their ISO-639
codes.
All <tu>
elements must contain a pair of <tuv>
elements with the same source
and target languages. If a <tu>
element contains more than two <tuv>
elements, the Cloud Translation API processes only the first <tuv>
matching the
source language and the first matching the target language and ignores the rest.
If a <tu>
element does not have a matching pair of <tuv>
elements, the
Cloud Translation API skips over the invalid <tu>
element.
The Cloud Translation API strips the markup tags from around a <seg>
element
before processing it. If a <tuv>
element contains more than one <seg>
element, the Cloud Translation API concatenates their text into a single element with a
space between them.
If the file contains XML tags other than those shown above, the Cloud Translation API ignores them.
If the file does not conform to proper XML and TMX format – for example,
if it is missing an end tag or a <tmx>
element – the Cloud Translation API
aborts processing it. The Cloud Translation API also aborts processing if it skips
more than 1024 invalid <tu>
elements.
Equivalent term sets (CSV)
For equivalent term sets, Cloud Translation API only accepts files in the CSV format. To define equivalent term sets, create a multi-column CSV file in which each row lists a single glossary term in multiple languages.
The first row in the file is a header row identifying the language for each column, using its ISO-639 or BCP-47 language code. You can also include optional columns for part of speech ( pos) and a description ( description). The algorithm does not currently use posinformation, and specific posvalues are not validated.
Each subsequent row contains equivalent glossary terms in the languages identified in the header. You can leave columns blank if the term is not available in all languages.
Create a glossary resource
After you have the equivalent glossary terms identified, make the glossary file available to the Cloud Translation API by creating a glossary resource.
Unidirectional glossary
When creating a unidirectional glossary, you must indicate the language pair
( language_pair
), by specifying the source language
( source_language_code
), and the target language
( target_language_code
). The following example uses the REST API and
command line, but you can also use the client
libraries
to create a
unidirectional glossary.
REST
When you create a new glossary, you supply a glossary id (a resource name). For example:projects/my-project/locations/us-central1/glossaries/my-en-to-ru-glossary
my-project
is the PROJECT_NUMBER_OR_ID, and my-en-ru-glossary
is the glossary-id provided by you.
Before using any of the request data, make the following replacements:
- PROJECT_NUMBER_OR_ID : the numeric or alphanumeric ID of your Google Cloud project
- glossary-id : your glossary ID, for example., my_en_ru_glossary
- bucket-name : name of bucket where your glossary file is located
- glossary-filename : filename of your glossary
HTTP method and URL:
POST https://translation.googleapis.com/v3/projects/ PROJECT_NUMBER_OR_ID /locations/us-central1/glossaries
Request JSON body:
{ "name":"projects/ PROJECT_NUMBER_OR_ID /locations/us-central1/glossaries/ glossary-id ", "languagePair": { "sourceLanguageCode": "en", "targetLanguageCode": "ru" }, "inputConfig": { "gcsSource": { "inputUri": "gs:// bucket-name / glossary-filename " } } }
To send your request, expand one of these options:
You should receive a JSON response similar to the following:
{ "name": "projects/ project-number /locations/us-central1/operations/ operation-id ", "metadata": { "@type": "type.googleapis.com/google.cloud.translation.v3beta1.CreateGlossaryMetadata", "name": "projects/ project-number /locations/us-central1/glossaries/ glossary-id ", "state": "RUNNING", "submitTime": "2019-11-19T19:05:10.650047636Z" } }
Equivalent term sets glossary
Once you have the glossary terms identified in your equivalent term set, make the glossary file available to the Cloud Translation API by creating a glossary resource.
REST
Before using any of the request data, make the following replacements:
- PROJECT_NUMBER_OR_ID : the numeric or alphanumeric ID of your Google Cloud project
- glossary-id : your glossary ID
- bucket-name : name of bucket where your glossary file is located
- glossary-filename : filename of your glossary
HTTP method and URL:
POST https://translation.googleapis.com/v3/projects/ PROJECT_NUMBER_OR_ID /locations/us-central1/glossaries
Request JSON body:
{ "name":"projects/ PROJECT_NUMBER_OR_ID /locations/us-central1/glossaries/ glossary-id ", "languageCodesSet": { "languageCodes": ["en", "en-GB", "ru", "fr", "pt-BR", "pt-PT", "es"] }, "inputConfig": { "gcsSource": { "inputUri": "gs:// bucket-name / glossary-file-name " } } }
To send your request, expand one of these options:
You should receive a JSON response similar to the following:
{ "name": "projects/ project-number /locations/us-central1/operations/20191103-09061569945989-5d937985-0000-21ac-816d-f4f5e80782d4", "metadata": { "@type": "type.googleapis.com/google.cloud.translation.v3beta1.CreateGlossaryMetadata", "name": "projects/ project-number /locations/us-central1/glossaries/ glossary-id ", "state": "RUNNING", "submitTime": "2019-11-03T16:06:29.134496675Z" } }
Go
Before trying this sample, follow the Go setup instructions in the Cloud Translation quickstart using client libraries . For more information, see the Cloud Translation Go API reference documentation .
To authenticate to Cloud Translation, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
Java
Before trying this sample, follow the Java setup instructions in the Cloud Translation quickstart using client libraries . For more information, see the Cloud Translation Java API reference documentation .
To authenticate to Cloud Translation, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
Node.js
Before trying this sample, follow the Node.js setup instructions in the Cloud Translation quickstart using client libraries . For more information, see the Cloud Translation Node.js API reference documentation .
To authenticate to Cloud Translation, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
Python
Before trying this sample, follow the Python setup instructions in the Cloud Translation quickstart using client libraries . For more information, see the Cloud Translation Python API reference documentation .
To authenticate to Cloud Translation, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
Additional languages
C#: Please follow the C# setup instructions on the client libraries page and then visit the Cloud Translation reference documentation for .NET.
PHP: Please follow the PHP setup instructions on the client libraries page and then visit the Cloud Translation reference documentation for PHP.
Ruby: Please follow the Ruby setup instructions on the client libraries page and then visit the Cloud Translation reference documentation for Ruby.
Operation status
Creating a glossary resource is a long-running operation, so it may take a substantial amount of time to complete. You can poll the status of this operation to see if it has completed, or you can cancel the operation.
For more information, see Long-running operations .
Use glossaries
Translate text with a glossary
In Cloud Translation - Advanced, you explicitly specify which translation model to use for translating the text. Translation LLM, NMT, and custom models can be called in combination with glossary, but only Translation LLM supports the latest quality enhancement with contextual glossary. To apply this feature, set BOOLEAN_CONTEXTUAL: to TRUE. You can also identify a glossary to use for domain-specific terminology.
REST
This example translates text using the default NMT model and a glossary.
Before using any of the request data, make the following replacements:
- PROJECT_NUMBER_OR_ID : the numeric or alphanumeric ID of your Google Cloud project.
- GLOSSARY_ID
: Your glossary ID, such as
my-en-ru-glossary. - BOOLEAN
: Whether matches are case insensitive. By default this
value is
false. - BOOLEAN_CONTEXTUAL
: Whether considers context in glossary translation. By default this
value is
false.
HTTP method and URL:
POST https://translation.googleapis.com/v3/projects/ PROJECT_NUMBER_OR_ID /locations/us-central1:translateText
Request JSON body:
{ "sourceLanguageCode": "en", "targetLanguageCode": "ru", "contents": "Dr. Watson, please discard your trash. You've shared unsolicited email with me. Let's talk about spam and importance ranking in a confidential mode.", "glossaryConfig": { "glossary": "projects/ project-number /locations/us-central1/glossaries/ GLOSSARY_ID ", "ignoreCase": BOOLEAN , "contextual_translation_enabled": BOOLEAN_CONTEXTUAL } }
To send your request, expand one of these options:
You should receive a JSON response similar to the following:
{ "glossaryTranslations": { "translatedText": "Доктор Ватсон, пожалуйста, откажитесь от своего мусора. Вы поделились нежелательной электронной почтой со я . Давайте поговорим о спаме и важности рейтинга в конфиденциальном режиме.", "glossaryConfig": { "glossary": "projects/ project-number /locations/us-central1/glossaries/ GLOSSARY_ID ", "ignoreCase": BOOLEAN , "contextual_translation_enabled": BOOLEAN_CONTEXTUAL } }, "translations": { "translatedText": "Доктор Ватсон, пожалуйста, откажитесь от своего мусора. Вы поделились нежелательной электронной почтой со мной. Давайте поговорим о спаме и важности рейтинга в конфиденциальном режиме.", } }
The translations
field contains the regular machine translation before the glossary is applied;
the glossaryTranslations
field contains the translation after the glossary is applied.
When you set the contextual_translation_enabled
field to true
, the response will only contain the glossaryTranslations
field and not the translations
field.
Go
Before trying this sample, follow the Go setup instructions in the Cloud Translation quickstart using client libraries . For more information, see the Cloud Translation Go API reference documentation .
To authenticate to Cloud Translation, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
Java
Before trying this sample, follow the Java setup instructions in the Cloud Translation quickstart using client libraries . For more information, see the Cloud Translation Java API reference documentation .
To authenticate to Cloud Translation, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
Node.js
Before trying this sample, follow the Node.js setup instructions in the Cloud Translation quickstart using client libraries . For more information, see the Cloud Translation Node.js API reference documentation .
To authenticate to Cloud Translation, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
Python
Before trying this sample, follow the Python setup instructions in the Cloud Translation quickstart using client libraries . For more information, see the Cloud Translation Python API reference documentation .
To authenticate to Cloud Translation, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
Additional languages
C#: Please follow the C# setup instructions on the client libraries page and then visit the Cloud Translation reference documentation for .NET.
PHP: Please follow the PHP setup instructions on the client libraries page and then visit the Cloud Translation reference documentation for PHP.
Ruby: Please follow the Ruby setup instructions on the client libraries page and then visit the Cloud Translation reference documentation for Ruby.
Adaptive translations with glossary
To use a glossary with an adaptive translation request, include a glossary configuration with your request.
REST
Before using any of the request data, make the following replacements:
- PROJECT_NUMBER_OR_ID : the numeric or alphanumeric ID of your Google Cloud project
- LOCATION
: The region where your source dataset is located, such
as
us-central1. - DATASET_NAME
: The name of the dataset that
Cloud Translation uses to customize your translations, formatted as
projects/ PROJECT_ID /locations/ LOCATION /adaptiveMtDatasets/ DATASET_ID. You can get dataset names by listing all datasets in your project. - SOURCE_TEXT : The text to translate.
- MIME_TYPE
(Optional): The format of the source text, such as
text/htmlortext/plain. By default, the MIME type is set totext/plain. - GLOSSARY_ID
: Your glossary ID, such as
my-en-ru-glossary. - BOOLEAN
(Optional): Whether matches are case insensitive. By default this
value is
false. - BOOLEAN_CONTEXTUAL
(Optional): Whether to consider context in glossary translation. By default this
value is
false.
HTTP method and URL:
POST https://translation.googleapis.com/v3/projects/ PROJECT_ID /locations/ LOCATION :adaptiveMtTranslate
Request JSON body:
{ "dataset": " DATASET_NAME ", "content": [" SOURCE_TEXT "], "mimeType": " MIME_TYPE ", "glossaryConfig": { "glossary": "projects/ project-number /locations/ LOCATION /glossaries/ GLOSSARY_ID ", "ignoreCase": BOOLEAN , "contextual_translation_enabled": BOOLEAN_CONTEXTUAL } }
To send your request, expand one of these options:
You should receive a JSON response similar to the following:
{ "translations": [ { "translatedText": " TRANSLATED_TEXT " } ], "languageCode": " TARGET_LANGUAGE ", "glossaryTranslations": { "translatedText": " TRANSLATED_TEXT ", "glossaryConfig": { "glossary": "projects/ project-number /locations/ LOCATION /glossaries/ GLOSSARY_ID ", "ignoreCase": BOOLEAN , "contextual_translation_enabled": BOOLEAN_CONTEXTUAL } }, }
Java
Before trying this sample, follow the Java setup instructions in the Cloud Translation quickstart using client libraries . For more information, see the Cloud Translation Java API reference documentation .
To authenticate to Cloud Translation, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
{ % dynamic setvar launch_stage % } preview { % dynamic endsetvar % } { % dynamic setvar launch_type % } feature { % dynamic endsetvar % } { % include "cloud/_shared/_info_launch_stage_disclaimer.html" % } /** Translates using AdaptiveMt. */ private static void adaptiveMtTranslate ( TranslationServiceClient translationServiceClient , String projectId , String datasetId ) { String adaptiveMtDatasetName = String . format ( "projects/%s/locations/LOCATION/adaptiveMtDatasets/%s" , projectId , datasetId ); String glossaryId = "your-glossary-display-name" ; GlossaryName glossaryName = GlossaryName . of ( projectId , location , glossaryId ); GlossaryConfig glossaryConfig = GlossaryConfig . newBuilder (). setGlossary ( glossaryName . toString ()). build (); AdaptiveMtTranslateRequest request = AdaptiveMtTranslateRequest . newBuilder () . setParent ( LocationName . of ( projectId , "us-central1" ). toString ()) . setDataset ( adaptiveMtDatasetName ) . addContent ( "Sample translation text" ) . setGlossaryConfig ( glossaryConfig ) . build (); AdaptiveMtTranslateResponse response = translationServiceClient . adaptiveMtTranslate ( request ); System . out . println ( "Translating using AdaptiveMt" ); System . out . println ( response ); }
