Starting April 29, 2025, Gemini 1.5 Pro and Gemini 1.5 Flash models are not available in projects that have no prior usage of these models, including new projects. For details, seeModel versions and lifecycle.
Stay organized with collectionsSave and categorize content based on your preferences.
Package tokenizer provides local token counting for Gemini models. This
tokenizer downloads its model from the web, but otherwise doesn't require
an API call for every [CountTokens] invocation.
CountTokens counts the tokens in all the given parts and returns their
sum. Only [genai.Text] parts are suppored; an error will be returned if
non-text parts are provided.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-09-04 UTC."],[],[],null,["# Package cloud.google.com/go/vertexai/genai/tokenizer (v0.13.3)\n\n**Note:** To get more information about this package, such as access to older versions, view [this package on pkg.go.dev](https://pkg.go.dev/cloud.google.com/go/vertexai/genai/tokenizer). \n\u003cbr /\u003e\n\nPackage tokenizer provides local token counting for Gemini models. This\ntokenizer downloads its model from the web, but otherwise doesn't require\nan API call for every \\[CountTokens\\] invocation. \n\nCountTokensResponse\n-------------------\n\n type CountTokensResponse struct {\n \tTotalTokens https://pkg.go.dev/builtin#int32\n }\n\nCountTokensResponse is the response of \\[Tokenizer.CountTokens\\]. \n\nTokenizer\n---------\n\n type Tokenizer struct {\n \t// contains filtered or unexported fields\n }\n\nTokenizer is a local tokenizer for text. \n\n### func New\n\n func New(modelName https://pkg.go.dev/builtin#string) (*#cloud_google_com_go_vertexai_genai_tokenizer_Tokenizer, https://pkg.go.dev/builtin#error)\n\nNew creates a new \\[Tokenizer\\] from a model name; the model name is the same\nas you would pass to a \\[genai.Client.GenerativeModel\\]. \n\n### func (\\*Tokenizer) CountTokens\n\n func (tok *#cloud_google_com_go_vertexai_genai_tokenizer_Tokenizer) CountTokens(parts .../go/docs/reference/cloud.google.com/go/vertexai/latest/genai./go/docs/reference/cloud.google.com/go/vertexai/latest/genai#cloud_google_com_go_vertexai_genai_Part) (*#cloud_google_com_go_vertexai_genai_tokenizer_CountTokensResponse, https://pkg.go.dev/builtin#error)\n\nCountTokens counts the tokens in all the given parts and returns their\nsum. Only \\[genai.Text\\] parts are suppored; an error will be returned if\nnon-text parts are provided. \n\n#### Example\n\n package main\n\n import (\n \t\"fmt\"\n \t\"log\"\n\n \t\"cloud.google.com/go/vertexai/genai\"\n \t\"cloud.google.com/go/vertexai/genai/tokenizer\"\n )\n\n func main() {\n \ttok, err := tokenizer.New(\"gemini-1.5-flash\")\n \tif err != nil {\n \t\tlog.Fatal(err)\n \t}\n\n \tntoks, err := tok.CountTokens(genai.https://cloud.google.com/vertex-ai/generative-ai/docs/reference/go/latest/genai.html#cloud_google_com_go_vertexai_genai_Text(\"a prompt\"), genai.https://cloud.google.com/vertex-ai/generative-ai/docs/reference/go/latest/genai.html#cloud_google_com_go_vertexai_genai_Text(\"another prompt\"))\n \tif err != nil {\n \t\tlog.Fatal(err)\n \t}\n\n \tfmt.Println(\"total token count:\", ntoks.TotalTokens)\n\n }"]]
