Package tokenizer provides local token counting for Gemini models. This tokenizer downloads its model from the web, but otherwise doesn't require an API call for every [CountTokens] invocation.
CountTokensResponse
type
CountTokensResponse
struct
{
TotalTokens
int32
}
CountTokensResponse is the response of [Tokenizer.CountTokens].
Tokenizer
type
Tokenizer
struct
{
// contains filtered or unexported fields
}
Tokenizer is a local tokenizer for text.
func New
New creates a new [Tokenizer] from a model name; the model name is the same as you would pass to a [genai.Client.GenerativeModel].
func (*Tokenizer) CountTokens
func
(
tok
*
Tokenizer
)
CountTokens
(
parts
...
genai
.
Part
)
(
*
CountTokensResponse
,
error
)
CountTokens counts the tokens in all the given parts and returns their sum. Only [genai.Text] parts are suppored; an error will be returned if non-text parts are provided.
Example
package
main
import
(
"fmt"
"log"
"cloud.google.com/go/vertexai/genai"
"cloud.google.com/go/vertexai/genai/tokenizer"
)
func
main
()
{
tok
,
err
:=
tokenizer
.
New
(
"gemini-1.5-flash"
)
if
err
!=
nil
{
log
.
Fatal
(
err
)
}
ntoks
,
err
:=
tok
.
CountTokens
(
genai
.
Text
(
"a prompt"
),
genai
.
Text
(
"another prompt"
))
if
err
!=
nil
{
log
.
Fatal
(
err
)
}
fmt
.
Println
(
"total token count:"
,
ntoks
.
TotalTokens
)
}
