ConvertHfTokenizer(arg0: str, arg1: str) -> None
mp
.
model_ckpt_util
.
ConvertHfTokenizer
()
Converts the HuggingeFace BPE tokenizer to internal SentencePiece vocab model.
ConvertHfTokenizer(arg0: str, arg1: str) -> None
mp
.
model_ckpt_util
.
ConvertHfTokenizer
()
Converts the HuggingeFace BPE tokenizer to internal SentencePiece vocab model.
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License , and code samples are licensed under the Apache 2.0 License . For details, see the Google Developers Site Policies . Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2026-05-28 UTC.