mp.model_ckpt_util.ConvertHfTokenizer

ConvertHfTokenizer(arg0: str, arg1: str) -> None

Converts the HuggingeFace BPE tokenizer to internal SentencePiece vocab model.

Design a Mobile Site
View Site in Mobile | Classic
Share by: