mp.model_ckpt_util.ConvertHfTokenizer

ConvertHfTokenizer(arg0: str, arg1: str) -> None

Converts the HuggingeFace BPE tokenizer to internal SentencePiece vocab model.

Create a Mobile Website
View Site in Mobile | Classic
Share by: