guilhermelmello/tokenizer-bpe-pt-5k
Updated
Tokenizers Qwen-PT
Tokenizers trained with NFC normalization on Corpus Carolina (v2.0.1).
-
-
guilhermelmello/tokenizer-bpe-pt-8k
Updated -
guilhermelmello/tokenizer-bpe-pt-10k
Updated -
guilhermelmello/tokenizer-bpe-pt-15k
Updated -
guilhermelmello/tokenizer-bpe-pt-30k
Updated -
guilhermelmello/tokenizer-bpe-pt-50k
Updated -
guilhermelmello/tokenizer-unigram-pt-5k
Updated -
guilhermelmello/tokenizer-unigram-pt-8k
Updated -
guilhermelmello/tokenizer-unigram-pt-10k
Updated -
guilhermelmello/tokenizer-unigram-pt-15k
Updated -
guilhermelmello/tokenizer-unigram-pt-30k
Updated -
guilhermelmello/tokenizer-unigram-pt-50k
Updated