Hugging Face
Models
Datasets
Spaces
Buckets
new
Docs
Enterprise
Pricing
Log In
Sign Up
nmstech
/
turk-tokenizer
like
0
Turkish
turk-tokenizer
tokenizer
morphology
turkish
nlp
License:
mit
Model card
Files
Files and versions
xet
Community
main
turk-tokenizer
/
turk_tokenizer
31.7 MB
Ctrl+K
Ctrl+K
1 contributor
History:
1 commit
nmstech
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
ca41c16
verified
25 days ago
data
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
25 days ago
__init__.py
Safe
657 Bytes
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
25 days ago
_acronym_dict.py
Safe
4.31 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
25 days ago
_allomorph.py
Safe
2.23 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
25 days ago
_compound.py
Safe
2.95 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
25 days ago
_context_aware.py
Safe
2.09 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
25 days ago
_java_check.py
Safe
2.48 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
25 days ago
_medical_vocab.py
Safe
7.06 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
25 days ago
_normalizer.py
4.35 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
25 days ago
_preprocessor.py
5.54 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
25 days ago
_root_validator.py
Safe
6.66 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
25 days ago
_suffix_expander.py
Safe
8.39 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
25 days ago
_tdk_vocab.py
2.66 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
25 days ago
tokenizer.py
11.7 kB
Initial release: TurkTokenizer v1.0.0 — TR-MMLU 92%
25 days ago
