llama.cpp-prismml / tests /peg-parser /simple-tokenize.h
OpenTransformer's picture
Q1_0_g128 CPU kernel fix + AVX2 SIMD (fork of PrismML-Eng/llama.cpp)
03ba2cd verified
#pragma once
#include <string>
#include <vector>
std::vector<std::string> simple_tokenize(const std::string &);