aboutsummaryrefslogtreecommitdiff
path: root/models
diff options
context:
space:
mode:
authorGeorgi Gerganov <ggerganov@gmail.com>2023-03-21 17:29:41 +0200
committerGitHub <noreply@github.com>2023-03-21 17:29:41 +0200
commiteb34620aeceaf9d9df7fcb19acc17ad41b9f60f8 (patch)
treeefa10a024845c09ed2630bbe0675618cc7ba0a50 /models
parent2e664f1ff413995506c9a54f3a8d5b8c64e37a91 (diff)
Add tokenizer test + revert to C++11 (#355)
* Add test-tokenizer-0 to do a few tokenizations - feel free to expand * Added option to convert-pth-to-ggml.py script to dump just the vocabulary * Added ./models/ggml-vocab.bin containing just LLaMA vocab data (used for tests) * Added utility to load vocabulary file from previous point (temporary implementation) * Avoid using std::string_view and drop back to C++11 (hope I didn't break something) * Rename gpt_vocab -> llama_vocab * All CMake binaries go into ./bin/ now
Diffstat (limited to 'models')
-rw-r--r--models/ggml-vocab.binbin0 -> 432578 bytes
1 files changed, 0 insertions, 0 deletions
diff --git a/models/ggml-vocab.bin b/models/ggml-vocab.bin
new file mode 100644
index 0000000..aba94bd
--- /dev/null
+++ b/models/ggml-vocab.bin
Binary files differ