aboutsummaryrefslogtreecommitdiff
path: root/Makefile
diff options
context:
space:
mode:
authorRonsor <ronsor@ronsor.pw>2023-03-15 12:37:50 -0700
committerGitHub <noreply@github.com>2023-03-15 21:37:50 +0200
commit956dfda8ad8cea7961e22e0384bbc315bf79aed2 (patch)
tree57210ba963ca22ecab007fe2841f02100ad423a8 /Makefile
parent113e685d18ac4edb20f647fd34b000941556f6a6 (diff)
Use `tokenizer.vocab_size()` instead of hardcoding 32000 in convert-pth-to-ggml.py (#142)
There are ways that special tokens or other new tokens could be added to the tokenizer; therefore it's probably best not to assume the vocabulary is only 32000 tokens.
Diffstat (limited to 'Makefile')
0 files changed, 0 insertions, 0 deletions