diff options
| author | Ronsor <ronsor@ronsor.pw> | 2023-03-15 12:37:50 -0700 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-03-15 21:37:50 +0200 |
| commit | 956dfda8ad8cea7961e22e0384bbc315bf79aed2 (patch) | |
| tree | 57210ba963ca22ecab007fe2841f02100ad423a8 /README.md | |
| parent | 113e685d18ac4edb20f647fd34b000941556f6a6 (diff) | |
Use `tokenizer.vocab_size()` instead of hardcoding 32000 in convert-pth-to-ggml.py (#142)
There are ways that special tokens or other new tokens could be added to the tokenizer; therefore it's probably best not to assume the vocabulary is only 32000 tokens.
Diffstat (limited to 'README.md')
0 files changed, 0 insertions, 0 deletions