diff options
| author | ldwang <ftgreat@163.com> | 2023-07-25 21:22:09 +0800 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-07-25 16:22:09 +0300 |
| commit | fce48caf9a6b9930eee9e2a5971428cdff403ba8 (patch) | |
| tree | 23682a47a7223d0033fc99fb7e0018dc8d05dbf0 /examples/train-text-from-scratch | |
| parent | 875086bdb95ce1f3294439811536533199e3b579 (diff) | |
convert.py : support bpe tokenizer (#2228)
* support bpe tokenizer in convert
Signed-off-by: ldwang <ftgreat@gmail.com>
* support bpe tokenizer in convert
Signed-off-by: ldwang <ftgreat@gmail.com>
* support bpe tokenizer in convert, fix
Signed-off-by: ldwang <ftgreat@gmail.com>
---------
Signed-off-by: ldwang <ftgreat@gmail.com>
Co-authored-by: ldwang <ftgreat@gmail.com>
Diffstat (limited to 'examples/train-text-from-scratch')
0 files changed, 0 insertions, 0 deletions