| Age | Commit message (Collapse) | Author |
|---|
|
I was able to get llama-cpp-python working but only when I build libllama.so with make.
|
|
|
|
* Always sort logits before nucleus sampling
* remove second normalization
- fix windows build
- remove normalization since std::discrete_distribution does not require it
|
|
Otherwise observing this in the interactive mode:
/usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12/bits/stl_vector.h:1230: reference std::vector<int>::back() [_Tp = int, _Alloc = std::allocator<int>]: Assertion '!this->empty()' failed.
|
|
|
|
|
|
ggml :
- added ggml_view_3d()
- ggml_view_tensor() now inherits the stride too
- reimplement ggml_cpy() to account for dst stride
- no longer require tensor->data to be memory aligned
llama :
- compute RoPE on 32-bit tensors (should be more accurate)
- store RoPE-ed K in the KV cache
- store transposed V in the KV cache (significant speed-up)
- avoid unnecessary Q copy
|
|
|
|
* Define non-positive top_k; top_k range check
* minor : brackets
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
|
|
|
|
|
|
|
|
Co-authored-by: Locria Cyber <74560659+locriacyber@users.noreply.github.com>
|
|
|
|
* README: Update with CMake and windows example
* README: update with code-review for cmake build
|
|
* Add Miku.sh to examples
* Add missing line to prompt in Miku.sh
* Add --keep param to Miku.sh
* Remove '[end_of_conversation]' line from Miku.sh
No longer is necessary.
|
|
|
|
|
|
* Performance improvement of AVX2 code
* Fixed problem with MSVC compiler
* Reviewer comments: removed double semicolon, deleted empty line 1962
|
|
|
|
By using `pip install torch --index-url https://download.pytorch.org/whl/cpu`
instead of `pip install torch` we can specify we want to install a CPU-only version
of PyTorch without any GPU dependencies. This reduces the size of the Docker image
from 7.32 GB to 1.62 GB
|
|
`migrate-ggml-2023-03-30-pr613.py` is needed to get gpt4all running.
|
|
The api provides access methods for retrieving the current memory buffer for the kv_cache and its token number.
It also contains a method for setting the kv_cache from a memory buffer.
This makes it possible to load/save history - maybe support --cache-prompt paramater as well?
Co-authored-by: Pavol Rusnak <pavol@rusnak.io>
|
|
|
|
|
|
Fixes sanitizer CI
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
* ggml : add AVX quantize_row_q4_0()
* ggml : add AVX ggml_vec_dot_q4_0()
* ggml : refactor AVX part of ggml_vec_dot_q4_0()
https://github.com/ggerganov/llama.cpp/pull/617#issuecomment-1489985645
|
|
- use f-strings where possible
- drop first param of encode/decode functions since "utf-8" is the default
|
|
|
|
|
|
If you deleted your old Meta LLaMA .pth files, then the
migrate-ggml-2023-03-30-pr613.py script will allow you to convert your
old ggml files into the new mmap()'able format.
See #613
|
|
|
|
This is a breaking change that's going to give you three benefits:
1. Your inference commands should load 100x faster
2. You may be able to safely load models 2x larger
3. You can run many concurrent inference processes
This was accomplished by changing the file format so we can mmap()
weights directly into memory without having to read() or copy them
thereby ensuring the kernel can make its file cache pages directly
accessible to our inference processes; and secondly, that the file
cache pages are much less likely to get evicted (which would force
loads to hit disk) because they're no longer competing with memory
pages that were needlessly created by gigabytes of standard i/o.
The new file format supports single-file models like LLaMA 7b, and
it also supports multi-file models like LLaMA 13B. Our Python tool
now merges the foo.1, foo.2, etc. files back into a single file so
that the C++ code which maps it doesn't need to reshape data every
time. That's made llama.cpp so much simpler. Much of its load code
has now been deleted.
Furthermore, this change ensures that tensors are aligned properly
on a 32-byte boundary. That opens the door to seeing if we can get
additional performance gains on some microprocessors, by using ops
that require memory alignment.
Lastly note that both POSIX and the Windows platform are supported
Fixes #91
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
* It seems some new warning were added recently that exposed this. I wrote the code that included this unused variable originally and it is indeed not needed.
|
|
...there was no check. ported upstream from https://github.com/zanussbaum/gpt4all.cpp/pull/2 (I dont see any clean path for upstream patches)
|
|
|
