aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2023-04-23A better `packNibbles` and `mul_sum_i8_pairs_float` implementation using ↵Yishuo Wang
AVX512 (#1119)
2023-04-22ggml : fix Q4_3 cuBLASGeorgi Gerganov
2023-04-22ci : trigger CI for drafts, but not most PR actions (#1125)Stephan Walter
2023-04-22Fix CI: ARM NEON, quantization unit tests, editorconfig (#1122)Stephan Walter
2023-04-22ggml : unit test for quantization functions (#953)unbounded
* Unit test for quantization functions Use the ggml_internal_get_quantize_fn function to loop through all quantization formats and run a sanity check on the result. Also add a microbenchmark that times these functions directly without running the rest of the GGML graph. * test-quantize-fns: CI fixes Fix issues uncovered in CI - need to use sizes divisible by 32*8 for loop unrolling - use intrinsic header that should work on Mac * test-quantize: remove Per PR comment, subsumed by test-quantize-fns * test-quantize: fix for q8_0 intermediates
2023-04-22llama : print timings on ctrl+c exit (#1021)wbpxre150
* print timings on ctrl+c exit * remove redundant free memory call. * add global pointer to ctx.
2023-04-22llama : have n_batch default to 512 (#1091)eiery
* set default n_batch to 512 when using BLAS * spacing * alternate implementation of setting different n_batch for BLAS * set n_batch to 512 for all cases
2023-04-22cmake : fix build under Windows when enable BUILD_SHARED_LIBS (#1100)Howard Su
* Fix build under Windows when enable BUILD_SHARED_LIBS * Make AVX512 test on Windows to build the shared libs
2023-04-22ggml : fix AVX build + update to new Q8_0 formatGeorgi Gerganov
2023-04-22ggml : alternative Q4_3 implementation using modified Q8_0 (#1109)Georgi Gerganov
* ggml : prefer vzip to vuzp This way we always use the same type of instruction across all quantizations * ggml : alternative Q4_3 implementation using modified Q8_0 * ggml : fix Q4_3 scalar imlpementation * ggml : slight improvement of Q4_3 - no need for loop unrolling * ggml : fix AVX paths for Q8_0 quantization
2023-04-22ggml : AVX2 optimization for vec_dot_q4_3_q8_0 and refactoring (#1099)Stephan Walter
* AVX2 optimization for vec_dot_q4_3_q8_0 and refactoring * finish AVX vectorization of quantize_row_q8_0 * Rename hsum_int_8 to hsum_i32_8
2023-04-22examples : Improve Alpaca Default Repeat Penalty: Better Match Alpaca.cpp ↵Clint Herron
Experience (#1107) * Moving parameters to separate lines for readability. * Increasing repeate_penalty to 1.1 to make alpaca more usable by default. * Adding trailing newline.
2023-04-22llama : add api for getting/setting the complete state: rng, logits, ↵xaedes
embedding and kv_cache (#1105) * reserve correct size for logits * add functions to get and set the whole llama state: including rng, logits, embedding and kv_cache * remove unused variables * remove trailing whitespace * fix comment
2023-04-21Improve cuBLAS performance by using a memory pool (#1094)slaren
* Improve cuBLAS performance by using a memory pool * Move cuda specific definitions to ggml-cuda.h/cu * Add CXX flags to nvcc * Change memory pool synchronization mechanism to a spin lock General code cleanup
2023-04-21llama : fixed rlimit error message (#888)apaz
2023-04-21cmake : link threads publicly to ggml (#1042)源文雨
* fix: ld link test-tokenizer-0 error ``` cmake3 --build . --config Release [ 5%] Built target ggml [ 16%] Built target llama [ 22%] Linking CXX executable ../bin/test-tokenizer-0 ../libllama.a(ggml.c.o):在函数‘ggml_graph_compute’中: ggml.c:(.text+0xf2db):对‘pthread_create’未定义的引用 ggml.c:(.text+0xf9d4):对‘pthread_join’未定义的引用 collect2: error: ld returned 1 exit status gmake[2]: *** [bin/test-tokenizer-0] 错误 1 gmake[1]: *** [tests/CMakeFiles/test-tokenizer-0.dir/all] 错误 2 gmake: *** [all] 错误 2 ``` * Update CMakeLists.txt * Update CMakeLists.txt * Update CMakeLists.txt
2023-04-21main : evaluate tokens in batches after swapping context (#1014)Alex Klinkhamer
* examples : evaluate tokens in batches after swapping context * Update examples/main/main.cpp --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-04-21llama : remember and restore kv cache data pointers (#1104)xaedes
because their value is stored in buf and overwritten by memcpy
2023-04-21ggml : a faster version for Q4_1 x Q8_0 dot products (#1083)Kawrakow
* A faster version for Q4_1 x Q8_0 dot products The idea nehind being that Q8_0 quantized values get used many times in the matrix multiplications where they are involved. In the current implementations, when we are evaluating the dot products, we need to compute the sum of the quants in the Q8_0 vector, so the same operation is repeated many times. Here we pre-compute the sum during Q8_0 quantization, store it in the now modified block_q8_0 struct, and then reuse this result in the subsequent dot products. In a synthetic benchmark (just compute a bunch of dot products), this change speeds up the Q4_1 * Q8_0 dot product by 80%, making the performance identical to Q4_0 * Q8_0. In practical application, I see a ~15% gain in speed for token prediction on M2, and ~5% gain on Ryzen 7950X. The speed gain in the prompt evaluation is much bigger (around 50%). I have only done the change for the scalar version, ARM_NEON, and AVX2, so we still need an AVX implementation. * Cleaning up --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
2023-04-21Show perplexity ETA in hours and minutes (#1096)slaren
2023-04-21llama : fix comment for "output.weight" tensorGeorgi Gerganov
2023-04-20Add ggml-model-*.bin checksums for 7B, 13B, 30B, 65B (#1088)Stephan Walter
* Add ggml-model-*.bin checksums for 7B, 13B, 30B * Add ggml-model-*.bin checksums for 65B --------- Co-authored-by: Pavol Rusnak <pavol@rusnak.io>
2023-04-20ggml : sync ggml (add GPT-NeoX RoPE implementation)Georgi Gerganov
2023-04-20ggml : fix bug in ggml_compute_forward_dup_f32()Georgi Gerganov
2023-04-20Add Q4_3 support to cuBLAS (#1086)slaren
2023-04-20ggml : do not break cuBLAS build (Q4_3 is not yet implemented)Georgi Gerganov
2023-04-20ggml : fix Q4_3 quantizationGeorgi Gerganov
Broke it during conflict resolution in last PR
2023-04-20llama : multi-threaded quantization (#1075)Kawrakow
* Multi-threading quantization. Not much gain for simple quantizations, bit it will be important for quantizations that require more CPU cycles. * Multi-threading for quantize-stats It now does the job in ~14 seconds on my Mac for Q4_0, Q4_1 and Q4_2. Single-threaded it was taking more than 2 minutes after adding the more elaborate version of Q4_2. * Reviewer comments * Avoiding compiler confusion After changing chunk_size to const int as suggested by @ggerganov, clang and GCC starting to warn me that I don't need to capture it in the lambda. So, I removed it from the capture list. But that makes the MSVC build fail. So, making it a constexpr to make every compiler happy. * Still fighting with lambda captures in MSVC --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-04-20ggml : add Q4_3 quantization (#1082)Georgi Gerganov
2023-04-20ci : remove the LLAMA_ACCELERATE matrix dimension from Ubuntu builds in the ↵Ivan Komarov
CI (#1074) [Accelerate](https://developer.apple.com/documentation/accelerate) is an Apple framework which can only be used on macOS, and the CMake build [ignores](https://github.com/ggerganov/llama.cpp/blob/master/CMakeLists.txt#L102) the `LLAMA_ACCELERATE` variable when run on non-Apple platforms. This implies setting `LLAMA_ACCELERATE` is a no-op on Ubuntu and can be removed. This will reduce visual noise in CI check results (in addition to reducing the number of checks we have to run for every PR). Right now every sanitized build is duplicated twice for no good reason (e.g., we have `CI / ubuntu-latest-cmake-sanitizer (ADDRESS, Debug, ON)` and `CI / ubuntu-latest-cmake-sanitizer (ADDRESS, Debug, OFF)`).
2023-04-20fix: LLAMA_CUBLAS=1 undefined reference 'shm_open' (#1080)源文雨
2023-04-20AVX2 optimization for vec_dot_q4_2_q8_0 (#1068)Stephan Walter
2023-04-20Improve cuBLAS performance by dequantizing on the GPU (#1065)slaren
2023-04-19Minor: Readme fixed grammar, spelling, and misc updates (#1071)CRD716
2023-04-19Q4_2 quantization with rmse-optimized scale and quants (#1062)Kawrakow
* Q4_2 quantization with rmse-optimized scale and quants For quantize-stats we get q4_2: rmse 0.00159301, maxerr 0.17480469, 95pct<0.0030, median<0.0012 For 7B perplexity with BLAS enabled we get 6.2038 after 655 chunks. Quantization is slow (~90 seconds on my Mac for 7B) as not multi-threaded as in PR #896. * ggml : satisfy the sanitizer builds Not sure why this makes them fail * Better follow ggml conventions for function names * Fixed type as per reviewer comment --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-04-19ggml : use 8-bit precision for Q4_1 intermediate results (#1047)Georgi Gerganov
* ggml : use 8-bit precision for Q4_1 intermediate results (ARM) * ggml : optimize ggml_vec_dot_q4_1_q8_0() via vmalq_n_f32 56 ms/token with Q4_1 ! * ggml : AVX2 implementation of ggml_vec_dot_q4_1_q8_0 (#1051) * gitignore : ignore ppl-*.txt files --------- Co-authored-by: slaren <2141330+slaren@users.noreply.github.com>
2023-04-19readme : add warning about Q4_2 and Q4_3Georgi Gerganov
2023-04-19ggml : Q4 cleanup - remove 4-bit dot product code (#1061)Stephan Walter
* Q4 cleanup * Remove unused AVX512 Q4_0 code
2023-04-19Add NVIDIA cuBLAS support (#1044)slaren
2023-04-19Multi-threaded ggml_cpy (#1035)slaren
* Multi-threaded ggml_cpy * Update ggml.c Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Also fix wdata offset in ggml_compute_forward_add_q_f32 --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-04-18ggml : add new Q4_2 quantization (ARM only) (#1046)Georgi Gerganov
* ggml : Q4_2 ARM * ggml : add ggml_is_quantized() * llama : update llama_type_name() with Q4_2 entry * ggml : speed-up q4_2 - 4 threads: ~100ms -> ~90ms - 8 threads: ~55ms -> ~50ms * ggml : optimize q4_2 using vmlaq_n_f32 + vmulq_n_f32
2023-04-18ggml : scratch that - vmlaq_n_f32 is always betterGeorgi Gerganov
Had a background process that was messing with the timings
2023-04-18gitignore : vdotGeorgi Gerganov
2023-04-18ggml : optimize ggml_vec_dot_q4_0_q8_0() using vectorized accumulatorsGeorgi Gerganov
2023-04-18Adding a simple program to measure speed of dot products (#1041)Kawrakow
On my Mac, the direct Q4_1 product is marginally slower (~69 vs ~55 us for Q4_0). The SIMD-ified ggml version is now almost 2X slower (~121 us). On a Ryzen 7950X CPU, the direct product for Q4_1 quantization is faster than the AVX2 implementation (~60 vs ~62 us). --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
2023-04-18readme : update hot topics about new LoRA functionalityGeorgi Gerganov
2023-04-18ci : do not run on draftsGeorgi Gerganov
2023-04-18Do not close file after mmap (Windows version) (#1034)Ivan Komarov
2023-04-17readme : add Ruby bindings (#1029)Atsushi Tatsuma
2023-04-17add 4_0 to default outfile namestr dict (#1031)Cameron
this came up when trying to convert the gpt4all-lora-unfiltered-quantized.bin file