aboutsummaryrefslogtreecommitdiff
path: root/examples/quantize
diff options
context:
space:
mode:
authorKawrakow <48489457+ikawrakow@users.noreply.github.com>2023-06-05 22:56:18 +0300
committerGitHub <noreply@github.com>2023-06-05 22:56:18 +0300
commit99009e72f8072fa552eb02efee436be596c71cdd (patch)
treed9a7e29f42c45eaaae69c423735a1ce69db0dab7 /examples/quantize
parent5220a991a5e92bddad9542267ab445a2c033681c (diff)
ggml : add SOTA 2,3,4,5,6 bit k-quantizations (#1684)
* Starting to add k-quantization to ggml I think it is better to have quantization separate from ggml. For now just adding the k-quants there, but it would be better to also factor out the existing ggml quantizations. * Adding Q3_K and Q8_K (de)-quantization * Q3_K now working on CUDA and AVX2/scalar CUDA is not ideal - ~50% slower than Q4_0 for single token prediction, about the same in batch mode (perplexity). CPU single token is ~55 ms (on Ryzen 7950X). * Some improvement for Q3_K on CUDA It is now ~22.5 ms/token on my GPU, so ~30% slower than Q4_0. * Some more CUDA optimizations for Q3_K Single token is now 20.5 ms/token (~20% slower than Q4_0). Perplexity is on par with Q4_0. * Adding Q4_K - scalar, AVX2, CUDA Performance is the same or perhaps very slightly better than Q4_0 on the CPU. On the GPU, single token prediction is ~10% better than Q4_0, batch mode (perplexity is about the same). * Adding Q6_K - scalar, AVX2, CUDA Performance is ~40% lower compared to Q4_K on the CPU. This is to be expected, considering that we are memory bound on the CPU and the 6-bit model is ~44% larger than the 4-bit. On the GPU, single token prediction is ~6% lower than Q4_0, batch mode (perplexity) is even closer (but still slower). * Adding Q5_K - scalar, AVX2, CUDA Performance is ~20% lower compared to Q4_K on the CPU. This is to be expected, considering that we are memory bound on the CPU and the 5-bit model is ~22% larger than the 4-bit. On the GPU, single token prediction is about the same as Q4_0 for both, single token and batch prediction. * Per convention, all QX_K quantizations use Q5_K for output.weight * Adding quantization mixes * Quantization mixes: didn't quite get what I wanted in the last commit * Q4_K dot product for ARM_NEON * Q6_K dot product for ARM_NEON * Q5_K dot product for ARM_NEON * Adding Q3_K dot for ARM_NEON It is 22% slower than Q4_K, despite the smaller model size. On x86_64, where we are memory bound, the Q3_K model is quite a bit faster than Q4_K. * A very slightly faster ARM_NEON Q3_K dot * Adding Q2_K - just CUDA for now Token prediction is pretty good - about 15.5 ms on a RTX 4080. Perplexity is about the same as Q4_K. * Adding scalar and AVX2 Q2_K dot * Adding ARM_NEON Q2_K dot About the same performance as Q4_K. * A slightly faster ARM_NEON Q2_K dot Single token prediction is now ~36 ms on M2 Max. The code is much simpler too. * Fixed bug in Q2_K CUDA dot product kernel Stranegly enough, for the few prompts I tried with the 7B model the responses looked perfectly reasonable. Only realized something is not quite right when I tried the larger models and started getting nonse back. In any case, Q2_K single token evaluation time on an RTX 4080 in a Ryzen7950X box iusing CUDA and model fully loaded on the GPU are ~15.5 ms for 7B, ~25.4 ms for 13B, and ~55.8 ms for 30B. The max number of layers that fit in VRAM for The 65B is 32. With that, we get ~330 ms per token, which is not that much faster than just running on the CPU (~470 ms per token). * Don't print zeros/NaNs when no count histogram has been collected * A 10% faster CUDA vector dot kernel for Q3_K Q3_K is now running at ~18.5 ms / token on CUDA, so the gap to Q4_0 is only 10%. It seems memory acccess pattern is more important for performance than the amount of computation the kernel does. * A slightly daster Q4_K AVX2 dot product For perplexity, where we are less memory bound, time per pass drops by ~5%. Barely measurable difference for single token prediction. * A slightly faster ARM_NEON A4_K dot product * Minor * Fix quantization error test We cannot possibly be expecting rmse < 0.002 for 2- and 3-bit quantization variants. * Fix docker build I have been sloppy with vector reinterpret casts on ARM_NEON. It seems clang is very forgiving in that regard. * Added forgotten ggml.o dependence on k_quants.h to the Makefile * Had unintentionally committed the Makefile with -Ofast enabled * ggml : rename k_quants -> ggml-quants-k, use lowercase in code --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Diffstat (limited to 'examples/quantize')
-rw-r--r--examples/quantize/quantize.cpp22
1 files changed, 17 insertions, 5 deletions
diff --git a/examples/quantize/quantize.cpp b/examples/quantize/quantize.cpp
index 769dd36..947b402 100644
--- a/examples/quantize/quantize.cpp
+++ b/examples/quantize/quantize.cpp
@@ -7,11 +7,23 @@
#include <string>
static const std::map<std::string, llama_ftype> LLAMA_FTYPE_MAP = {
- {"q4_0", LLAMA_FTYPE_MOSTLY_Q4_0},
- {"q4_1", LLAMA_FTYPE_MOSTLY_Q4_1},
- {"q5_0", LLAMA_FTYPE_MOSTLY_Q5_0},
- {"q5_1", LLAMA_FTYPE_MOSTLY_Q5_1},
- {"q8_0", LLAMA_FTYPE_MOSTLY_Q8_0},
+ {"q4_0", LLAMA_FTYPE_MOSTLY_Q4_0},
+ {"q4_1", LLAMA_FTYPE_MOSTLY_Q4_1},
+ {"q5_0", LLAMA_FTYPE_MOSTLY_Q5_0},
+ {"q5_1", LLAMA_FTYPE_MOSTLY_Q5_1},
+ {"q8_0", LLAMA_FTYPE_MOSTLY_Q8_0},
+ {"q2_K", LLAMA_FTYPE_MOSTLY_Q2_K},
+ {"q3_K", LLAMA_FTYPE_MOSTLY_Q3_K_M},
+ {"q3_K_S", LLAMA_FTYPE_MOSTLY_Q3_K_S},
+ {"q3_K_M", LLAMA_FTYPE_MOSTLY_Q3_K_M},
+ {"q3_K_L", LLAMA_FTYPE_MOSTLY_Q3_K_L},
+ {"q4_K", LLAMA_FTYPE_MOSTLY_Q4_K_M},
+ {"q4_K_S", LLAMA_FTYPE_MOSTLY_Q4_K_S},
+ {"q4_K_M", LLAMA_FTYPE_MOSTLY_Q4_K_M},
+ {"q5_K", LLAMA_FTYPE_MOSTLY_Q5_K_M},
+ {"q5_K_S", LLAMA_FTYPE_MOSTLY_Q5_K_S},
+ {"q5_K_M", LLAMA_FTYPE_MOSTLY_Q5_K_M},
+ {"q6_K", LLAMA_FTYPE_MOSTLY_Q6_K},
};
bool try_parse_ftype(const std::string & ftype_str, llama_ftype & ftype, std::string & ftype_str_out) {