diff options
author | comex <comexk@gmail.com> | 2023-03-24 08:19:05 -0700 |
---|---|---|
committer | GitHub <noreply@github.com> | 2023-03-24 17:19:05 +0200 |
commit | 563cdc391dde140f1084d1012234e8e6f57f881f (patch) | |
tree | 64053534b3100c4399ec33cae763c882c1f33874 /llama.h | |
parent | 8d4a855c241ecb0f3ddc03447fe56002ebf27a37 (diff) |
Support calling mlock() on loaded model data on Linux and macOS (#453)
* Support calling mlock() on loaded model data on Linux and macOS
This is enabled by a new --mlock command line option.
Using mlock() disables swapping and memory compression for the model
data. Doing so can be useful on systems where the model takes up a
large fraction of system RAM. In my experience, macOS is quite eager to
start compressing llama.cpp's memory, which then makes it halt for a few
seconds while it decompresses, even with a model that uses "only" 25GB
out of 32GB.
Of course, this comes at the cost of forcing the system to swap or
compress other processes' memory instead, so it needs to be used with
care and shouldn't be enabled by default.
In theory it should be possible to support this on Windows as well using
VirtualLock(), but I'm not much of a Windows user.
* Update llama.cpp
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Diffstat (limited to 'llama.h')
-rw-r--r-- | llama.h | 1 |
1 files changed, 1 insertions, 0 deletions
@@ -53,6 +53,7 @@ extern "C" { bool f16_kv; // use fp16 for KV cache bool logits_all; // the llama_eval() call computes all logits, not just the last one bool vocab_only; // only load the vocabulary, no weights + bool use_mlock; // force system to keep model in RAM bool embedding; // embedding mode only }; |