diff options
Diffstat (limited to 'examples/server/README.md')
| -rw-r--r-- | examples/server/README.md | 5 |
1 files changed, 3 insertions, 2 deletions
diff --git a/examples/server/README.md b/examples/server/README.md index 089e890..bba513c 100644 --- a/examples/server/README.md +++ b/examples/server/README.md @@ -285,7 +285,8 @@ Test(); ## Common Options - `-m FNAME, --model FNAME`: Specify the path to the LLaMA model file (e.g., `models/7B/ggml-model.bin`). -- `-c N, --ctx_size N`: Set the size of the prompt context. The default is 512, but LLaMA models were built with a context of 2048, which will provide better results for longer input/inference. +- `-c N, --ctx-size N`: Set the size of the prompt context. The default is 512, but LLaMA models were built with a context of 2048, which will provide better results for longer input/inference. +- `-ngl N, --n-gpu-layers N`: When compiled with appropriate support (currently CLBlast or cuBLAS), this option allows offloading some layers to the GPU for computation. Generally results in increased performance. - `--embedding`: Enable the embedding mode. **Completion function doesn't work in this mode**. - `--host`: Set the hostname or ip address to listen. Default `127.0.0.1`; - `--port`: Set the port to listen. Default: `8080`. @@ -304,7 +305,7 @@ The RNG seed is used to initialize the random number generator that influences t ### Memory Float 32 -- `--memory_f32`: Use 32-bit floats instead of 16-bit floats for memory key+value, allowing higher quality inference at the cost of higher memory usage. +- `--memory-f32`: Use 32-bit floats instead of 16-bit floats for memory key+value. This doubles the context memory requirement but does not appear to increase generation quality in a measurable way. Not recommended. ## Limitations: |