diff options
| author | Evan Jones <evan.q.jones@gmail.com> | 2023-07-23 23:58:10 -0400 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-07-23 23:58:10 -0400 |
| commit | 84e09a7d8bc4ab6d658b5cd81295ac0add60be78 (patch) | |
| tree | 934c5480d917325ac8baa29f4edfae99137b56bb /grammars/json.gbnf | |
| parent | 2f9cf974a066ac0e03fbb235d834b01b0164d743 (diff) | |
llama : add grammar-based sampling (#1773)
* llama, main : constrain sampling to grammar
* allow loading grammar from file
* fix whitespace errors
* handle & print parser errors
* add comments to grammar syntax and allow newlines where unambiguous
* add missing include
* support alternates in root rule
* fix bugs with empty token and EOS
* adjust JSON grammar
* remove swp file
* rewrite ternary expressions
Co-authored-by: Henri Vasserman <henv@hot.ee>
* use struct for grammar elements and add Unicode support
* add unicode escapes
* add inverse char ranges
* only sample full tokens (no peeking or truncation)
* llama : minor style changes
blindly applied in online editor - hopefully I didn't break something
* update help text
* add warning message if EOS is disabled
---------
Co-authored-by: Henri Vasserman <henv@hot.ee>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Diffstat (limited to 'grammars/json.gbnf')
| -rw-r--r-- | grammars/json.gbnf | 29 |
1 files changed, 29 insertions, 0 deletions
diff --git a/grammars/json.gbnf b/grammars/json.gbnf new file mode 100644 index 0000000..40fa2b6 --- /dev/null +++ b/grammars/json.gbnf @@ -0,0 +1,29 @@ +# Grammar for subset of JSON - doesn't support full string or number syntax + +root ::= object +value ::= object | array | string | number | boolean | "null" + +object ::= + "{" ws ( + string ":" ws value + ("," ws string ":" ws value)* + )? "}" + +array ::= + "[" ws ( + value + ("," ws value)* + )? "]" + +string ::= + "\"" ( + [^"\\] | + "\\" (["\\/bfnrt] | "u" [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F]) # escapes + )* "\"" ws + +# Only plain integers currently +number ::= "-"? [0-9]+ ws +boolean ::= ("true" | "false") ws + +# Optional space: by convention, applied in this grammar after literal chars when allowed +ws ::= ([ \t\n] ws)? |