2024-02-23 04:54:08 -05:00

21 KiB


All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.



  • feat: Update llama.cpp to ggerganov/llama.cpp@15499eb942
  • feat: Add Google's Gemma formatting via chat_format="gemma" by @alvarobartt in #1210
  • feat: support minItems/maxItems in JSON grammar converter by @nopperl in 3921e10770
  • fix: Update from_pretrained defaults to match hf_hub_download and pull to local cache folder by @abetlen in e6d6260a91
  • fix: Raise exceptions when llama model or context fails to load by @abetlen in dd22010e85
  • docs: Update README.md to fix pip install llama cpp server by @audip in #1187



  • feat: Update llama.cpp to ggerganov/llama.cpp@ba2135ccae
  • feat: Pull models directly from huggingface by @abetlen in #1206
  • feat(low-level-api): Improve API static type-safety and performance. Low level api functions are positional args only now. by @abetlen in #1205



  • feat: Update llama.cpp to ggerganov/llama.cpp@4524290e87
  • fix: create_embedding broken response for input type str by @abetlen in 0ce66bc080
  • fix: Use '\n' seperator for EventSourceResponse by @khimaros in #1188
  • fix: Incorporate embedding pooling layer fixes by @iamlemec in #1194


  • feat: Update llama.cpp to ggerganov/llama.cpp@8084d55440
  • feat: Support batch embeddings by @iamlemec in #1186
  • fix: submodule kompute is not included in sdist by @abetlen in 7dbbfdecad
  • fix: fix: Update openbuddy prompt format by @abetlen in 07a783779a




  • feat: Update llama.cpp to ggerganov/llama.cpp@3bdc4cd0f5
  • feat: Generic chatml Function Calling using chat_format="chatml-function-calling"` by @abetlen in #957
  • fix: Circular dependancy preventing early Llama object free by @notwa in #1176
  • docs: Set the correct command for compiling with syscl support by @akarshanbiswas in #1172
  • feat: use gpu backend for clip if available by @iamlemec in #1175



  • feat: Update llama.cpp to ggerganov/llama.cpp@1cfb5372cf
  • feat: Add speculative decoding by @abetlen in #1120
  • fix: Pass raise_exception and add_generation_prompt to jinja2 chat template by @abetlen in 078cca0361






  • feat: Update llama.cpp to ggerganov/llama.cpp@faa3526a1e
  • feat(server): include llama-cpp-python version in openapi spec by @abetlen in cde7514c3d
  • fix: use both eos and bos tokens as stop sequences for hf-tokenizer-config chat format. by @abetlen in 5b982d0f8c
  • fix: GGUF metadata KV overrides, re #1011 by @phiharri in #1116
  • fix: llama_log_set should be able to accept null pointer by @abetlen in c970d41a85


  • feat: Update llama.cpp to ggerganov/llama.cpp@504dc37be8
  • fix: from_json_schema oneof/anyof bug by @jndiogo in d3f5528ca8
  • fix: pass chat handler not chat formatter for huggingface autotokenizer and tokenizer_config formats by @abetlen in 24f39454e9
  • feat: Add add_generation_prompt option for jinja2chatformatter by @abetlen in 7f3209b1eb
  • feat: Add Jinja2ChatFormatter by @abetlen in be09318c26
  • feat: Expose gguf model metadata in metadata property by @abetlen in 5a34c57e54



  • feat: Update llama.cpp to ggerganov/llama.cpp@57e2a7a52a
  • feat(server): Add ability to load chat format from huggingface autotokenizer or tokenizer_config.json files by @abetlen in b8fc1c7d83
  • feat: Integration of Jinja2 Templating for chat formats by @teleprint-me in #875
  • fix: Offload KQV by default by @abetlen in 48c3b77e6f
  • fix: Support Accept text/event-stream in chat and completion endpoints, resolves #1083 by @aniljava in #1088
  • fix(cli): allow passing n_ctx=0 to openAI API server args to use model n_ctx_train field per #1015 by @K-Mistele in #1093


  • feat: Update llama.cpp to ggerganov/llama.cpp@4483396751
  • feat: Add split_mode option by @abetlen in 84615adbc6
  • feat: Implement GGUF metadata KV overrides by @phiharri in #1011
  • fix: Avoid "LookupError: unknown encoding: ascii" when open() called in a destructor by @yieldthought in #1012
  • fix: Fix low_level_api_chat_cpp example to match current API by @aniljava in #1086
  • fix: Fix Pydantic model parsing by @DeNeutoy in #1087


  • feat: Update llama.cpp to ggerganov/llama.cpp@6efb8eb30e
  • feat: Add ability to pass in penalize_nl param by @shankinson in #1068
  • fix: print_grammar to stderr by @turian in #1052


  • feat: Update llama.cpp to ggerganov/llama.cpp@b3a7c20b5c
  • feat: Add saiga chat format by @femoiseev in #1050
  • feat: Added chatglm3 chat format by @xaviviro in #1059
  • fix: Correct typo in README.md by @qeleb in (#1058)



  • feat(server): Multi model support by @D4ve-R in #931
  • feat(server): Support none defaulting to infinity for completions by @swg in #111
  • feat(server): Implement openai api compatible authentication by @docmeth2 in #1010
  • fix: text_offset of multi-token characters by @twaka in #1037
  • fix: ctypes bindings for kv override by @phiharri in #1011
  • fix: ctypes definitions of llama_kv_cache_view_update and llama_kv_cache_view_free. by @e-c-d in #1028


  • feat: Update llama.cpp to ggerganov/llama.cpp@0e18b2e7d0
  • feat: Add offload_kqv option to llama and server by @abetlen in 095c650006
  • feat: n_ctx=0 now uses the n_ctx_train of the model by @DanieleMorotti in #1015
  • feat: logits_to_logprobs supports both 2-D and 3-D logits arrays by @kddubey in #1002
  • fix: Remove f16_kv, add offload_kqv fields in low level and llama apis by @brandonrobertz in #1019
  • perf: Don't convert logprobs arrays to lists by @kddubey in #1021
  • docs: Fix README.md functionary demo typo by @evelynmitchell in #996
  • examples: Update low_level_api_llama_cpp.py to match current API by @jsoma in #1023


  • Update llama.cpp to ggerganov/llama.cpp@948ff137ec
  • Add qwen chat format by @yhfgyyf in #1005
  • Add support for running the server with SSL by @rgerganov in #994
  • Replace logits_to_logprobs implementation with numpy equivalent to llama.cpp by @player1537 in #991
  • Fix UnsupportedOperation: fileno in suppress_stdout_stderr by @zocainViken in #961
  • Add Pygmalion chat format by @chiensen in #986
  • README.md multimodal params fix by @zocainViken in #967
  • Fix minor typo in README by @aniketmaurya in #958



  • Update llama.cpp to ggerganov/llama.cpp@64e64aa255
  • Make building llava optional by setting CMAKE_ARGS="-DLLAVA_BUILD=OFF" and using LLAVA_CPP_LIB to specify alternative path to shared library by @abetlen in e3941d9c67


  • Update llama.cpp to ggerganov/llama.cpp@b38a16dfcf
  • Add zephyr chat format by @fakerybakery in #938
  • Add baichuan chat format by @caiyesd in #938
  • Add baichuan-2 chat format by @caiyesd in #936
  • Improve documentation for server chat formats by @jooray in #934
  • Fix typo in README by @antonvice in 940
  • Fix typo in the Open Orca chat format by @gardner in #947


  • Update llama.cpp to ggerganov/llama.cpp@0b871f1a04
  • Fix #569: stop parameter in chat completion api should accept str by @abetlen in 128dc4731f
  • Document server host and port parameters by @jamesbraza in #768
  • Do not set grammar to None when initializing LlamaGrammar by @mthuurne in #834
  • Add mistrallite, intel, and openchat formats by @fakerybakery in #927
  • Add support for min_p parameter by @tk-master in #921
  • Fix #929: tokenizer adding leading space when generating from empty prompt by @abetlen in a34d480141
  • Fix low level api example by @zocainViken in #925
  • Fix missing package in openblas docker image by @ZisisTsatsas in #920




  • Update llama.cpp to ggerganov/llama.cp@a75fa576ab
  • Add set_seed to Llama class by @abetlen in fd41ed3a90
  • Fix server doc arguments by @kjunggithub in #892
  • Fix response_format handler in llava chat handler by @abetlen in b62c449839
  • Fix default max_tokens, chat completion is now unlimited (to context length) and completion is 16 tokens to match OpenAI defaults by @abetlen in e7962d2c73
  • Fix json_schema_to_gbnf helper so that it takes a json schema string as input instead by @abetlen in faeae181b1
  • Add support for $ref and $def in json_schema_to_gbnf to handle more complex function schemas by @abetlen in 770df34436
  • Update functionary chat handler for new OpenAI api by abetlen in 1b376c62b7
  • Fix add default stop sequence to chatml chat format by @abetlen in b84d76a844
  • Fix sampling bug when logits_all=False by @abetlen in 6f0b0b1b84


  • Update llama.cpp to ggerganov/llama.cpp@0a7c980b6f
  • Add support for Llava1.5 multimodal models by @damian0815 and @abetlen in #821
  • Update OpenAI API compatibility to match dev day update by @abetlen in #821
  • Add seed parameter to completion and chat_completion functions of Llama class by @abetlen in 86aeb9f3a1
  • Add JSON mode support to constrain chat completion to JSON objects by @abetlen in b30b9c338b


  • Update llama.cpp to ggerganov/llama.cpp@f0b30ef7dc
  • Add support for Huggingface Autotokenizer Chat Formats by @bioshazard and @abetlen in #790 and bbffdaebaa
  • Fix llama-2 chat format by @earonesty in #869
  • Add support for functionary chat format by @abetlen in #784
  • Migrate inference from deprecated llama_evalAPI to llama_batch and llama_decode by @abetlen in #795


  • Update llama.cpp to ggerganov/llama.cpp@51b2fc11f7
  • Fix name 'open' is not defined exception when deleting model by @abetlen in 011b95d7f3
  • Fix tokenization of special characters by @antoine-lizee in #850


  • Update llama.cpp to ggerganov/llama.cpp@50337961a6
  • Fix missing n_seq_id in llama_batch by @NickAlgra in #842
  • Fix for shared libraries on Windows that start with lib prefix by @sujeendran in #848
  • Fix exception raised in __del__ when freeing models by @cebtenzzre in #846
  • Performance improvement for logit bias by @zolastro in #851
  • Fix suffix check arbitrary code execution bug by @mtasic85 in #854
  • Fix typo in function_call parameter in llama_types.py by @akatora28 in #849
  • Fix streaming not returning finish_reason by @gmcgoldr in #798
  • Fix n_gpu_layers check to allow values less than 1 for server by @hxy9243 in #826
  • Supppress stdout and stderr when freeing model by @paschembri in #803
  • Fix llama2 chat format by @delock in #808
  • Add validation for tensor_split size by @eric1932 #820
  • Print stack trace on server error by @abetlen in d6a130a052
  • Update docs for gguf by @johnccshen in #783
  • Add chatml chat format by @abetlen in 305482bd41


  • Fix bug in llama_model_params object has no attribute logits_all by @abetlen in d696251fbe


  • Fix bug 'llama_model_params' object has no attribute 'embedding' by @abetlen in 42bb721d64


  • Fix critical bug in pip installation of v0.2.8 due to .git directory in ac853e01e1


  • Update llama.cpp to ggerganov/llama.cpp@40e07a60f9
  • Add configurable chat formats by @abetlen in #711
  • Fix rope scaling bug by @Josh-XT in #767
  • Fix missing numa parameter in server by @abetlen in d9bce17794


  • Update llama.cpp to ggerganov/llama.cpp@a98b1633d5
  • Install required runtime dlls to package directory on windows by @abetlen in 8d75016549
  • Add openai-processing-ms to server response header by @Tradunsky in #748
  • Bump minimum version of scikit-build-core to 0.5.1 to fix msvc cmake issue by @abetlen in 1ed0f3ebe1
  • Update llama_types.py to better match the openai api, old names are aliased to new ones by @abetlen in dbca136fea


  • Update llama.cpp to 80291a1d02a07f7f66666fb576c5b1e75aa48b46


  • Fix docker images missing starlette-context dependency by @abetlen in 2291798900
  • Fix loading dll in Windows Isolation Containers by @abetlen in 8474665625
  • Fix build issue on m1 macs by @abetlen in dbd3a6d1ed
  • Update docs to gguf and add hw acceleration docs for server by @jasonacox in #688


  • Add NUMA support. NOTE low level api users must call llama_backend_init at the start of their programs by abetlen in f4090a0bb2
  • Fix tensor_split server cli argument by @abetlen in c4c440ba2d
  • Made all Llama init parameters into keyword-only parameters by @abetlen in c8f9b8a734
  • Added server params for low_vram, main_gpu, lora_base, and lora_path by @abetlen in 2920c4bf7e
  • Removed server params for rms_norm_eps and n_gqa by @abetlen in 2920c4bf7e
  • Fix boolean cli options by @abetlen in c999325e8e and 0449d29b9f
  • Silence Pydantic Settings warnings about model_alias setting by @earonesty in #705


  • Update llama.cpp to ggerganov/llama.cpp@71ca2fad7d
  • Add X-Request-ID request header for mirroring custom IDs by @devrimcavusoglu in #703
  • Add pyproject extra for scikit-build-core to ensure compatible pathspec version by @abetlen in 6cfc54284b
  • Fix issue with Literal and Optional cli arguments not working by @abetlen in #702


  • Fix bug in pip install of v0.2.1 due to scikit-build-core removing all .metal files in the source distribution (see #701)


  • Fix bug in pip install of v0.2.0 due to .git folder being included in the source distribution (see #701)


  • Migrated to scikit-build-core build system by @abetlen in #499
  • Use numpy views for LogitsProcessor and StoppingCriteria instead of python lists by @abetlen in #499
  • Drop support for end-of-life Python3.7 by @abetlen in #499
  • Convert low level llama.cpp constants to use basic python types instead of ctypes types by @abetlen in #499


  • Add llama_cpp.__version__ attribute by @janvdp in #684
  • Fix low level api examples by @jbochi in #680


  • Update llama.cpp


  • Update llama.cpp


  • Update llama.cpp


  • Update llama.cpp


  • Update llama.cpp


  • GGUF Support (breaking change requiring new model format)


  • Grammar based sampling via LlamaGrammar which can be passed to completions
  • Make n_gpu_layers == -1 offload all layers


  • (llama.cpp) Update llama.cpp add support for LLaMa 2 70B
  • (server) Add temporary n_gqa and rms_norm_eps parameters required for LLaMa 2 70B


  • (llama.cpp) Update llama.cpp add support for LLaMa 2 70B


  • Update llama.cpp


  • (server) OpenAI style error responses


  • (server) Add rope parameters to server settings


  • (llama.cpp) Update llama.cpp added custom_rope for extended context lengths


  • (llama.cpp) Update llama.cpp

  • (server) Fix several pydantic v2 migration bugs


  • (Llama.create_completion) Revert change so that max_tokens is not truncated to context_size in create_completion
  • (server) Fixed changed settings field names from pydantic v2 migration


  • (server) Streaming requests can are now interrupted pre-maturely when a concurrent request is made. Can be controlled with the interrupt_requests setting.
  • (server) Moved to fastapi v0.100.0 and pydantic v2
  • (docker) Added a new "simple" image that builds llama.cpp from source when started.
  • (server) performance improvements by avoiding unnecessary memory allocations during sampling


  • (llama.cpp) Update llama.cpp


  • Fix performance bug in Llama model by pre-allocating memory tokens and logits.
  • Fix bug in Llama model where the model was not free'd after use.


  • (llama.cpp) New model API

  • Performance issue during eval caused by looped np.concatenate call

  • State pickling issue when saving cache to disk


  • (llama.cpp) Fix struct misalignment bug


  • (llama.cpp) Update llama.cpp
  • Fix docs for seed. Set -1 for random.


  • (llama.cpp) Add full gpu utilisation in CUDA
  • (llama.cpp) Add get_vocab
  • (llama.cpp) Add low_vram parameter
  • (server) Add logit_bias parameter


  • Metal support working
  • Cache re-enabled


  • Fix broken pip installation


NOTE: This release was deleted due to a bug with the packaging system that caused pip installations to fail.

  • Truncate max_tokens in create_completion so requested tokens doesn't exceed context size.
  • Temporarily disable cache for completion requests


  • (llama.cpp) k-quants support
  • (server) mirostat sampling parameters to server
  • Support both .so and .dylib for libllama on MacOS


  • (llama.cpp) Metal Silicon support


  • (llama.cpp) OpenLlama 3B support


  • (misc) Added first version of the changelog
  • (server) Use async routes
  • (python-api) Use numpy for internal buffers to reduce memory usage and improve performance.
  • (python-api) Performance bug in stop sequence check slowing down streaming.