Convert Figma logo to code with AI

ggml-org logollama.cpp

LLM inference in C/C++

84,151
12,595
84,151
794
View on GitHub

Top Related Projects

meta-llama's avatar

llama

58,578

Inference code for Llama models

karpathy's avatar

llama2.c

18,609

Inference Llama 2 in one file of pure C

abetlen's avatar

llama-cpp-python

9,403

Python bindings for llama.cpp

ggml-org's avatar

whisper.cpp

41,097

Port of OpenAI's Whisper model in C/C++

openai's avatar

whisper

85,961

Robust Speech Recognition via Large-Scale Weak Supervision

huggingface's avatar

transformers

146,142

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

Quick Overview

llama.cpp is a port of Facebook's LLaMA model to C/C++. It allows for efficient inference of LLaMA models on CPU, with optional GPU acceleration. The project aims to make large language models more accessible by optimizing them for consumer hardware.

Pros

  • Enables running LLaMA models on consumer-grade hardware
  • Highly optimized for CPU inference, with optional GPU acceleration
  • Supports various quantization methods to reduce model size and memory usage
  • Active community and frequent updates

Cons

  • Requires obtaining LLaMA model weights separately (not included due to licensing)
  • Limited to LLaMA and its derivatives, not a general-purpose framework
  • May have lower performance compared to specialized GPU frameworks for larger models
  • Setup process can be complex for non-technical users

Code Examples

  1. Basic model inference:
#include "llama.h"

int main() {
    llama_context * ctx = llama_init_from_file("model.bin", 0);
    llama_eval(ctx, {0, 1, 2, 3}, 4, 0, 1);
    float * logits = llama_get_logits(ctx);
    llama_free(ctx);
}
  1. Tokenization:
#include "llama.h"

int main() {
    llama_context * ctx = llama_init_from_file("model.bin", 0);
    std::vector<llama_token> tokens = llama_tokenize(ctx, "Hello, world!", false);
    llama_free(ctx);
}
  1. Generating text:
#include "llama.h"

int main() {
    llama_context * ctx = llama_init_from_file("model.bin", 0);
    const char * prompt = "Once upon a time";
    std::vector<llama_token> tokens = llama_tokenize(ctx, prompt, true);
    
    for (int i = 0; i < 50; i++) {
        llama_eval(ctx, tokens.data(), tokens.size(), i, 1);
        llama_token new_token = llama_sample_top_p_top_k(ctx, nullptr, 0, 40, 0.9, 0.0, 0.0);
        tokens.push_back(new_token);
        printf("%s", llama_token_to_str(ctx, new_token));
    }
    
    llama_free(ctx);
}

Getting Started

  1. Clone the repository:

    git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

  2. Build the project:

    make

  3. Download a compatible LLaMA model and convert it:

    python3 convert.py /path/to/llama/model

  4. Run the example:

    ./main -m models/7B/ggml-model-q4_0.bin -p "Once upon a time"

Note: Ensure you have the necessary dependencies installed and have obtained the LLaMA model weights legally before proceeding.

Competitor Comparisons

meta-llama logo

llama

58,578

Inference code for Llama models

Pros of llama

  • Official implementation from Meta, ensuring alignment with the original model architecture
  • Potentially more comprehensive documentation and support from the Meta team
  • May include advanced features or optimizations not present in community implementations

Cons of llama

  • Possibly more complex setup and usage compared to community-driven alternatives
  • May have stricter licensing or usage restrictions
  • Could be less optimized for specific use cases or hardware configurations

Code comparison

llama:

from llama import Llama

llm = Llama(model_path="path/to/model")
output = llm("Hello, how are you?")
print(output)

llama.cpp:

#include "llama.h"

llama_context * ctx = llama_init_from_file("path/to/model", params);
llama_eval(ctx, tokens, n_tokens, n_past, n_threads);
llama_print_timings(ctx);
llama_free(ctx);

The llama repository provides a Python interface, while llama.cpp offers a C++ implementation. llama.cpp is known for its efficiency and ability to run large language models on consumer hardware, making it popular for local deployments and experimentation. The official llama implementation may be more suitable for research or production use cases where alignment with Meta's original design is crucial.

karpathy logo

llama2.c

18,609

Inference Llama 2 in one file of pure C

Pros of llama2.c

  • Simplicity and educational value: Single C file implementation, easier to understand and learn from
  • Minimal dependencies: Relies only on standard C libraries, making it highly portable
  • Lightweight: Smaller codebase, potentially easier to modify or extend

Cons of llama2.c

  • Limited features: Fewer optimizations and functionalities compared to llama.cpp
  • Performance: May not be as fast or efficient as llama.cpp for larger models or production use
  • Compatibility: Supports fewer model formats and quantization options

Code Comparison

llama2.c:

float* malloc_float32(size_t n) { return (float*)malloc(n * sizeof(float)); }
int* malloc_int32(size_t n) { return (int*)malloc(n * sizeof(int)); }

llama.cpp:

template <typename T>
static T* new_array(size_t size) {
    return new T[size];
}

llama2.c focuses on simplicity with direct malloc calls, while llama.cpp uses templates for more flexible memory allocation. This exemplifies the difference in complexity and abstraction between the two projects.

abetlen logo

llama-cpp-python

9,403

Python bindings for llama.cpp

Pros of llama-cpp-python

  • Python bindings for easier integration into Python projects
  • Simplified API for loading models and generating text
  • Supports both CPU and GPU inference

Cons of llama-cpp-python

  • May have slightly lower performance due to Python overhead
  • Fewer low-level customization options compared to C++ version
  • Potentially slower updates and bug fixes

Code Comparison

llama.cpp (C++):

#include "llama.h"

llama_context * ctx = llama_init_from_file("model.bin", params);
llama_eval(ctx, tokens.data(), tokens.size(), n_past, n_threads);
llama_free(ctx);

llama-cpp-python (Python):

from llama_cpp import Llama

llm = Llama(model_path="model.bin")
output = llm("Hello, how are you?", max_tokens=100)
print(output)

The llama-cpp-python repository provides a more user-friendly Python interface to the underlying llama.cpp library, making it easier to integrate LLaMA models into Python projects. However, it may sacrifice some performance and low-level control compared to the original C++ implementation. The code comparison demonstrates the simplified API of the Python version, which abstracts away many of the details handled explicitly in the C++ version.

ggml-org logo

whisper.cpp

41,097

Port of OpenAI's Whisper model in C/C++

Pros of whisper.cpp

  • Specialized for speech recognition tasks
  • Potentially faster and more efficient for audio processing
  • Simpler implementation due to focused functionality

Cons of whisper.cpp

  • Limited to speech-to-text applications
  • Less versatile compared to llama.cpp's language modeling capabilities
  • Smaller community and fewer contributions

Code Comparison

whisper.cpp:

// Load model
struct whisper_context * ctx = whisper_init_from_file("ggml-base.en.bin");

// Process audio
whisper_full_default(ctx, wparams, pcmf32.data(), pcmf32.size());

// Print result
const char * text = whisper_full_get_segment_text(ctx, 0);
printf("%s\n", text);

llama.cpp:

// Load model
llama_context * ctx = llama_init_from_file("ggml-model-q4_0.bin", params);

// Generate text
llama_eval(ctx, tokens.data(), tokens.size(), n_past, n_threads);

// Get completion
float * logits = llama_get_logits(ctx);
top_k_top_p_filtering(logits, n_vocab, top_k, top_p, temp);

Both repositories utilize the GGML library for efficient machine learning operations, but they focus on different tasks. whisper.cpp is tailored for speech recognition, while llama.cpp is designed for general language modeling and text generation. The code snippets illustrate the different APIs and workflows for each project.

openai logo

whisper

85,961

Robust Speech Recognition via Large-Scale Weak Supervision

Pros of Whisper

  • Specialized for speech recognition and transcription tasks
  • Supports multiple languages and can perform translation
  • Backed by OpenAI's extensive research and development

Cons of Whisper

  • Limited to audio processing and transcription
  • Requires more computational resources for real-time processing
  • Less flexible for general-purpose language tasks

Code Comparison

Whisper:

import whisper

model = whisper.load_model("base")
result = model.transcribe("audio.mp3")
print(result["text"])

llama.cpp:

#include "llama.h"

llama_context * ctx = llama_init_from_file("model.bin", params);
llama_eval(ctx, tokens, n_tokens, n_past, n_threads);
llama_print_timings(ctx);
llama_free(ctx);

Key Differences

  • Whisper focuses on audio processing and transcription, while llama.cpp is a general-purpose language model implementation
  • Whisper is implemented in Python, making it easier for rapid prototyping and integration with data science workflows
  • llama.cpp is written in C++, offering better performance and lower-level control
  • Whisper provides pre-trained models for immediate use, while llama.cpp requires separate model files
  • llama.cpp is more suitable for edge devices and applications requiring low-latency inference
huggingface logo

transformers

146,142

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

Pros of Transformers

  • Supports a wide range of models and architectures
  • Extensive documentation and community support
  • Seamless integration with PyTorch and TensorFlow

Cons of Transformers

  • Higher resource requirements for running models
  • Steeper learning curve for beginners
  • Less optimized for inference on edge devices

Code Comparison

Transformers:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
input_text = "Hello, how are you?"
input_ids = tokenizer.encode(input_text, return_tensors="pt")
output = model.generate(input_ids, max_length=50)

llama.cpp:

#include "llama.h"

llama_context * ctx = llama_init_from_file("model.bin", params);
llama_eval(ctx, tokens, n_tokens, n_past, n_threads);
llama_print_timings(ctx);
llama_free(ctx);

The Transformers library offers a high-level API for working with various models, while llama.cpp provides a more low-level, C++ implementation specifically optimized for the LLaMA model. Transformers is more versatile but requires more resources, whereas llama.cpp is lightweight and efficient for inference on edge devices.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

llama.cpp

llama

License: MIT Release Server

Manifesto / ggml / ops

LLM inference in C/C++

Recent API changes

Hot topics

Quick start

Getting started with llama.cpp is straightforward. Here are several ways to install it on your machine:

Once installed, you'll need a model to work with. Head to the Obtaining and quantizing models section to learn more.

Example command:

# Use a local model file
llama-cli -m my_model.gguf

# Or download and run a model directly from Hugging Face
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

# Launch OpenAI-compatible API server
llama-server -hf ggml-org/gemma-3-1b-it-GGUF

Description

The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

  • Plain C/C++ implementation without any dependencies
  • Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
  • AVX, AVX2, AVX512 and AMX support for x86 architectures
  • 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
  • Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
  • Vulkan and SYCL backend support
  • CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is the main playground for developing new features for the ggml library.

Models

Typically finetunes of the base models below are supported as well.

Instructions for adding support for new models: HOWTO-add-model.md

Text-only

Multimodal

Bindings
UIs

(to have a project listed here, it should clearly state that it depends on llama.cpp)

Tools
  • akx/ggify – download PyTorch models from HuggingFace Hub and convert them to GGML
  • akx/ollama-dl – download models from the Ollama library to be used directly with llama.cpp
  • crashr/gppm – launch llama.cpp instances utilizing NVIDIA Tesla P40 or P100 GPUs with reduced idle power consumption
  • gpustack/gguf-parser - review/check the GGUF file and estimate the memory usage
  • Styled Lines (proprietary licensed, async wrapper of inference part for game development in Unity3d with pre-built Mobile and Web platform wrappers and a model example)
Infrastructure
  • Paddler - Stateful load balancer custom-tailored for llama.cpp
  • GPUStack - Manage GPU clusters for running LLMs
  • llama_cpp_canister - llama.cpp as a smart contract on the Internet Computer, using WebAssembly
  • llama-swap - transparent proxy that adds automatic model switching with llama-server
  • Kalavai - Crowdsource end to end LLM deployment at any scale
  • llmaz - ☸️ Easy, advanced inference platform for large language models on Kubernetes.
Games
  • Lucy's Labyrinth - A simple maze game where agents controlled by an AI model will try to trick you.

Supported backends

BackendTarget devices
MetalApple Silicon
BLASAll
BLISAll
SYCLIntel and Nvidia GPU
MUSAMoore Threads GPU
CUDANvidia GPU
HIPAMD GPU
VulkanGPU
CANNAscend NPU
OpenCLAdreno GPU
WebGPU [In Progress]All
RPCAll

Obtaining and quantizing models

The Hugging Face platform hosts a number of LLMs compatible with llama.cpp:

You can either manually download the GGUF file or directly use any llama.cpp-compatible models from Hugging Face or other model hosting sites, such as ModelScope, by using this CLI argument: -hf <user>/<model>[:quant]. For example:

llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

By default, the CLI would download from Hugging Face, you can switch to other options with the environment variable MODEL_ENDPOINT. For example, you may opt to downloading model checkpoints from ModelScope or other model sharing communities by setting the environment variable, e.g. MODEL_ENDPOINT=https://www.modelscope.cn/.

After downloading a model, use the CLI tools to run it locally - see below.

llama.cpp requires the model to be stored in the GGUF file format. Models in other data formats can be converted to GGUF using the convert_*.py Python scripts in this repo.

The Hugging Face platform provides a variety of online tools for converting, quantizing and hosting models with llama.cpp:

To learn more about model quantization, read this documentation

llama-cli

A CLI tool for accessing and experimenting with most of llama.cpp's functionality.

  • Run in conversation mode

    Models with a built-in chat template will automatically activate conversation mode. If this doesn't occur, you can manually enable it by adding -cnv and specifying a suitable chat template with --chat-template NAME

    llama-cli -m model.gguf

# > hi, who are you?
# Hi there! I'm your helpful assistant! I'm an AI-powered chatbot designed to assist and provide information to users like you. I'm here to help answer your questions, provide guidance, and offer support on a wide range of topics. I'm a friendly and knowledgeable AI, and I'm always happy to help with anything you need. What's on your mind, and how can I assist you today?
#
# > what is 1+1?
# Easy peasy! The answer to 1+1 is... 2!
  • Run in conversation mode with custom chat template 
    # use the "chatml" template (use -h to see the list of supported templates)
llama-cli -m model.gguf -cnv --chat-template chatml

# use a custom template
llama-cli -m model.gguf -cnv --in-prefix 'User: ' --reverse-prompt 'User:'
  • Run simple text completion

    To disable conversation mode explicitly, use -no-cnv

    llama-cli -m model.gguf -p "I believe the meaning of life is" -n 128 -no-cnv

# I believe the meaning of life is to find your own truth and to live in accordance with it. For me, this means being true to myself and following my passions, even if they don't align with societal expectations. I think that's what I love about yoga – it's not just a physical practice, but a spiritual one too. It's about connecting with yourself, listening to your inner voice, and honoring your own unique journey.
  • Constrain the output with a custom grammar 
    llama-cli -m model.gguf -n 256 --grammar-file grammars/json.gbnf -p 'Request: schedule a call at 8pm; Command:'

# {"appointmentTime": "8pm", "appointmentDetails": "schedule a a call"}

    The grammars/ folder contains a handful of sample grammars. To write your own, check out the GBNF Guide.

    For authoring more complex JSON grammars, check out https://grammar.intrinsiclabs.ai/

llama-server

A lightweight, OpenAI API compatible, HTTP server for serving LLMs.

  • Start a local HTTP server with default configuration on port 8080 
    llama-server -m model.gguf --port 8080

# Basic web UI can be accessed via browser: http://localhost:8080
# Chat completion endpoint: http://localhost:8080/v1/chat/completions
  • Support multiple-users and parallel decoding 
    # up to 4 concurrent requests, each with 4096 max context
llama-server -m model.gguf -c 16384 -np 4
  • Enable speculative decoding 
    # the draft.gguf model should be a small variant of the target model.gguf
llama-server -m model.gguf -md draft.gguf
  • Serve an embedding model 
    # use the /embedding endpoint
llama-server -m model.gguf --embedding --pooling cls -ub 8192
  • Serve a reranking model 
    # use the /reranking endpoint
llama-server -m model.gguf --reranking
  • Constrain all outputs with a grammar 
    # custom grammar
llama-server -m model.gguf --grammar-file grammar.gbnf

# JSON
llama-server -m model.gguf --grammar-file grammars/json.gbnf

llama-perplexity

A tool for measuring the perplexity 1 (and other quality metrics) of a model over a given text.

  • Measure the perplexity over a text file 
    llama-perplexity -m model.gguf -f file.txt

# [1]15.2701,[2]5.4007,[3]5.3073,[4]6.2965,[5]5.8940,[6]5.6096,[7]5.7942,[8]4.9297, ...
# Final estimate: PPL = 5.4007 +/- 0.67339
  • Measure KL divergence 
    # TODO

llama-bench

Benchmark the performance of the inference for various parameters.

  • Run default benchmark 
    llama-bench -m model.gguf

# Output:
# | model               |       size |     params | backend    | threads |          test |                  t/s |
# | ------------------- | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
# | qwen2 1.5B Q4_0     | 885.97 MiB |     1.54 B | Metal,BLAS |      16 |         pp512 |      5765.41 ± 20.55 |
# | qwen2 1.5B Q4_0     | 885.97 MiB |     1.54 B | Metal,BLAS |      16 |         tg128 |        197.71 ± 0.81 |
#
# build: 3e0ba0e60 (4229)

llama-run

A comprehensive example for running llama.cpp models. Useful for inferencing. Used with RamaLama 2.

  • Run a model with a specific prompt (by default it's pulled from Ollama registry) 
    llama-run granite-code

llama-simple

A minimal example for implementing apps with llama.cpp. Useful for developers.

  • Basic text completion 
    llama-simple -m model.gguf

# Hello my name is Kaitlyn and I am a 16 year old girl. I am a junior in high school and I am currently taking a class called "The Art of

Contributing

  • Contributors can open PRs
  • Collaborators can push to branches in the llama.cpp repo and merge PRs into the master branch
  • Collaborators will be invited based on contributions
  • Any help with managing issues, PRs and projects is very appreciated!
  • See good first issues for tasks suitable for first contributions
  • Read the CONTRIBUTING.md for more information
  • Make sure to read this: Inference at the edge
  • A bit of backstory for those who are interested: Changelog podcast

Other documentation

Development documentation

Seminal papers and background on the models

If your issue is with model generation quality, then please at least scan the following links and papers to understand the limitations of LLaMA models. This is especially important when choosing an appropriate model size and appreciating both the significant and subtle differences between LLaMA models and ChatGPT:

XCFramework

The XCFramework is a precompiled version of the library for iOS, visionOS, tvOS, and macOS. It can be used in Swift projects without the need to compile the library from source. For example:

// swift-tools-version: 5.10
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "MyLlamaPackage",
    targets: [
        .executableTarget(
            name: "MyLlamaPackage",
            dependencies: [
                "LlamaFramework"
            ]),
        .binaryTarget(
            name: "LlamaFramework",
            url: "https://github.com/ggml-org/llama.cpp/releases/download/b5046/llama-b5046-xcframework.zip",
            checksum: "c19be78b5f00d8d29a25da41042cb7afa094cbf6280a225abe614b03b20029ab"
        )
    ]
)

The above example is using an intermediate build b5046 of the library. This can be modified to use a different version by changing the URL and checksum.

Completions

Command-line completion is available for some environments.

Bash Completion

$ build/bin/llama-cli --completion-bash > ~/.llama-completion.bash
$ source ~/.llama-completion.bash

Optionally this can be added to your .bashrc or .bash_profile to load it automatically. For example:

$ echo "source ~/.llama-completion.bash" >> ~/.bashrc

Dependencies

  • yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
  • stb-image - Single-header image format decoder, used by multimodal subsystem - Public domain
  • nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
  • minja - Minimal Jinja parser in C++, used by various tools/examples - MIT License
  • linenoise.cpp - C++ library that provides readline-like line editing capabilities, used by llama-run - BSD 2-Clause License
  • curl - Client-side URL transfer library, used by various tools/examples - CURL License
  • miniaudio.h - Single-header audio format decoder, used by multimodal subsystem - Public domain

Footnotes

  1. https://huggingface.co/docs/transformers/perplexity ↩

  2. RamaLama ↩