llama.cpp: High-Performance Open-Source LLM Inference in C/C++

Summary

llama.cpp: a dependency-free C/C++ LLM/VLM inference engine on GGML, enabling quantized local execution across Apple Silicon, CUDA, Vulkan and CPUs, the backbone of much local AI tooling.

Why it matters

Serves as the critical infrastructure powering offline local LLM deployment, private security research environments, and air-gapped agent operations without data leakage risks.

Published: 2026-09-06 22:10
Collected: 2026-09-07 03:05

📦 Full-Text Content Snapshot

R2 Archive · Full Article Preserved139 lines · 8.2 KB

📄 Raw .md ⬇️ Download .md

Source: github.com/ggml-org/llama.cpp · Published: 2026-09-06T22:10:56Z

llama

LLM inference in C/C++

License: MIT

ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API

Quick start

A few options to get llama.cpp installed on your machine:

bash

# curl
curl -LsSf https://llama.app/install.sh | sh

# powershell
irm https://llama.app/install.ps1 | iex

Visit https://llama.app and follow the instructions
Run with Docker - see our Docker documentation
Download pre-built binaries from the releases page
Build from source by cloning this repository - check out our build guide

Once installed:

bash

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

Plain C/C++ implementation without any dependencies
Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
AVX, AVX2, AVX512 and AMX support for x86 architectures
RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
Vulkan and SYCL backend support
CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is built on top of the ggml library.

Supported backends

Backend	Target devices
BLAS	All
BLIS	All
CANN	Ascend NPU
CUDA	Nvidia GPU
HIP	AMD GPU
Hexagon	Snapdragon
IBM zDNN	IBM Z & LinuxONE
MUSA	Moore Threads GPU
Metal	Apple Silicon
OpenCL	Adreno GPU
[OpenVINO [In Progress]](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/OPENVINO.md)	Intel CPUs, GPUs, and NPUs
RPC	All
SYCL	Intel GPU
VirtGPU	VirtGPU APIR
Vulkan	GPU
WebGPU	All
ZenDNN	AMD CPU

Documentation

Tools

Development

Contributing

Contributors can open PRs
Collaborators will be invited based on contributions
Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
Any help with managing issues, PRs and projects is very appreciated!
Read the CONTRIBUTING.md for more information

Acknowledgements

yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain

📦 Full-Text Markdown Snapshot archived by Phying News · Original source: github.com/ggml-org/llama.cpp

---
id: 7530
canonical_url: "https://phying.com/item/7530"
title: "llama.cpp: High-Performance Open-Source LLM Inference in C/C++"
type: "tool"
cve: null
cvss: null
severity: "unknown"
vendor: null
product: "llama.cpp"
affected_versions: null
source_domain: "github.com"
source_url: "https://github.com/ggml-org/llama.cpp"
published_at: "2026-09-06T22:10:56Z"
snapshot_at: "2026-10-26T05:55:00Z"
archived_by: "Phying News (https://phying.com/item/7530)"
lang: "en"
---

# llama.cpp: High-Performance Open-Source LLM Inference in C/C++

*Source: [github.com/ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) · Published: 2026-09-06T22:10:56Z*

[![llama](https://raw.githubusercontent.com/ggml-org/llama.brand/refs/heads/master/cover/llama-cpp/cover-llama-cpp-dark.svg)](https://raw.githubusercontent.com/ggml-org/llama.brand/refs/heads/master/cover/llama-cpp/cover-llama-cpp-dark.svg)

**LLM inference in C/C++**

[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![Release](https://img.shields.io/github/v/release/ggml-org/llama.cpp?filter=v*&color=brightgreen)](https://github.com/ggml-org/llama.cpp/releases?q=tag:v0)
[![Nightly](https://img.shields.io/github/v/release/ggml-org/llama.cpp?label=nightly&filter=b*&color=orange)](https://github.com/ggml-org/llama.cpp/releases?q=b)
[![Server](https://img.shields.io/github/actions/workflow/status/ggml-org/llama.cpp/server.yml?label=Server)](https://github.com/ggml-org/llama.cpp/actions/workflows/server.yml)
[![Docker](https://img.shields.io/github/actions/workflow/status/ggml-org/llama.cpp/docker.yml?label=Docker)](https://github.com/ggml-org/llama.cpp/actions/workflows/docker.yml)
[![Winget](https://img.shields.io/github/actions/workflow/status/ggml-org/llama.cpp/winget.yml?label=Winget)](https://github.com/ggml-org/llama.cpp/actions/workflows/winget.yml)

[ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md) / [maintainer PRs](https://github.com/ggml-org/llama.cpp/issues?q=is%3Apr%20is%3Aopen%20draft%3AFalse) / [dev stats](https://github.com/ggml-org/llama.cpp-dev) / [lib llama API](https://github.com/ggml-org/llama.cpp/issues/9289) / [llama-server REST API](https://github.com/ggml-org/llama.cpp/issues/9291)

## Quick start

A few options to get `llama.cpp` installed on your machine:

```bash
# curl
curl -LsSf https://llama.app/install.sh | sh

# powershell
irm https://llama.app/install.ps1 | iex
```

- Visit [https://llama.app](https://llama.app) and follow the instructions
- Run with Docker - see our [Docker documentation](https://github.com/ggml-org/llama.cpp/blob/master/docs/docker.md)
- Download pre-built binaries from the [releases page](https://github.com/ggml-org/llama.cpp/releases)
- Build from source by cloning this repository - check out [our build guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md)

Once installed:

```bash
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
```

## Description

The main goal of `llama.cpp` is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

- Plain C/C++ implementation without any dependencies
- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
- AVX, AVX2, AVX512 and AMX support for x86 architectures
- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
- Vulkan and SYCL backend support
- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The `llama.cpp` project is built on top of the [ggml](https://github.com/ggml-org/ggml) library.

## Supported backends

| Backend | Target devices |
| --- | --- |
| [BLAS](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#blas-build) | All |
| [BLIS](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/BLIS.md) | All |
| [CANN](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#cann) | Ascend NPU |
| [CUDA](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#cuda) | Nvidia GPU |
| [HIP](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#hip) | AMD GPU |
| [Hexagon](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/snapdragon/README.md) | Snapdragon |
| [IBM zDNN](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/zDNN.md) | IBM Z & LinuxONE |
| [MUSA](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#musa) | Moore Threads GPU |
| [Metal](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#metal-build) | Apple Silicon |
| [OpenCL](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/OPENCL.md) | Adreno GPU |
| [OpenVINO [In Progress]](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/OPENVINO.md) | Intel CPUs, GPUs, and NPUs |
| [RPC](https://github.com/ggml-org/llama.cpp/tree/master/tools/rpc) | All |
| [SYCL](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/SYCL.md) | Intel GPU |
| [VirtGPU](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/VirtGPU.md) | VirtGPU APIR |
| [Vulkan](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#vulkan) | GPU |
| [WebGPU](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#webgpu) | All |
| [ZenDNN](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#zendnn) | AMD CPU |

## Documentation

#### Tools
- [cli](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md)
- [completion](https://github.com/ggml-org/llama.cpp/blob/master/tools/completion/README.md)
- [server](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)
- [GBNF grammars](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md)

#### Development
- [How to build](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md)
- [Running on Docker](https://github.com/ggml-org/llama.cpp/blob/master/docs/docker.md)
- [Build on Android](https://github.com/ggml-org/llama.cpp/blob/master/docs/android.md)
- [Multi-GPU usage](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md)
- [Performance troubleshooting](https://github.com/ggml-org/llama.cpp/blob/master/docs/development/token_generation_performance_tips.md)
- [GGML tips & tricks](https://github.com/ggml-org/llama.cpp/wiki/GGML-Tips-&-Tricks)
- [XCFramework](https://github.com/ggml-org/llama.cpp/blob/master/docs/xcframework.md)
- [Completions](https://github.com/ggml-org/llama.cpp/blob/master/docs/completions.md)
- [Models](https://github.com/ggml-org/llama.cpp/blob/master/docs/models.md)
- [Release process](https://github.com/ggml-org/llama.cpp/blob/master/docs/release.md)

## Contributing

- Contributors can open PRs
- Collaborators will be invited based on contributions
- Maintainers can push to branches in the `llama.cpp` repo and merge PRs into the `master` branch
- Any help with managing issues, PRs and projects is very appreciated!
- Read the [CONTRIBUTING.md](https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md) for more information

## Acknowledgements

- [yhirose/cpp-httplib](https://github.com/yhirose/cpp-httplib) - Single-header HTTP server, used by `llama-server` - MIT license
- [nothings/stb](https://github.com/nothings/stb) - Single-header image format decoder, used by multimodal subsystem - Public domain
- [nlohmann/json](https://github.com/nlohmann/json) - Single-header JSON library, used by various tools/examples - MIT License
- [mackron/miniaudio](https://github.com/mackron/miniaudio) - Single-header audio format decoder, used by multimodal subsystem - Public domain
- [sheredom/subprocess.h](https://github.com/sheredom/subprocess.h) - Single-header process launching solution for C and C++ - Public domain

---
> 📦 **Full-Text Markdown Snapshot** archived by [Phying News](https://phying.com/item/7530) · Original source: [github.com/ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp)

Back