ByteZone
Back to listClaim this project

ggml-orgDeveloper tools

llama.cpp

A lightweight LLM inference engine implemented in C/C++, supporting various hardware backends and model quantization.

  • Backend
  • DevOps
  • AI coding
  • Automation
  • CLI
  • Library/framework
  • Windows
  • macOS
  • Linux
  • Android
  • Browser
  • Self-hostable
  • Runs locally
  • Docker supported
Open websiteView on GitHub
llama.cpp screenshot
Popularity
124.8k Stars
GitHub stars
Recent activity
8/20/2026
Updated in the last 30 days
License
MIT
Permissive

Why it matters

We look beyond stars: what problem it solves, whether it creates real utility, and what makes its approach worth noticing.

Last 90 days

Problem

Addresses the challenge of running Large Language Models (LLMs) and Vision Language Models (VLMs) locally across diverse hardware (such as Apple Silicon, various GPUs, and CPUs) with minimal setup and dependency requirements.

Practical value

Provides a plain C/C++ implementation, multiple quantization options (1.5-bit to 8-bit), hybrid CPU+GPU inference, and various backends (CUDA, Metal, Vulkan, etc.), enabling users to run models directly via CLI or an OpenAI-compatible API server with a built-in web UI.

Innovation / differentiation

Built on top of the ggml library, achieving dependency-free cross-platform model inference through extensive hardware architecture optimizations (such as ARM NEON, AVX, RISC-V vector extensions) and custom hardware kernels.

Leverage potential

Supports Docker, pre-built binaries, and building from source, and can be integrated as a library or embedded into other local and cloud applications via command-line tools and API servers.

Why now

As local deployment of large models and multimodal interaction needs become widespread, the project provides an efficient inference solution across diverse hardware.

Community activity

In the last 90 days there were 1291 new issues and 2288 pull requests; the bounded issue/PR samples include 87 issue authors and 65 PR contributors, with at least 50 releases.

Maintainer responsiveness

The 100-issue window sample had a 93% close rate, and the 100-pull-request sample had a 51% merge rate. Maintainer-comment observations covered only 8% of that issue sample, so response rate and first-response speed are not reported.

65 contributors in PR sampleAt least 50 releasesPR merge rate 51% · 100 sampled in window

Key highlights

  • Pure C/C++ implementation with no external dependencies
  • Supports quantization formats from 1.5-bit to 8-bit
  • Extensive hardware backend support (CUDA, Metal, Vulkan, SYCL, etc.)

Quick start

How it is installed, how hard it is, and where to start.

Where it runs

Runs locally

Difficulty

Medium — some setup needed

Docker supportedSelf-hostableRuns locally
  1. 01Install llama.cpp by building from source or downloading pre-built binaries.
  2. 02Run a GGUF model from Hugging Face using the CLI tool.
  3. 03Launch the built-in server to provide an OpenAI-compatible API interface.

Best for

  • Teams that want data on their own servers
  • Developers who want to try it on their machine
  • People who prefer Docker deploys

More about it

llama.cpp is a high-performance inference engine for LLMs and VLMs, designed to run models on diverse hardware with minimal setup. Built on the ggml library, it achieves efficient performance through deep optimizations for specific architectures, including Metal for Apple Silicon and CUDA for NVIDIA GPUs.

The project supports various quantization techniques to reduce memory footprint and increase inference speed. Users can run models directly via CLI tools or launch the built-in HTTP server for integration into existing applications.

Sources

Each field shows its status and source — expand to review.

15 · Expand

  • capability tags

    Verified

    ai_coding, automation

    Source: admin_cms · cms editor · 8/21/2026

  • editor note

    Verified

    {"en":"This project serves as a foundation for running LLMs locally, implemented in pure C/C++ with minimal dependencies. It is suitable for developers deploying models on consumer hardware or edge devices.","zh":"该项目是本地运行大模型的基石，通过纯 C/C++ 实现，几乎不依赖外部库。适合需要将模型部署在消费级硬件或边缘设备上的开发者。"}

    Source: admin_cms · cms editor · 8/21/2026

  • how to use

    Verified

    {"steps":[{"en":"Install llama.cpp by building from source or downloading pre-built binaries.","zh":"通过源码编译或下载预构建二进制文件安装 llama.cpp。"},{"en":"Run a GGUF model from Hugging Face using the CLI tool.","zh":"使用命令行工具运行 Hugging Face 上的 GGUF 模型。"},{"en":"Launch the built-in server to provide an OpenAI-compatible API interface.","zh":"启动内置服务器以提供 OpenAI 兼容的 API 接口。"}],"installAt":"local","difficulty":"medium"}

    Source: admin_cms · cms editor · 8/21/2026

  • intro

    Verified

    {"en":"llama.cpp is a high-performance inference engine for LLMs and VLMs, designed to run models on diverse hardware with minimal setup. Built on the ggml library, it achieves efficient performance through deep optimizations for specific architectures, including Metal for Apple Silicon and CUDA for NVIDIA GPUs.\n\nThe project supports various quantization techniques to reduce memory footprint and increase inference speed. Users can run models directly via CLI tools or launch the built-in HTTP server for integration into existing applications.","zh":"llama.cpp 是一个高性能的 LLM 和 VLM 推理引擎，通过极简的设置在各类硬件上运行大模型。它基于 ggml 库构建，通过针对不同架构（如 Apple Silicon 的 Metal、NVIDIA 的 CUDA）的深度优化，实现了高效的推理性能。\n\n该项目支持多种量化技术，能减少内存占用并提升推理速度。用户可以通过命令行工具直接运行模型，或启动内置的 HTTP 服务以集成到现有应用中。"}

    Source: admin_cms · cms editor · 8/21/2026

  • Latest release

    Verified

    b10507

    Source: GitHub API · latest_release=b10507 · 8/20/2026

  • License

    Verified

    MIT

    Source: GitHub API · license.spdx_id=MIT · 8/20/2026

  • needs api key

    Verified

    No

    Source: admin_cms · cms editor · 8/21/2026

  • One-liner

    Verified

    {"en":"A lightweight LLM inference engine implemented in C/C++, supporting various hardware backends and model quantization.","zh":"基于 C/C++ 实现的轻量级大语言模型推理引擎，支持多种硬件加速与模型量化。"}

    Source: admin_cms · cms editor · 8/21/2026

  • Platforms

    Verified

    windows, macos, linux, android, browser

    Source: admin_cms · cms editor · 8/21/2026

  • Category hint

    Inferred from materials

    ai-apps

    Source: Project README · hint=ai-apps · 8/20/2026

  • product forms

    Verified

    cli, library_framework

    Source: admin_cms · cms editor · 8/21/2026

  • role tags

    Verified

    backend, devops

    Source: admin_cms · cms editor · 8/21/2026

  • supports docker

    Verified

    Yes

    Source: admin_cms · cms editor · 8/21/2026

  • supports local

    Verified

    Yes

    Source: admin_cms · cms editor · 8/21/2026

  • supports self host

    Verified

    Yes

    Source: admin_cms · cms editor · 8/21/2026

Related projects

Other verified projects matched by category, capabilities, and intended roles.

claude-code

An agentic coding tool for the terminal that understands codebases, handles routine development tasks, explains code, and works with Git workflows.

Same Developer tools categoryBoth support AI coding

spec-kit

An open-source CLI toolkit for Spec-Driven Development that guides AI coding agents through structured specification, planning, and code generation steps.

Same Developer tools categoryBoth support AI coding

codex

OpenAI’s coding agent that runs locally, with terminal, IDE, and desktop-app entry points.

Same Developer tools categoryBoth support AI coding

CLI-Anything

A toolkit for generating and managing agent-callable CLIs for existing software, with CLI-Hub, generation workflows, and skill/plugin integrations.

Same Developer tools categoryBoth support AI coding

playwright

A cross-browser automation framework for testing and AI agent control, supporting Chromium, Firefox, and WebKit.

Same Developer tools categoryBoth support AI coding

codebase-memory-mcp

A high-performance MCP server that builds local persistent knowledge graphs for millisecond-latency code intelligence and structural queries.

Same Developer tools categoryBoth support AI coding