sudoingXDeveloper tools
Qwen38 Mtp
qwen38-mtp
Unlocks 33% to 107% decode speedup for Qwen3.8 on consumer GPUs using a built-in llama.cpp flag without converting files.
- Backend
- DevOps
- Data
- Productivity
- CLI
- Windows
- macOS
- Linux
- Browser
- Self-hostable
- Runs locally
- Popularity
- 213 Stars
- GitHub stars
- Recent activity
- 8/18/2026
- Updated in the last 30 days
- License
- APACHE-2.0
- Permissive
Why it matters
We look beyond stars: what problem it solves, whether it creates real utility, and what makes its approach worth noticing.
Problem
Discovers and exposes a llama.cpp command-line flag combination that activates the built-in MTP head in Qwen3.8-27B, directly addressing slow decode speeds on consumer GPUs.
Practical value
Delivers immediate performance gains (33% to 107% faster decode speeds) without requiring file modification or re-quantization, backed by a complete testing script and cross-hardware community benchmarks.
Innovation / differentiation
Highlights the neglected multi-token prediction layers already present in model weights, converting them into practical performance gains via inference server configuration.
Leverage potential
Offers exceptionally high leverage by using simple parameter adjustments to scale hardware efficiency for existing local large model deployments.
Why now
Follows closely on the heels of the model release, quickly consolidating reusable configuration rules and validation tools during the community exploration phase.
Community activity
In the last 90 days there were 4 new issues and 60 pull requests; the bounded issue/PR samples include 4 issue authors and 49 PR contributors, with 0 releases.
Maintainer responsiveness
The 4-issue window sample had a 50% close rate, and the 60-pull-request sample had a 7% merge rate. Maintainer-comment observations covered only 100% of that issue sample, so response rate and first-response speed are not reported.
Key highlights
- Unlocks built-in MTP head for Qwen3.8-27B via specific llama.cpp flags to accelerate decode speed on consumer GPUs
- Includes probe.py benchmarking tool to measure and compare streaming server token generation rates
- Supports tuning parameters like --spec-draft-n-max and --spec-draft-p-min to optimize speculative decoding across hardware
Quick start
How it is installed, how hard it is, and where to start.
Where it runs
Runs locally
Difficulty
Medium — some setup needed
- 01Download the Qwen3.8-27B GGUF model files containing MTP weights
- 02Start the local inference server using llama-server with spec-type flags
- 03Run the included probe script to measure baseline and accelerated performance
Best for
- Teams that want data on their own servers
- Developers who want to try it on their machine
More about it
Qwen3.8-27B MTP is a community-driven benchmark and configuration guide for accelerating large language model inference. The Qwen3.8 model natively includes multi-token prediction heads which are preserved inside distributed GGUF files.
By supplying specific runtime flags to llama.cpp servers, users can instantly engage built-in speculative decoding without file conversions. The repository documents optimal tuning parameters across various consumer GPUs.
Sources
Each field shows its status and source — expand to review.
13 · Expand
Sources
Each field shows its status and source — expand to review.
capability tags
Verifiedproductivity
Source: admin_cms · cms editor · 8/22/2026
editor note
Verified{"en":"This project highlights how a single server flag activates built-in MTP heads in GGUF models for major decode speedups, backed by community benchmarks across varied hardware.","zh":"该项目展示了如何仅通过命令行参数激活 GGUF 模型自带的 MTP 头来获得显著提速，并整理了不同硬件配置下的最佳 n-max 和 p-min 参数。适合需要在消费级显卡上高吞吐量运行大语言模型的开发者。"}
Source: admin_cms · cms editor · 8/22/2026
how to use
Verified{"steps":[{"en":"Download the Qwen3.8-27B GGUF model files containing MTP weights","zh":"下载包含 MTP 权重的 Qwen3.8-27B GGUF 模型文件"},{"en":"Start the local inference server using llama-server with spec-type flags","zh":"使用带有 spec-type 标志的 llama-server 启动本地推理服务"},{"en":"Run the included probe script to measure baseline and accelerated performance","zh":"运行配套的探测脚本对基准性能进行测量"}],"installAt":"local","difficulty":"medium"}
Source: admin_cms · cms editor · 8/22/2026
intro
Verified{"en":"Qwen3.8-27B MTP is a community-driven benchmark and configuration guide for accelerating large language model inference. The Qwen3.8 model natively includes multi-token prediction heads which are preserved inside distributed GGUF files.\n\nBy supplying specific runtime flags to llama.cpp servers, users can instantly engage built-in speculative decoding without file conversions. The repository documents optimal tuning parameters across various consumer GPUs.","zh":"Qwen3.8-27B MTP 是一个针对大语言模型推理性能优化的开源实践指南与社区基准记录。Qwen3.8 模型在训练时已内置多 token 预测头（MTP），其量化后的 GGUF 文件同样保留了相关权重张量。\n\n通过配置 llama.cpp 的服务器启动参数，用户无需转换任何文件或构建自定义版本，即可直接驱动内置投机解码功能。社区通过多台机器的实测数据，总结出了不同显卡型号在代码、文本生成等场景下的最佳参数组合。"}
Source: admin_cms · cms editor · 8/22/2026
License
VerifiedApache-2.0
Source: GitHub API · license.spdx_id=Apache-2.0 · 8/26/2026
needs api key
VerifiedNo
Source: admin_cms · cms editor · 8/22/2026
One-liner
Verified{"en":"Unlocks 33% to 107% decode speedup for Qwen3.8 on consumer GPUs using a built-in llama.cpp flag without converting files.","zh":"通过启用 llama.cpp 的内置标志，无需转换文件即可为消费级显卡上的 Qwen3.8 模型提速 33% 至 107%。"}
Source: admin_cms · cms editor · 8/22/2026
Platforms
Verifiedwindows, macos, linux, browser
Source: admin_cms · cms editor · 8/22/2026
Category hint
Inferred from materialsai-apps
Source: Project README · hint=ai-apps · 8/16/2026
product forms
Verifiedcli
Source: admin_cms · cms editor · 8/22/2026
role tags
Verifiedbackend, devops, data
Source: admin_cms · cms editor · 8/22/2026
supports local
VerifiedYes
Source: admin_cms · cms editor · 8/22/2026
supports self host
VerifiedYes
Source: admin_cms · cms editor · 8/22/2026
Related projects
Other verified projects matched by category, capabilities, and intended roles.
spec-kit
An open-source CLI toolkit for Spec-Driven Development that guides AI coding agents through structured specification, planning, and code generation steps.
CLI-Anything
A toolkit for generating and managing agent-callable CLIs for existing software, with CLI-Hub, generation workflows, and skill/plugin integrations.
WSL
Windows Subsystem for Linux allows you to run Linux command-line tools and applications directly on Windows without the overhead of a traditional virtual machine.
zstd
Zstandard is a high-performance lossless compression algorithm that provides real-time compression speeds with zlib-level or better compression ratios.
openwiki
A CLI that uses agents to generate and continuously maintain Markdown wikis for codebases or personal knowledge, with a local graph visualizer.
opencodex
A universal local proxy that enables the use of any LLM, such as Claude, Gemini, or DeepSeek, within Codex and Claude Code, while managing ChatGPT account pools.