Shypd.aiThe largest open index of AI toolsgemma.cpp is a lightweight, standalone C++ inference engine for Google's Gemma models, enabling developers to implement and deploy Gemma foundation models. It supports CPU-only inference for Gemma 2-3 and PaliGemma 2 models, focusing on simplicity and directness.
Trending
About
gemma.cpp is a lightweight, standalone C++ inference engine specifically designed for Google's Gemma foundation models. It provides a minimalist implementation for Gemma 2-3 and PaliGemma 2 models, prioritizing simplicity and directness over full generality, making it suitable for experimentation and research. The engine supports CPU-only inference, offering features like sampling with TopK and temperature, and a backward pass (VJP) with Adam optimizer for Gemma research. It includes optimizations such as mixed-precision GEMM (fp8, bf16, fp32, fp64 bit), automatic runtime autotuning, and integrated weight compression. The project leverages the Google Highway Library for portable SIMD, ensuring efficient CPU inference. It offers C++ APIs with streaming for single and batched inference, a basic interactive command-line app, and Python bindings. gemma.cpp is designed to be easily embeddable in other projects with minimal dependencies and is highly modifiable, featuring a small core implementation.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
