Shypd.aiThe largest open index of AI toolsonnxruntime-genai is an open-source tool that provides generative AI extensions for ONNX Runtime. It offers a flexible and performant way to run large language models (LLMs) on devices, including pre and post-processing, inference, and KV cache management.
Trending
About
onnxruntime-genai provides generative AI extensions for ONNX Runtime, enabling users to run generative AI models efficiently on various devices. This API implements the complete generative AI loop for ONNX models, encompassing pre and post-processing, inference with ONNX Runtime, logits processing, search and sampling, KV cache management, and grammar specification for tool calling. It powers applications like Foundry Local, Windows ML, and the Visual Studio Code AI Toolkit. The tool supports a wide range of model architectures, including Llama, Mistral, Gemma, and Phi, and offers APIs for Python, C#, C/C++, and Java across multiple operating systems and hardware accelerations like CUDA, DirectML, and OpenVINO. Key features include multi-LoRA, continuous decoding, constrained decoding, and speculative decoding.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
