OnnxStream

Visit Tool

OnnxStream is an Open Source & Models tool that provides a lightweight inference library for ONNX files. It can run large models like Stable Diffusion XL 1.0 on low-resource devices such as a Raspberry Pi Zero 2 with minimal RAM.

Claim this tool

1View

At a glance

Pricing

Open Source

Free tier

Yes

API

Yes

Skill level

Technical

About

What is OnnxStream?

OnnxStream is a lightweight inference library written in C++ designed to run ONNX models with minimal memory consumption. It excels at enabling complex AI models, such as Stable Diffusion XL 1.0 and Mistral 7B, to operate on resource-constrained devices like the Raspberry Pi Zero 2, requiring as little as 298MB of RAM. The library supports various architectures including ARM, x86, WASM, and RISC-V, and is accelerated by XNNPACK. It features a decoupled inference engine from its WeightsProvider, allowing flexible data loading and caching strategies. Python, C#, and JavaScript (WASM) bindings are available, making it versatile for different development environments and applications, including web-based demos.

Best used for

Ideal for developers and data scientists who need to deploy large AI models like Stable Diffusion XL 1.0 or Mistral 7B on low-resource hardware, optimize memory usage during inference, and integrate AI capabilities into various platforms including web browsers. Especially valuable for projects requiring efficient, embedded AI solutions.

Common actions

deploy AI models

optimize model inference

run ONNX files

develop AI applications

integrate AI models

workflowsopen-sourceautomated workflowlow-code/no-codecollaborationface swappinggithub copilot"AI Agents"deepfake

Capabilities

Key features

Lightweight ONNX inference
Low memory consumption
ARM, x86, WASM, RISC-V support
Python, C#, JS bindings
XNNPACK acceleration
Attention slicing
Dynamic/static quantization

Target Audience

developerdata scientist

Integrations

Not yet documented

Pricing & Plans

Open Source

Free

FAQs

What kind of AI models can OnnxStream run?

OnnxStream is designed to run ONNX (Open Neural Network Exchange) models. It has been demonstrated to run large models such as Stable Diffusion XL 1.0, Stable Diffusion XL Turbo 1.0, TinyLlama 1.1B, Mistral 7B, YOLOv8, and OpenAI's Whisper, even on resource-constrained devices.

What hardware architectures does OnnxStream support?

OnnxStream supports a wide range of hardware architectures including ARM, x86, WASM (WebAssembly), and RISC-V. This broad compatibility allows for deployment on various devices from Raspberry Pi Zero 2 to desktops and servers.

How does OnnxStream achieve such low memory consumption?

OnnxStream minimizes memory usage through techniques like attention slicing, dynamic and static quantization (8-bit unsigned), and a decoupled inference engine from its WeightsProvider. This allows for efficient loading and caching of model parameters, significantly reducing RAM requirements.

Trending

Subcategories trending in Coding & Development

Code Assistants DevOps & Infrastructure No-Code / Low-Code Testing & QA Backend & APIs Prompt Engineering

Trending

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce