Lm.Rs

Visit Tool

lm.rs is a Coding & Development tool for minimal LLM inference in Rust. It allows running language models locally on the CPU, supporting multimodal models like PHI-3.5-vision.

Claim this tool

1View

At a glance

Pricing

Open Source

Free tier

Yes

API

Skill level

Technical

About

What is lm.rs?

lm.rs is an open-source Rust library designed for minimal Language Model (LLM) inference directly on the CPU. Inspired by projects like llama2.c and llm.c, it aims to provide a lightweight solution for running LLMs locally without relying on extensive machine learning libraries. The tool supports various models, including Google's Gemma 2, Llama 3.2, and multimodal PHI-3.5-vision, enabling both text and vision capabilities. It offers features like batch processing for improved image encoding speed and provides prepared quantized models for efficient inference. Users can convert models from Hugging Face or build the Rust code to run chat applications or a web UI backend.

Best used for

Ideal for developers who need to run language models locally on the CPU, experiment with different model quantizations, and integrate AI inference into Rust-based applications. Especially valuable for those seeking a minimal, self-contained solution for LLM deployment without heavy ML frameworks.

Common actions

run language models

perform AI inference

develop AI applications

optimize model performance

automated workflowcollaborationworkflowsdeepfakelow-code/no-codeopen-source"AI Agents"github copilotface swapping

Capabilities

Key features

Local CPU inference
Multimodal model support
Model quantization
Batch processing
Rust implementation
WebUI backend

Target Audience

developer

Integrations

Not yet documented

Pricing & Plans

Open Source

Free

FAQs

What types of language models does lm.rs support?

lm.rs supports various language models, including Google's Gemma 2, Llama 3.2, and multimodal PHI-3.5-vision. It allows for both text-only and vision-enabled inference, with options for different quantization levels to optimize performance and model size.

Can I convert my own models to work with lm.rs?

Yes, lm.rs provides Python scripts (export.py and tokenizer.py) to convert models from Hugging Face. You can convert .safetensors and config.json files into the LMRS format, including options for quantization and multimodal configurations for models like PHI-3.5-vision.

What are the performance characteristics of lm.rs?

Performance varies by model and quantization. For example, the Llama 3.2 1B IT Q8_0 model can achieve 50 tok/s on a 16-core AMD Epyc machine. The tool also features batch processing to boost image encoding speed by up to ~3x for multimodal models.

Trending

Subcategories trending in Coding & Development

Open Source & Models DevOps & Infrastructure No-Code / Low-Code Testing & QA Backend & APIs Prompt Engineering

Trending

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce