LMDrive

Visit Tool

LMDrive is an AI Agents & Automation tool that provides a closed-loop, end-to-end driving framework using large language models. It interacts with dynamic environments via multi-modal sensor data and natural language instructions.

Claim this tool

1View

At a glance

Pricing

Open Source

Free tier

Yes

API

Skill level

Technical

About

What is LMDrive?

LMDrive is an open-source, closed-loop, end-to-end autonomous driving framework that leverages large language models (LLMs). It is designed to interact with dynamic environments by processing multi-modal and multi-view sensor data, alongside natural language instructions. This framework facilitates the development and research of advanced autonomous driving systems. Key features include vision encoder pre-training to generate visual tokens from sensor inputs and an instruction finetuning stage to align language instructions with control signals. The project provides a comprehensive dataset collected in the CARLA simulator, including sensor data, navigation instructions, and human notice instructions, making it a robust platform for researchers and developers in the autonomous driving domain.

Best used for

Ideal for developers and data scientists who need to research and develop advanced autonomous driving systems, train language-guided autonomous vehicles, and process multi-modal sensor data. Especially valuable for those working with the CARLA simulator and requiring robust, open-source frameworks.

Common actions

develop autonomous driving

train language models

process sensor data

integrate natural language

automated workflowlow-code/no-codeworkflowsopen-sourcedeepfakecollaboration"AI Agents"github copilotface swapping

Capabilities

Key features

Closed-loop end-to-end driving
Large Language Model integration
Multi-modal sensor data processing
Natural language instruction processing
Vision encoder pre-training
Instruction finetuning
CARLA simulator dataset

Target Audience

developerdata scientist

Integrations

Not yet documented

Pricing & Plans

Open Source

Free

FAQs

What kind of data does LMDrive use for training?

LMDrive utilizes a comprehensive dataset collected in the CARLA simulator. This dataset includes multi-view camera and LiDAR sensor data, navigation instructions, and human notice instructions. Each data clip contains a sequence of multi-modal sensor data and control signals, along with various instructions.

What are the main stages of LMDrive's training process?

LMDrive's training consists of two primary stages. The first is vision encoder pre-training, which generates visual tokens from sensor inputs. The second stage is instruction finetuning, where the system aligns language instructions and visual information with the appropriate control signals for autonomous driving.

What hardware is recommended for training LMDrive?

LMDrive is trained on 8 A100 GPUs with 80GB memory. However, the vision encoder pre-training stage can be performed on GPUs with 32GB memory. For setups with fewer GPUs, it is recommended to reduce the batch size and learning rate proportionally.

Trending

Subcategories trending in AI Agents & Automation

AI Frameworks & Infra Chatbots & Conversational AI Workflow Agents Personal Assistants RAG & Document AI Voice Agents

Trending

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce