Shypd.aiThe largest open index of AI tools
PaLM-E is a research project focused on developing an embodied multimodal language model that integrates real-world continuous sensor modalities into language models. It enables general inference in the real world for robotics problems, including manipulation planning and visual question answering.
Trending
About
PaLM-E is an embodied multimodal language model designed to bridge the gap between large language models and real-world physical interaction, particularly for robotics. It achieves this by directly incorporating continuous sensor modalities, such as visual and state estimation inputs, into the language embedding space of a pre-trained language model like PaLM. This allows PaLM-E to process multi-modal sentences that interleave visual, continuous state, and textual input encodings. The model is trained end-to-end for various embodied tasks, including sequential robotic manipulation planning, visual question answering, and captioning. PaLM-E demonstrates the ability to address diverse embodied reasoning tasks across multiple observation modalities and embodiments, exhibiting positive transfer from diverse joint training across internet-scale language, vision, and visual-language domains. The largest version, PaLM-E-562B, also achieves state-of-the-art performance on OK-VQA while retaining generalist language capabilities.
Capabilities
Pricing & Plans
Likely Free
Open Source
FAQs
Trending
