Shypd.aiThe largest open index of AI toolsMMMU is an open-source benchmark tool for evaluating multimodal AI models. It provides evaluation code for assessing AI performance on complex, multi-discipline tasks, including a more robust version called MMMU-Pro.
Trending
About
MMMU is a comprehensive benchmark designed to evaluate multimodal AI models on massive, multi-discipline tasks requiring college-level subject knowledge and deliberate reasoning. It features 11.5K meticulously collected multimodal questions from exams, quizzes, and textbooks across six core disciplines and 30 subjects, encompassing 32 heterogeneous image types. The benchmark challenges models with advanced perception and reasoning using domain-specific knowledge. MMMU-Pro, a more robust version, further refines assessment by filtering text-only answerable questions, augmenting candidate options, and using a vision-only input setting. The tool provides evaluation code for both benchmarks, allowing researchers to assess model performance locally or submit predictions to an EvalAI server. It highlights significant room for improvement in current advanced models like GPT-4V.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
