Multilingual multimodal reasoning is a core component in achieving human-level intelligence. However, most existing benchmarks for multilingual multimodal reasoning struggle to differentiate between models of varying performance; even language models without visual capabilities can easily achieve high scores. This leaves a comprehensive evaluation of leading multilingual multimodal models largely unexplored. In this work, we introduce M4U, a novel and challenging benchmark for assessing the capability of multi-discipline multilingual multimodal understanding and reasoning. M4U contains 8,931 samples covering 64 disciplines across 16 subfields in Science, Engineering, and Healthcare in Chinese, English, and German. Using M4U, we conduct extensive evaluations of 21 leading Large Multimodal Models (LMMs) and Large Language Models (LLMs) with external tools. The evaluation results show that the state-of-the-art model, GPT-4o, achieves only 47.6% average accuracy on M4U. Additionally, we observe that the leading LMMs exhibit significant language preferences. Our in-depth analysis indicates that leading LMMs, including GPT-4o, suffer performance degradation when prompted with cross-lingual multimodal questions, such as images with key textual information in Chinese while the question is in German. We believe that M4U can serve as a crucial tool for systematically evaluating LMMs based on their multilingual multimodal reasoning capabilities and monitoring their development.

An illustration of multi-discipline multilingual multimodal understanding. Both textual questions and images contain the multilingual contents. We highlight the Chinese contents in yellow. English translations are provided for better readability.

Our codes and data are available at Hugging Face Dataset.

The detailed statistics of M4U dataset.

To boost the development of multilingual multimodal models, in this work, we introduce M4U, a novel and challenging benchmark for evaluating foundation models on the expert-level multilingual multimodal understanding and reasoning. Specifically, we recruit a team of over 10 college students and graduate students to collect a high-quality data and assess its difficulty and correctness. M4U consists of 8,931 multiple choice questions, covering 64 disciplines of 16 subfields from Science, Engineering and Healthcare in Chinese, English and German.. To minimize the risk of data contamination, the samples are collected from college exams, the quizzes of online video lectures. Further a large portion 35% of M4U are written by our team according to the textbooks.

With M4U, we conduct extensive evaluations for 21 leading LMMs and LLMs with external tools. The evaluation results show that the state-of-the-art model, GPT-4o, only achieves 47.6% average accuracy on M4U. Besides, we observe that the leading LMMs have significant language preferences.

Our in-depth analysis shows that the leading LMMs, including GPT-4o, suffer from the performance degradation when they are prompted with cross-lingual multimodal questions, e.g., the images have key textual information in Chinese, while the question is in English or German.

We further analyze the impact of different types of visual content and image positions. The experimental results show that GPT-4o significantly outperforms the other models in medical images, and LLaVA-NeXT has difficulty answering questions where images are included in the options.