MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMs

Abstract

Audiovisual arts encompass diverse creative disciplines, including cinema, visual arts, stage performance, and game design, where artistic meaning arises from deliberate combinations of visual, auditory, and narrative elements. True artistic understanding extends beyond recognizing what is depicted to reasoning about why it is expressed through particular creative choices. Despite the strong progress of multimodal large language models (MLLMs), this critical aspect of artistic understanding remains underexplored, as existing benchmarks largely measure perceptual recognition while overlooking reasoning about creative intent. To address this gap, we introduce MuseBench, a comprehensive benchmark designed to evaluate MLLMs on nuanced artistic understanding. It comprises 4,016 questions spanning cinematic arts, static visual arts, stage performing arts, and game arts, distilled from over 10K candidate video essays that pair professional commentary with visual demonstration. Comprehensive zero-shot evaluation of 28 state-of-the-art MLLMs reveals that even the best-performing model achieves only 48.29% accuracy, substantially below human expert performance of 87.18%.

Benchmark Overview

Construction Pipeline

MuseBench is constructed through three phases: (I) video collection and preprocessing with audio transcription, (II) question and answer annotation with adversarial distractor generation, and (III) human-in-the-loop iterative quality review.

Example Questions

Sample questions from each of the four art categories, illustrating single-select and multi-select formats with varying option counts.

Results

Zero-shot evaluation across 28 MLLMs. ACC = Accuracy (%), CAA = Chance-Adjusted Accuracy (%), EM = Exact Match (%). Per-category CAA shown for single-select questions.

Key Findings

F2: Game arts are a shared weakness

Across all model categories, game arts consistently yield the lowest scores, suggesting that game design reasoning (level design intent, mechanics-narrative interplay, aesthetic systems) is uniquely challenging for MLLMs.

F6: Open-source MLLMs exhibit pronounced first-position bias

Open-source models disproportionately favor answer options in the first position, regardless of correctness. This positional bias inflates accuracy on standard orderings and highlights a systematic vulnerability in multiple-choice reasoning.

BibTeX

If you find MuseBench useful for your research, please consider citing:

@article{fan2026musebench,
  title   = {MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMs},
  author  = {Fan, Yuxuan and Seo, Gyusik and Hao, Jing and Cho, Jaemin and Bansal, Mohit and Yoon, Jaehong},
  year    = {2026},
  journal = {arXiv preprint arXiv:2606.30026},
  url     = {https://arxiv.org/abs/2606.30026},
}