ChartAnno: Evaluating MLLMs for Chart Annotation Generation

Zhenghan Chen*1Zekai Shao*1Lidan Tan1Xin Lin2Xingchen Zeng3Yi Shan1Ziyue Lin1 Xiaoliang Fu1Xinyuan Liu1Yuetong Guo1Fen Wang1Bongshin Lee†4Siming Chen†1

1Fudan University 2Sun Yat-sen University 3HKUST (Guangzhou) 4Yonsei University

*Equal contribution †Co-corresponding authors

Real-world annotated charts: arrows, highlights, and text labels explain data, emphasize key findings, and guide attention, motivating MLLM-assisted annotation authoring.
Figure 1. Motivating example of chart annotation generation. Given an existing chart and an annotation instruction, an MLLM generates executable code to add annotations that communicate the intended message.

Abstract

Annotations are essential to communicative visualization, helping explain data, emphasize key findings, and guide attention. While multimodal large language models (MLLMs) offer new opportunities for automatic chart annotation authoring, their capabilities in this task remain underexplored. To address this gap, we introduce ChartAnno, a comprehensive benchmark for evaluating MLLMs on chart annotation generation. ChartAnno contains 1,200 real-world charts with paired annotated and unannotated executable code, along with 3,600 annotation instructions spanning three levels of specificity. We also develop a multidimensional evaluation framework combining rule-based and LLM-judged metrics to assess execution, structural compliance, semantic consistency, and design effectiveness. We evaluate 10 representative MLLMs under two primary chart input settings: (1) chart code alone and (2) both code and chart image. Results reveal that proprietary models lead overall, though open-source models narrow the gap. While higher instruction specificity improves annotation quality, inferring abstract communicative intent remains difficult across all models. Providing chart images yields marginal benefit when code is available. We also examine the effect of chart code through an image-only ablation and analyze the effects of multiple task complexity indicators and instruction-level transitions. Further analyses characterize common failure modes and validate the reliability of the LLM-based judge. Experiments with D3 and SVG demonstrate the generalizability of ChartAnno beyond its primary Python setting.

Benchmark

Task design of ChartAnno: three instruction levels (Intent, Operation, Implementation), two input settings (Code, Code + Image), model-generated code rendered into annotated charts, and multidimensional evaluation.
Figure 2. Task design of ChartAnno. Primary chart inputs are Code or Code + Chart Image, with annotation instructions ranging from abstract Intent through Operation to concrete Implementation. For each chart input–instruction combination, the MLLM generates annotated code and its rendered chart, evaluated with reference to the unannotated code, instruction, and annotated ground truth.

Leaderboard

Results of 10 MLLMs under three input settings across three instruction levels. Exec., Struct., Sem., and Design denote Execution Rate, Structural Compliance, Semantic Consistency, and Design Effectiveness. Cell shading scales with the score within each column; best results are bold. Click a column header to sort, and use the buttons above the table to filter by model group.

Main results of 10 MLLMs on ChartAnno under two input settings and three instruction levels
Model Intent-level Operation-level Implementation-level
Exec.Struct.*Sem.Design Exec.Struct.Sem.Design Exec.Struct.Sem.Design

*Struct. reports Chart Fidelity for Intent-level instructions. Sem. and Design are judged on a 1–5 scale; Exec. and Struct. are scores in [0, 1].

Citation

@misc{chen2026chartannoevaluatingmllmschart,
      title={ChartAnno: Evaluating MLLMs for Chart Annotation Generation},
      author={Zhenghan Chen and Zekai Shao and Lidan Tan and Xin Lin and Xingchen Zeng and Yi Shan and Ziyue Lin and Xiaoliang Fu and Xinyuan Liu and Yuetong Guo and Fen Wang and Bongshin Lee and Siming Chen},
      year={2026},
      eprint={2608.03464},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2608.03464}
}

Contact Us

If you have any questions about ChartAnno, feel free to reach out to us at chenzh26@m.fudan.edu.cn or zkshao23@m.fudan.edu.cn.