Metadata-Version: 2.4
Name: music-benchmark
Version: 0.0.2
Summary: Music benchmark library
Author-email: zouyee <zouyiiy@sina.com>
License: MIT License
        
        Copyright (c) 2026 zouyiiy@sina.com
        
        Permission is hereby granted, free of charge...
        
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: music21>=9.9.1
Requires-Dist: pydub>=0.25.1
Requires-Dist: cairosvg>=2.9.0
Requires-Dist: torch==2.6.0
Requires-Dist: transformers>=4.57.6
Dynamic: license-file


# Holistic Music Benchmark System: HY-MAST
Official codes and tools for paper: HY-MAST: A Holistic Benchmark for Evaluating Multimodal Language Models on Music Understanding

HY-MAST is a holistic music benchmark system capable of automatically generating comprehensive music understanding tasks and multimodal musical content.
It enables comprehensive evaluation of the music understanding performance of large language models (LLMs).
Its web-based application can be employed for fundamental music assessment intended for human musicians.


## Question Type
HY-MAST can automatically generate a rich variety of question types, thereby comprehensively evaluating the multidimensional music understanding capabilities of LLMs.
The automatically generated question types include:

Pitch Identification, Interval Identification, Chord Identification, Rhythm Transcription,
Tempo Estimation, Tonality Identification, Meter Identification, Harmonic Analysis, Instrument Identification, 
Genre Identification, Lyric Transcription, Melody Transcription, Work Identification

## Music Content and sources
For its LLM-oriented evaluation framework, HY-MAST provides audio music, symbolic sheet music in ABC notation, and music theory articles.
Audio music is derived from two sources: automatically synthesized standard music samples and authentic human-composed works.

### Online Automatic synthesis and AI Melody Generation
HY-MAST incorporates a complete set of tools for the automatic synthesis of standard samples. 
Guided by the rigorous norms of musicological theory, it can automatically synthesize accurate standard music samples (No AI) in accordance with specified instructions.

- Automatic synthesis  
Currently, this project supports the automatic synthesis of the following standard samples:  
single note, interval notes, chord, scale, rhythm  
Automatic synthesis tools:
Auto_Benchmark_Test_for_LLM_main.py  
Main function: randomly_make_questions_with_online_synthesis

- AI Melody Generation and Automatic Audio Rendering  
For the questions of Melody Transcription and Time Signature Recognition tasks, the target melody to be recognized is generated by an LM. The LM model first produces content in ABC notation, which is then automatically converted to MIDI and rendered into WAV format, before being finally supplied to the question system.  
The LM model and inference framework are saved in the folder: melody_generation

- sheet music images  
For the web application designed for human assessment, HY-MAST has added staff notation recognition tasks and automatically generates corresponding sheet music images based on the questions.  
Tools at music21_save_picture.py

### Real Music
We carefully select a large number of human-composed musical works. 
The labels and information for these music pieces are stored in the file:
real_audio/metadata.jsonl

Real-music resources are freely available for download. 
Their categories and corresponding online sources are listed as follows:
1. Pop Music  
https://www.billboard.com/charts/greatest-hot-100-singles  
save folder: real_audio/pop/audio/  
2. Classical Music  
https://www.kaggle.com/datasets/imsparsh/musicnet-dataset  
save folder: real_audio/classical/audio/  
3. World Music  
3.1. Turkey  
https://zenodo.org/records/1283349  
save folder: real_audio/folk/turkey/audio/  
3.2. Arab  
https://zenodo.org/records/1291776  
save folder: real_audio/folk/arab/audio/  
3.3. China  
https://zenodo.org/records/344932  
save folder: real_audio/folk/china/audio/  
3.4 India  
https://zenodo.org/records/4301737  
save folder: real_audio/folk/india/audio/  

Please download the music files from the above websites and save them to the designated folders.  

## Install and Usage
### requirement
1. Python library  
music21  
pydub  
cairosvg  
torch==2.6.0  
transformers==4.57.6  
2. Conda library  
fluidsynth  
Recommended installation method:  
***conda install -c conda-forge fluidsynth***

### Usage
1. Main program  
Run the main program directly to generate the complete Benchmark:  
python Auto_MultiModal_Music_Benchmark_Main.py --seed 0 --number_of_processes 2 --each_question_number 4  
each_question_number: Number of questions per category  
number_of_processes: Number of processes  
Optional parameters:  
--music_data_dir, type=str, default="real_audio", help="Path of the real music dataset folder. Please provide the absolute path.  
Attention! Each_question_number must be divisible by number_of_processes.

2. Tools  
Automatic fundamental perception questions generation tools:  
Auto_Benchmark_Test_for_LLM_main.py
Automatic synthesis tools:  
Algorithm_for_Music_Synthesize.py  
Batch generated abc to wav tools:  
lm_gen_abc_to_audio.py  
pseudorandom tools:  
pseudorandom_tools.py  
