Metadata-Version: 2.4
Name: aigve
Version: 0.0.1
Summary: A comprehensive and structured evaluation framework for assessing AI-generated video quality.
Home-page: https://www.aigve.org
Download-URL: https://github.com/ShaneXiangH/AIGVE_Tool
Author: Xinhao Xiang, Xiao Liu, Zizhong Li, Zhuosheng Liu
Author-email: xhxiang@ucdavis.edu
License: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch==2.1.0
Requires-Dist: torchvision==0.16.0
Requires-Dist: torchaudio==0.12.1
Requires-Dist: numpy==1.23.5
Requires-Dist: scipy==1.14.1
Requires-Dist: pandas==2.2.3
Requires-Dist: pyyaml==6.0.2
Requires-Dist: tqdm==4.67.1
Requires-Dist: matplotlib==3.10.0
Requires-Dist: mmcv==2.2.0
Requires-Dist: mmdet==3.3.0
Requires-Dist: mmengine==0.10.6
Requires-Dist: transformers==4.46.3
Requires-Dist: sentencepiece==0.2.0
Requires-Dist: einops==0.8.1
Requires-Dist: openai==1.65.1
Requires-Dist: gdown==5.2.0
Requires-Dist: fvcore==0.1.5.post20221221
Requires-Dist: decord==0.6.0
Requires-Dist: pytorchvideo==0.1.5
Requires-Dist: clip==1.0
Requires-Dist: mkdocs-material==9.6.5
Requires-Dist: neoteroi-mkdocs==1.1.0
Requires-Dist: mkdocs-macros-plugin==1.3.7
Requires-Dist: mkdocs-jupyter==0.25.1
Requires-Dist: mkdocstrings==0.28.2
Requires-Dist: mkdocs-rss-plugin==1.17.1
Requires-Dist: mkdocs-exclude==1.0.2
Requires-Dist: mkdocstrings-python==1.16.2
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: download-url
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# AIGVE-Tool
AI Generated Video Evaluation toolkit


## Implemented:

### Models:
#### Video-Only Neural Network-Based evaluation metrics:
1. [GSTVQA](.aigve/configs/gstvqa.py) 
2. [SimpleVQA](.aigve/configs/simplevqa.py) 
3. [LightVQA_Plus](.aigve/configs/lightvqa_plus.py)


#### Distribuition Comparison-Based evaluation metrics:
These metrics primarily assess the quality of generated samples by comparing distributions of real and generated data:
1. [FID](.aigve/configs/fid.py)
2. [FVD](.aigve/configs/fvd.py)
3. [IS](.aigve/configs/is_score.py)

#### Vision-Language Similarity-Based evaluation metrics:
These metrics primarily evaluate alignment, similarity, and coherence between visual and textual representations. They focus on how well images and text match, often using embeddings from models like CLIP and BLIP:
1. [CLIPSim](.aigve/configs/clipsim.py) 
2. [CLIPTemp](.aigve/configs/cliptemp.py) 
3. [BLIP](.aigve/configs/blipsim.py)
4. [Pickscore](.aigve/configs/pickscore.py)

#### Vision-Language Understanding-Based evaluation metrics:
These metrics assess higher-level understanding, reasoning, and factual consistency in vision-language models. They go beyond similarity, evaluating semantic correctness, factual alignment, and structured comprehension:
1. [VIEScore](.aigve/configs/viescore.py) 
2. [TIFA](.aigve/configs/tifa.py)
3. [DSG](.aigve/configs/dsg.py)


#### Multi-Faceted evaluation metrics
These metrics are structured, multi-dimensional evaluation metrics designed to assess AI models across diverse sub-evaluation dimensions. They provide a comprehensive benchmarking framework that integrates aspects like video understanding, physics-based reasoning, and modular evaluation, enabling a more holistic assessment of model performance.
1. [VideoPhy](.aigve/configs/videophy.py)
2. [VideoScore](.aigve/configs/viescore.py)

### Dataset:
1. [Toy dataset](.aigve/data/toy) 
2. [AIGVE-Bench](.aigve/data/AIGVE_Bench_Toy)



## Environment

conda env remove --name aigve
```
conda env create -f environment.yml
conda activate aigve
conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=11.8 -c pytorch -c nvidia
```
(MMCV from v1.7.2 support PyTorch 2.1.0 and 2.0.0)

## Run:
``
python main.py {metric_config_file}.py
``

Take Examples:

```
rm -rf ~/.cache
```

For GSTVQA:
``
cd VQA_Toolkit/aigve
python main_aigve.py AIGVE_Tool/aigve/configs/gstvqa.py --work-dir ./output
``

For SimpleVQA:
``
cd VQA_Toolkit/aigve
python main_aigve.py AIGVE_Tool/aigve/configs/simplevqa.py --work-dir ./output
``

For LightVQAPlus:
``
cd VQA_Toolkit/aigve
python main_aigve.py AIGVE_Tool/aigve/configs/lightvqa_plus.py --work-dir ./output

``

For GSTVQACrossData:
``
cd VQA_Toolkit/aigve
python main_aigve.py AIGVE_Tool/aigve/configs/gstvqa_crossdata.py --work-dir ./output
``

For CLIPSim:
``
cd VQA_Toolkit/aigve
python main_aigve.py AIGVE_Tool/aigve/configs/clipsim.py --work-dir ./output
``

For VideoPhy:
``
cd VQA_Toolkit/aigve
python main_aigve.py AIGVE_Tool/aigve/configs/clipsim.py --work-dir ./output
``


## Acknowledge

The Toolkit is build top the top of [MMEngine](https://github.com/open-mmlab/mmengine)

We acknowledge original repositories of various VQA methods:
[GSTVQA](https://github.com/Baoliang93/GSTVQA),
[CLIPSim](https://github.com/zhengxu-1997/),
<!-- [ModularBVQA](https://github.com/winwinwenwen77/ModularBVQA), -->
<!-- [StarVQA](https://github.com/GZHU-DVL/StarVQA) -->
