A benchmark for evaluating LLM agents on 120 real-world travel scene and e-commerce tool-use tasks in isolated Docker environments.
E-Commerce Last Exam evaluates how well LLM-based agents can solve real-world travel scene and e-commerce tasks using domain-specific CLI tools. Each of the 120 tasks runs inside an isolated Docker container pre-loaded with a SQLite database and purpose-built CLI tools (hotel search, flight booking, product comparison, etc.). Agents must interpret natural-language user requests, call the right tools in the right order, and produce a structured JSON answer scored 0.00–1.00 by an automated verifier. The benchmark covers two domains—travel (77 tasks across hotels, transport, and attractions) and e-commerce (43 tasks across shopping categories)—and is designed to be a reproducible, end-to-end evaluation of agent tool-use capabilities.
Python ≥ 3.10, Docker required
git clone https://github.com/alibaba-flyai/ecommerce_last_exam.git
cd ecommerce_last_exam
pip install flyai-bench
cp eval_config.yaml eval_config.local.yaml
Configure your LLM endpoint, then run
flyai-bench --config eval_config.local.yaml run \
--dataset-config travel \
--limit 5
Generate report and open a PR
flyai-bench --config eval_config.local.yaml report
flyai-bench --config eval_config.local.yaml submit \
--model your-model \
--provider your-provider