The Fin AIThe FinLLM Challenge
Teams built LLMs for three financial jobs: classifying arguments in earnings calls, summarizing financial news, and trading single stocks. Every task is built on FinBen.
Who entered what, and where they placed
Each row is a team; each cell is its final rank in that task. Gold, silver and bronze mark the podium. Hover a cell for every score.
Financial classification
Classify sentences from earnings-call transcripts as claims or premises. Ranked by F1. 7.75k training and 969 test examples.
Financial text summarization
Summarize financial news articles into concise texts. Ranked by ROUGE-1. 8k training and 2k test examples.
Single-stock trading
Make buy, sell or hold decisions, with reasoning, on DRIV, FORM, JNJ and MSFT. Ranked by Sharpe ratio: above zero means the strategy earned a positive risk-adjusted return. 291 data points.
Overall Sharpe ratio
Sharpe ratio by stock
Data Leakage Test (DLT)
To estimate whether a model saw the test set during training, DLT measures the perplexity gap between the training and test sets. A larger gap means leakage is less likely; a smaller one suggests it is more likely. Teams that reported it carry a DLT tag above.
Full results
Who ran the challenge
Author affiliations of the challenge overview paper, FinNLP-AgentScen-2024 Shared Task: Financial Challenges in Large Language Models (FinNLP-AgentScen @ IJCAI 2024).
About the challenge
The FinLLM Challenge (overview paper) ran as a shared task of the FinNLP-AgentScen workshop at IJCAI 2024. Tasks, data and metrics come from FinBen.
Keep evaluating
For ongoing evaluation of financial LLMs, see the Open FinLLM Leaderboard, which accepts new models.
Cite
@inproceedings{xie-etal-2024-finnlp,
title={FinNLP-AgentScen-2024 Shared Task: Financial Challenges in Large Language Models - FinLLMs},
author={Xie, Qianqian and Huang, Jimin and Li, Dong and Chen, Zhengyu and Xiang, Ruoyu and Xiao, Mengxi and Yu, Yangyang and Somasundaram, Vijayasai and Yang, Kailai and Yuan, Chenhan and Luo, Zheheng and Liu, Zhiwei and He, Yueru and Jiang, Yuechen and Li, Haohang and Feng, Duanyu and Liu, Xiao-Yang and Wang, Benyou and Wang, Hao and Lai, Yanzhao and Suchow, Jordan and Lopez-Lira, Alejandro and Peng, Min and Ananiadou, Sophia},
booktitle={Proceedings of the Eighth Financial Technology and Natural Language Processing and the 1st Agent AI for Scenario Planning},
year={2024}, address={Jeju, South Korea}, pages={119--126},
url={https://aclanthology.org/2024.finnlp-2.11/}
}
@misc{xie2024finbenholisticfinancialbenchmark,
title={FinBen: A Holistic Financial Benchmark for Large Language Models},
author={Qianqian Xie and Weiguang Han and Zhengyu Chen and Ruoyu Xiang and Xiao Zhang and Yueru He and Mengxi Xiao and Dong Li and Yongfu Dai and Duanyu Feng and Yijing Xu and Haoqiang Kang and Ziyan Kuang and Chenhan Yuan and Kailai Yang and Zheheng Luo and Tianlin Zhang and Zhiwei Liu and Guojun Xiong and Zhiyang Deng and Yuechen Jiang and Zhiyuan Yao and Haohang Li and Yangyang Yu and Gang Hu and Jiajia Huang and Xiao-Yang Liu and Alejandro Lopez-Lira and Benyou Wang and Yanzhao Lai and Hao Wang and Min Peng and Sophia Ananiadou and Jimin Huang},
year={2024}, eprint={2402.12659}, archivePrefix={arXiv}, primaryClass={cs.CL},
url={https://arxiv.org/abs/2402.12659}
}