Announcement of Open Japanese LLM Leaderboard

We release the Open Japanese LLM Leaderboard, covering a range of traditional to contemporary NLP tasks aimed at evaluating and analyzing Japanese LLMs. The Open Japanese LLM Leaderboard was created by open-source contributors of LLM-jp in partnership with researchers at Hugging Face.

For the Open Japanese LLM Leaderboard, submitted LLMs are automatically deployed using Hugging Face’s inference endpoints and evaluated with the llm-jp-eval library. This evaluation process utilizes a memory-efficient inference and serving engine, vLLM. We are able to compute all the evaluation scores on the mdx platform.

Open Japanese LLM Leaderboard: https://huggingface.co/spaces/llm-jp/open-japanese-llm-leaderboard

Hugging Face Blog Article: https://huggingface.co/blog/leaderboard-japanese