The evaluation of Large Language Models (LLMs) requires a systematic and multi-layered approach to accurately identify areas of improvement and limitations. As these models advance and become more intricate, their assessment presents greater challenges due to the diversity of tasks they are required to execute. Current benchmarks often employ non-precise, simplistic criteria such as “helpfulness” and “harmlessness.” They usually focus on specific tasks like instruction following, creating an incomplete assessment of overall model performance.
To resolve these difficulties, researchers have created a comprehensive and ethical benchmark known as BIGGEN BENCH. This tool is designed to evaluate nine different capabilities across 77 tasks, presenting a more precise and in-depth assessment of language models. The capabilities tested are: instruction following, grounding, planning, reasoning, refinement, safety, theory of mind, tool usage, and multilingualism.
A unique feature of BIGGEN BENCH is its use of instance-specific evaluation criteria, mimicking the complex, context-sensitive judgements made by humans. Rather than allotting a single score for helpfulness, the benchmark evaluates the model on its performance on specific tasks, such as explaining mathematical concepts or performing translation duties with cultural nuances in mind. With such specificity, BIGGEN BENCH can identify minute differences in model performance which conventional benchmarks may overlook.
This benchmark was used to evaluate 103 cutting-edge LMs, which included 14 proprietary models. The parameters of these models range between 1 billion to 141 billion, and the evaluation involved five different LMs to ensure reliable results.
The researchers outlined their main contributions to be as follows: they described in detail the process of building and evaluating BIGGEN BENCH, highlighting the use of a human-in-the-loop technique; they reported evaluation results for 103 LMs showing that fine-grained assessment consistently improved performance as model size scaled, while also showing that gaps in reasoning and tool usage abilities persist; furthermore, they demonstrated the reliability of their assessments by comparing evaluator scores with human evaluations, finding significant correlations for all abilities, and investigated methods to improve open-source evaluator LMs to match the performance of GPT-4 models.
In summary, the newly developed BIGGEN BENCH enables a more comprehensive, finer-grained evaluation of language models. It tests on a wider range of tasks and uses context-specific criteria for a more nuanced approach. It has already been used to evaluate over 100 models and has obtained robust and reliable results.
