Code generation is a critical domain for assessing and employing Large Language Models (LLMs). However, numerous existing coding benchmarks, such as HumanEval and MBPP, have reached solution rates over 90%, indicating the requirement for more challenging benchmarks. These would underline the limitations of current models and suggest ways to improve their algorithmic reasoning capabilities.
Competitive programming serves as a valuable means to evaluate the creation of unique algorithms and human reasoning under challenging situations. However, there hasn’t been sufficient problem diversity, comprehensive problem analyses, or unit tests for competitive programming evaluation.
In response to these issues, researchers have introduced USACO, a benchmark consisting of 307 difficult tasks from past USA Computing Olympiad contests. Each task requires a broad spectrum of algorithmic, mathematical, and common-sense skills, and creative and well-founded thinking to solve it. Unlike earlier benchmarks focusing on programming synthesis, models must now be able to reason across various settings and create original algorithms for each task to succeed in USACO.
The benchmark also includes official analysis, reference code solutions, high-quality unit tests, and educational materials, helping to investigate more inferencing techniques for competitive programming. Different baseline techniques based on self-reflection and retrieval and their combinations have been created using these resources. These strategies combined with self-reflection significantly boost performance, tripling GPT-4’s zero-shot solve rate. However, none of the techniques manage to solve the benchmark beyond the easiest level (bronze difficulty tier).
The study also implemented a human-in-the-loop model to gain in-depth understanding into the remaining challenges. This model suggests that providing GPT-4 with tailored suggestions helps it solve 13 out of 15 previously unsolvable problems, outperforming all prior models and methods examined.
Main contributions of this study include:
– Introduction of the USACO benchmark, the first created from Olympiad programming, equipped with carefully selected test cases, problem analysis, and additional resources for comprehensive analyses.
– Designing and analyzing LLM inference techniques specifically for Olympiad programming tasks.
– Contrary to automated tests simply considering execution success, this study evaluates the potential and boundaries of LLMs for Olympiad programming tasks, showing that only a subset of models can efficiently integrate feedback.
Overall, while combined retrieval and self-reflection strategies have shown promise in enhancing performance, a significant gap remains in fully answering the benchmark. However, their introduction and analysis have furthered understanding of LLMs’ potential and limitations in handling challenging Olympiad programming tasks.
