Large Language Models (LLMs) have shown significant impact across various tasks within the software engineering space. Leveraging extensive open-source code datasets and Github-enabled models like CodeLlama, ChatGPT, and Codex, they can generate code and documentation, translate between programming languages, write unit tests, and identify and rectify bugs. AlphaCode is a pre-trained model that can help optimize source code across several languages.
The researchers from Meta AI and Rice University introduced a unique approach within compiler optimization, utilizing LLMs with compiler feedback. These models aim to optimize the code size of LLVM (Low-Level Virtual Machine) assembly. The approach has been used previously in machine learning-driven code optimization research. Despite various methods, including manual feature crafting to sophisticated graph neural networks, they encountered common limitations—namely, an incomplete representation of the input program offered to the machine learning model, leading to a loss of vital information.
The approach proposed by researchers from Meta AI and Rice University overcomes this limitation. They use the LLM to fine-tune LLVM optimization, which has shown impressive code reasoning abilities. The method involves an unoptimized LLVM Intermediate Representation (IR) input, producing optimized IR, the best optimization passes, and counting instructions for both unoptimized and optimized IRs.
The input then undergoes compilation according to the generated optimization passes. It’s evaluated to check if the model-predicted instruction count is accurate, if the IR is compilable, and if it corresponds to the compiled code. This feedback is relayed back to the LLM for further optimization.
The next step is to assess the model generation consistency via the compiler, providing feedback regarding the pass list accuracy, and if the predicted instruction counts are correct and computed. The bleu score, which measures the similarity of the generated code to the compiler-generated code, is obtained, and this comparison is also incorporated in the feedback.
Early results demonstrate that the proposed model surpasses the original by 0.53%, reducing the discrepancy to the autotuner by 10%. When sampling is in operation, the original model can achieve up to 98% of the autotuner’s performance with 100 samples.
In summary, this unique approach of compiler-generated feedback offered to Large Language Models aims to optimize code size. The system incorporates feedback on the model’s initial generation, instructing the LLM to make further improvements. All forms of feedback have proven to outperform the original model.
Credit for this research goes to the researchers involved in the project. The original paper and further discussion can be accessed through the content body.
