Language Model

Language models (LMs) such as BERT or GPT-2 are faced with challenges in self-supervised learning due to a phenomenon referred to as representation degeneration. These models work by training neural networks using token sequences to generate contextual representations, with a language modeling head, often a linear layer with variable parameters, producing next-token distributions of probability.…

Due to the need for long-sequence support in large language models (LLMs), a solution to the problematic key-value (KV) cache bottleneck needs addressing. LLMs like GPT-4, Gemini, and LWM are becoming increasingly prominent in apps such as chatbots and financial analysis, but the substantial memory footprint of the KV cache and their auto-regressive nature make…

MLCommons, a joint venture of industry and academia, has built a collaborative platform to improve AI safety, efficiency, and accountability. The MLCommons AI Safety Working Group established in late 2023 focuses on creating benchmarks for evaluating AI safety, tracking its progress, and encouraging safety enhancements. Its members, with diverse expertise in technical AI, policy, and…

Researchers at UT Austin have developed an effective and efficient method for training smaller language models (LM). Called "Inheritune," the method borrows transformer blocks from larger language models and trains the smaller model on a minuscule fraction of the original training data, resulting in a language model with 1.5 billion parameters using just 1 billion…

Scaling up language learning models (LLMs) involves substantial computational power and the need for high-density datasets. Language models typically make use of billions of parameters and are trained using datasets that contain trillions of tokens, making the process resource-intensive. A group of researchers from the University of Texas at Austin have found a solution. They’ve…

Code generation is a critical domain for assessing and employing Large Language Models (LLMs). However, numerous existing coding benchmarks, such as HumanEval and MBPP, have reached solution rates over 90%, indicating the requirement for more challenging benchmarks. These would underline the limitations of current models and suggest ways to improve their algorithmic reasoning capabilities. Competitive programming…