Language models (LMs) such as BERT or GPT-2 are faced with challenges in self-supervised learning due to a phenomenon referred to as representation degeneration. These models work by training neural networks using token sequences to generate contextual representations, with a language modeling head, often a linear layer with variable parameters, producing next-token distributions of probability.…
Due to the need for long-sequence support in large language models (LLMs), a solution to the problematic key-value (KV) cache bottleneck needs addressing. LLMs like GPT-4, Gemini, and LWM are becoming increasingly prominent in apps such as chatbots and financial analysis, but the substantial memory footprint of the KV cache and their auto-regressive nature make…
MLCommons, a joint venture of industry and academia, has built a collaborative platform to improve AI safety, efficiency, and accountability. The MLCommons AI Safety Working Group established in late 2023 focuses on creating benchmarks for evaluating AI safety, tracking its progress, and encouraging safety enhancements. Its members, with diverse expertise in technical AI, policy, and…
Machine learning is the driving force behind data-driven, adaptive, and increasingly intelligent products and platforms. Algorithms of artificial intelligence (AI) systems, such as Content Recommender Systems (CRS), intertwine with users and content creators, in turn shaping viewer preferences and the available content on these platforms.
However, the current design and evaluation methodologies of these AI systems…
Exploring the interactions between reinforcement learning (RL) and large language models (LLMs) sheds light on an exciting area of computational linguistics. These models, largely enhanced by human feedback, show remarkable prowess in understanding and generating text that mirrors human conversation. Yet, they are always evolving to capture more subtle human preferences. The main challenge lies…
Researchers at UT Austin have developed an effective and efficient method for training smaller language models (LM). Called "Inheritune," the method borrows transformer blocks from larger language models and trains the smaller model on a minuscule fraction of the original training data, resulting in a language model with 1.5 billion parameters using just 1 billion…
Scaling up language learning models (LLMs) involves substantial computational power and the need for high-density datasets. Language models typically make use of billions of parameters and are trained using datasets that contain trillions of tokens, making the process resource-intensive.
A group of researchers from the University of Texas at Austin have found a solution. They’ve…
Code generation is a critical domain for assessing and employing Large Language Models (LLMs). However, numerous existing coding benchmarks, such as HumanEval and MBPP, have reached solution rates over 90%, indicating the requirement for more challenging benchmarks. These would underline the limitations of current models and suggest ways to improve their algorithmic reasoning capabilities.
Competitive programming…