Machine learning

Reinforcement Learning (RL) involves learning decision-making through interactions with an environment and has been used effectively in games, robotics, and autonomous systems. RL agents aim to maximize their results and increase their efficiency by improving performance through continually adapting to new data. However, the RL agent's sample inefficiency impedes its practical application by necessitating comprehensive…

Researchers from various institutions have recently unveiled a unique linear property of transformer decoders in natural language processing models such as GPT, LLaMA, OPT, and BLOOM. This discovery could have significant implications for future advancements in the field. These researchers discovered that there is a nearly perfect linear relationship in the embedding transformations between sequential…

Managing large language models (LLMs) often entails dealing with issues related to the size of key-value (KV) cache, given that it scales with both the sequence length and the batch size. While techniques have been employed to reduce the KV cache size, such as Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), they have only managed…

MIT researchers have developed a method known as Cross-Layer Attention (CLA) to alleviate the memory footprint bottleneck of the key-value (KV) cache in large language models (LLMs). As more applications demand longer input sequences, the KV cache's memory requirements limit batch sizes and necessitate costly offloading techniques. Additionally, persistently storing and retrieving KV caches to…