Researchers from MIT and other institutions have developed a method that prevents large AI language machines from crashing during lengthy dialogues. The solution, known as StreamingLLM, tweaks the key-value cache (a sort of conversation memory) of large language models to ensure the first few data pieces remain in memory. Typically, once the cache’s capacity is exceeded, the oldest data is bumped out, resulting in potential failure of the model. However, the implementation of StreamingLLM ensures the chatbot remains functional even when the conversation extends beyond 4 million words.
The concept of StreamingLLM revolves around the discovery that the first token, or ‘attention sink,’ is crucial to the model’s performance. If this token remains, the model will continue to perform efficiently, even if the cache size is exceeded. An attention sink is a token used by certain models in their attention mechanism, which allocates a score to each token, signifying its relevancy to other tokens. The first token often retains any spare attention score, which results in its significance to the model’s performance.
The researchers also determined that maintaining the positional encoding of each token, regardless of new tokens being added and older ones being bumped out, contributed to optimal performance. For instance, if token 5 is bumped out, token 6 should retain its encoding as 6, even though it is now the fifth token in the cache.
StreamingLLM outperformed other popular methods due to its convenience and efficiency. At a cache size of 4,096 tokens, the recomputation method takes 1,411 milliseconds to code a new token, whereas StreamingLLM requires just 65 milliseconds.
IT has the potential to transform AI applications due to its stability and performance. Streaming LLM has been implemented in NVIDIA’s large language model optimization library, TensorRT-LLM and has been used successfully on iPhones for the extension of large language model conversation lengths.
Despite its efficiency, StreamingLLM has a limitation in that the model cannot recall words not stored in its cache. Looking ahead, the researchers aim to overcome this by exploring ways to retrieve evicted tokens or enable the model to memorize previous dialogues.
This work has been partly funded by the MIT-IBM Watson AI Lab, the MIT Science Hub, and the U.S. National Science Foundation and will be presented at the International Conference on Learning Representations. This research has the potential to revolutionize AI-driven generation applications, making them more reliable and efficient.
