Artificial Intelligence’s ability to comprehend and generate natural language effectively has been a real mystery in the field of machine learning. The system’s capability to memorize and combine knowledge fragments has eluded traditional machine learning techniques until now. This paper explores the intriguing process through a new approach named “Memory Mosaics,” promising a better understanding and the potential for more refined, transparent AI systems.
Memory Mosaics are a learning system architecture with multiple associative memories cooperating to carry out prediction tasks. An associative memory is a device that stores key-value pairs as vectors. When given a query key, the retrieval process determines the conditional probability distribution based on the stored pairs and issues the conditional expectation as the predicted value. Gaussian kernel smoothing computes this conditional expectation, a technique akin to classical attention mechanisms.
While Memory Mosaics and transformer models share similarities, distinct differences set them apart. In a Memory Mosaics model, there’s no position encoding involved, keys and queries aren’t distinguished separately, and the prediction targets are explicitly represented via value extraction functions. The training process optimizes these key-value extraction functions, enabling each associative memory unit to specialize efficiently.
The “predictive disentanglement” concept used by Memory Mosaics allows for the breaking down of the overall prediction task into smaller, independent components that are allocated to individual associative memory units. This way of disentangling the problem makes predictions more efficient.
With a layered memory approach, Memory Mosaics employ certain memory units at a contextual level, memorizing and predicting based on the immediate input. Other units serve as persistent memories, retaining knowledge gathered from the training process itself. This replaces the attention heads and feed-forward networks found in other models with the persistent and contextual memory units.
Extensive evaluation showed that Memory Mosaics performed similarly to, if not better than, traditional transformer architectures in language modeling tasks. When applied to out-of-distribution data, they exhibited superior in-context learning abilities. They also proved effective when tested on the RegBench benchmark, which gauges a model’s capability to learn artificial languages.
Despite these promising results, further research is needed to scale these findings to larger Memory Mosaics models. Despite this, the approach offers an insightful look at compositional learning systems, shedding light on how AI understands and generates language by fragmenting and combining information.
