AI Paper Summary

Graph Neural Networks (GNNs) are essential for processing complex data structures in domains such as e-commerce and social networks. However, as graph data volume increases, existing systems struggle to efficiently handle data that exceed memory capacity. This warrants out-of-core solutions where data resides on disk. Yet, such systems have faced challenges balancing speed of data…

Large language models (LLMs) have introduced ground-breaking advancements to the field of natural language processing, such as improved machine translation, question-answering, and text generation. Yet, training these complex models poses significant challenges, including high resource requirements and lengthy training times. Former methods addressing these concerns involved loss-scaling and mixed-precision strategies, which aimed to further training efficiency…

Training large-scale Generative AI models can be challenging due to the immense computational resources and time they require. This complexity gives rise to frequent instabilities, manifested as disruptive loss spikes during prolonged training periods. These instabilities can result in costly interruptions, requiring the training process to be paused and restarted. For example, the LLaMA2's 70-billion…