Graphs play a critical role in providing a visual representation of complex relationships in various arenas like social networks, knowledge graphs, and molecular discovery. They have rich topological structures and nodes often have textual features that offer vital context. Graph Machine Learning (Graph ML), particularly Graph Neural Networks (GNNs), have become increasingly influential in effectively modeling such data, leveraging deep learning’s message-passing mechanism to encapsulate high-order relationships. Large Language Models (LLMs) have been increasingly incorporated within GNNs to tackle a variety of graph tasks and augment generalization competencies via self-supervised learning methodologies. This marriage of technologies demonstrates the vast potential of Graph ML and necessitates a comprehensive review of recent advancements in the field.
The evolution of graph learning has seen a transition from nascent methods like random walks and graph embedding, which facilitated node representation learning while preserving graph topology, to more refined techniques powered by deep learning. The introduction of GNNs has advanced graph learning significantly, with techniques like Graph Convolutional Networks (GCNs) and Graph Attention Networks (GATs) being developed to enhance node representation and concentrate on key nodes. Simultaneously, the emergence of LLMs has spurred advancements in graph learning, with models such as GraphGPT and GLEM using advanced language model techniques to efficiently understand and manipulate graph structures.
While Foundation Models (FMs) have had a transformative impact on the Natural Language Processing (NLP) and vision domains in the broader AI landscape, the development of Graph Foundation Models (GFMs) is still a work in progress. This presents an opportunity for further explorations to advance the capabilities of Graph ML.
A recent survey conducted by researchers from the Hong Kong Polytechnic University, Wuhan University, and North Carolina State University offers a comprehensive review of Graph ML in the era of LLMs. The study delves into the progression from early graph learning methods to the latest GFMs, thoroughly evaluates current LLM-enhanced Graph ML methods, and explores the potentialities of graph structures to address the limitations of LLMs. The researchers also examine the applications and prospective future directions of Graph ML and discuss research and practical applications in various fields.
While GNNs-based Graph ML faces certain obstacles, such as the requirement for labeled data and the existence of shallow text embeddings preventing semantic extraction, LLMs offer a promising solution. They are capable of handling the complexity of natural language, allowing for zero/few-shot predictions and providing unified feature spaces. The researchers explored how LLMs can elevate Graph ML by improving feature quality, aligning feature space, and tapping into their extensive parameter volume and abundant open-world knowledge.
However, despite these advancements, LLMs face difficulties in operational efficiency when processing large and complex graphs. The use of APIs, such as GPT4, can be costly, and deployment of large open-source models, like LLaMa, demand substantial computational resources and storage. Recent studies suggest applying novel techniques like LoRA and QLoRa for more resource-efficient parameter fine-tuning, and model pruning to eliminate redundant parameters or structures.
In summary, the survey presents a coherent picture of the progression of graph learning methods and analyzes the current LLM-enhanced Graph ML techniques. It acknowledges existing challenges with operational efficiency, but also highlights the potential solutions like parameter fine-tuning and model pruning that could help move the field forward. The fundamental inference is that the integration of LLMs with Graph ML, despite the challenges, displays a significant promise for continual advancement in the field.
