Researchers from Carnegie Mellon University are examining the challenge of a mediator coordinating a group of strategic agents without knowledge of their underlying utility functions, referred to as multi-agent imitation learning (MAIL). This is a complex issue as it involves providing personalised, strategic guidance to each agent without a comprehensive understanding of their circumstances or motives. Existing methodologies like behavioural cloning and inverse reinforcement learning have demonstrated limitations, revealing the need for a more refined approach.
The team proposes a fresh objective for MAIL in Markov Games called the regret gap, which considers potential deviations by agents. Their investigations reveal that minimising the value gap does not necessarily prevent the regret gap from escalating. This implies that achieving regret equivalence is a more complex challenge than just achieving value equivalence in MAIL.
In response to this issue, the team developed two reductions to no-regret online convex optimisation called MALICE (under a coverage assumption) and BLADES (with access to a queryable expert). Each of these approaches presents a different method to minimise the regret gap.
While the value gap is often considered a weaker objective, it can be viable in real-world applications where the agents are not strategic. The value gap can be efficiently minimised by generalising single-agent imitation learning algorithms to the multi-agent context.
Despite limitations in existing approaches, the team’s proposed methods to address the regret gap present promising directions for future research in improving multi-agent imitation learning. Future work will explore approximate practical variations of these idealised algorithms to further enhance efficiency and performance in real-world applications.
This research holds potential implications for a variety of real-world applications, including network routing and recommendation systems, showcasing the importance of continued developments in multi-agent imitation learning. By integratively handling both expert guidance and strategic deviations, these advancements could enable more effective decision-making and coordination among complex systems of agents.
