The artificial intelligence (AI) industry has seen many advancements, particularly in the area of game-playing agents such as AlphaGo, which are capable of superhuman performance via self-play techniques. Now, researchers from the University of California, Berkeley, have turned to these techniques to tackle a persistent challenge in AI—improving performance in cooperative or partially cooperative language tasks.
The researchers have introduced an innovative approach that uses self-play in both cooperative and competitive environments. They modified the semi-competitive negotiation game Deal or No Deal (DoND) to enable various objectives. This game was adapted to support different collaboration levels and function as a versatile testbed for AI training. The game could simulate a range of environments—fully cooperative, semi-competitive, and strictly competitive—by altering the reward structure.
The game involves two players who negotiate the division of items with private value functions. The researchers used filtered behavior cloning for self-play training and played 500 games per round over ten rounds. Initial models, which included GPT-3.5 and GPT-4, were evaluated without the use of so-called few-shot examples to avoid bias. To validate the model’s effectiveness, human experiments were conducted on Amazon Mechanical Turk with pre-screened workers.
The results of the self-play training showed considerable improvements. The model achieved up to 2.5 times better scores in cooperative settings and six times better scores in semi-competitive settings compared to initial benchmarks. However, the study found that performance improvements were minimal in strictly competitive environments, highlighting the difficulty of applying self-play in zero-sum scenarios.
In conclusion, the study found that self-play has potential for training language models in collaborative tasks. The results are significant because they challenge the prevailing assumption that self-play is only useful in competitive environments, or that models need vast amounts of human data to maintain language interpretability. The improvements observed after just ten rounds of self-play suggest that models with good generalization abilities can benefit from this approach. These findings pave the way for broader applications of self-play beyond competitive games, potentially improving AI’s function in various collaborative and real-world endeavors.
