Heterogeneous Agent Collaborative RL: A New Paradigm for Efficient Multi-Agent Learning
A new research paper introduces Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a novel paradigm designed to overcome the sample inefficiency of isolated on-policy optimization. The core innovation enables collaborative optimization with independent execution, allowing heterogeneous agents to share verified training data to mutually improve their performance while maintaining full autonomy during deployment. This approach, formalized in the HACPO algorithm, demonstrates significant efficiency gains, outperforming prior methods like GSPO by an average of 3.3% while using only half the rollout cost.
Beyond Isolated Training and One-Way Distillation
Traditional multi-agent and distillation methods present clear limitations for heterogeneous systems. Large Language Model-based Multi-Agent Reinforcement Learning (MARL) often requires tightly coordinated deployment, which is impractical for independently operating agents. Meanwhile, standard on- or off-policy distillation techniques typically facilitate only one-directional knowledge transfer from a teacher to a student model. HACRL fundamentally rethinks this dynamic by enabling bidirectional mutual learning among diverse agents. During a centralized training phase, agents contribute and learn from a shared pool of verified experience rollouts, creating a synergistic learning environment without imposing any coordination requirements at inference time.
The HACPO Algorithm: Principled Sharing with Theoretical Guarantees
Building on the HACRL paradigm, the proposed HACPO algorithm provides a concrete framework for principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. The key challenge in such a system is managing the capability discrepancies and policy distribution shifts that naturally occur between heterogeneous agents. To address this, HACPO incorporates four tailored mechanisms backed by theoretical guarantees, ensuring unbiased advantage estimation and optimization correctness despite the non-stationary data sources. This rigorous foundation allows agents to safely and effectively learn from each other's experiences, accelerating collective improvement.
Empirical Validation Across Diverse Benchmarks
The efficacy of HACPO was validated through extensive experiments across a variety of heterogeneous model combinations and complex reasoning benchmarks. The results consistently showed that all participating agents improved their performance through the collaborative process. Notably, the system achieved higher performance—an average improvement of 3.3% over GSPO—while being dramatically more sample-efficient, requiring just 50% of the rollout cost. This demonstrates HACRL's potential to significantly reduce the computational burden of training advanced AI agents, making the development of capable heterogeneous systems more feasible.
Why This Matters for AI Development
- Unlocks Efficient Heterogeneous Systems: HACRL enables the practical development of teams where agents have different architectures or capabilities, allowing them to learn collaboratively without being deployed as a monolithic unit.
- Reduces Training Costs: By maximizing the utility of every sample through verified rollout sharing, the paradigm cuts the computational cost of on-policy RL, a major bottleneck in AI research.
- Enables Bidirectional Learning: It moves beyond simple teacher-student distillation, fostering a more democratic and potentially more powerful form of knowledge exchange between AI models.
- Maintains Deployment Flexibility: Agents trained collaboratively with HACRL can execute independently, offering greater flexibility for real-world applications compared to coordinated MARL systems.