Co-Evolutionary Proximal Distilled Evolutionary Reinforcement Learning with Gated Knowledge Transfer

Evolutionary reinforcement learning (ERL) offers a compelling alternative for continuous control by combining the population-level exploration of evolutionary algorithms with the gradient-based exploitation of reinforcement learning. However, applying conventional genetic operators to deep networks can be highly destructive, often inducing abrupt behavioral shifts that erase previously learned skills. Proximal distilled evolutionary reinforcement learning (PDERL) addresses this issue with phenotype-aware operators, leveraging proximal mutation and distillation crossover to produce safer and more constructive variations. Despite these advances, PDERL and many ERL frameworks still exhibit a fundamental evaluation asymmetry: an evolving actor population is guided by a single, centralized critic for fitness evaluation and action filtering. This single-critic dependence creates a bottleneck and a potential single point of failure, where bias or instability in value estimation can misdirect the evolutionary search. To overcome this limitation, we propose co-evolutionary proximal distilled evolutionary reinforcement learning (Co-PDERL), a heterogeneous dual-population framework that co-evolves both actor and critic populations. Co-PDERL extends phenotype-aware evolution to the value-function landscape via a loss-filtered distillation crossover and a Jacobian-based proximal mutation tailored for critics, and employs a condition-gated synchronization mechanism to enable robust bidirectional knowledge transfer between the evolutionary populations and the reinforcement learning agent. Experiments on MuJoCo continuous control benchmarks show that Co-PDERL outperforms competitive baselines on most tasks, including standard ERL and PDERL, improving both sample efficiency and asymptotic performance by effectively alleviating the single-critic bottleneck.