Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill\n Discovery

Reinforcement learning requires manual specification of a reward function to\nlearn a task. While in principle this reward function only needs to specify the\ntask goal, in practice reinforcement learning can be very time-consuming or\neven infeasible unless the reward function is shaped so as to provide a smooth\ngradient towards a successful outcome. This shaping is difficult to specify by\nhand, particularly when the task is learned from raw observations, such as\nimages. In this paper, we study how we can automatically learn dynamical\ndistances: a measure of the expected number of time steps to reach a given goal\nstate from any other state. These dynamical distances can be used to provide\nwell-shaped reward functions for reaching new goals, making it possible to\nlearn complex tasks efficiently. We show that dynamical distances can be used\nin a semi-supervised regime, where unsupervised interaction with the\nenvironment is used to learn the dynamical distances, while a small amount of\npreference supervision is used to determine the task goal, without any manually\nengineered reward function or goal examples. We evaluate our method both on a\nreal-world robot and in simulation. We show that our method can learn to turn a\nvalve with a real-world 9-DoF hand, using raw image observations and just ten\npreference labels, without any other supervision. Videos of the learned skills\ncan be found on the project website:\nhttps://sites.google.com/view/dynamical-distance-learning.\n