Leveraging Structure to Represent Tasks in Sequential Decision Making

Abstract

In this thesis, I explore three structured modeling approaches for representing tasks in sequential decision making problems. Each approach leverages task structure to represent complex objects (goals and dynamics) as the sum of smaller, more tractable parts. This decomposition offers steps toward tractable task definitions, goal aggregation, and out-of-distribution exploration and generalization. The contributions are organized into three themes:

Axiomatic Task Specification: From a set of basic axioms, I derive two fundamental results about the structure of scalar reward functions. The first suggests that general purpose reward structures should allow for functional time preference. The second asserts that multi-objective agents may require non-Markovian rewards to precisely specify composed Markovian objectives. Causal Task Structures: I present a novel approach to causal modeling that uses locally independent causal mechanisms to provably generalize out-of-distribution, even in the absence of global independence. I apply this to data augmentation in model-free and model-based reinforcement learning, which enables an offline reinforcement learning agent to solve a never-before-seen task. Task Space Geometry: I present two methods that use goal space geometry to structure tasks for multi-goal reinforcement learning agents. The first uses a goal space embedding to choose exploration goals to maximize the entropy of an agent’s achievable goal distribution, enabling efficient exploration. The second represents the goal space as an asymmetric metric space, with a view toward goal interpolation. While general purpose alignment is a hard problem that may ultimately prove intractable, this thesis argues that leveraging task structure, as exemplified by the three angles above, offers viable modes of progress toward a solution.