The original hierarchical learning constructs were devised to improve the generalization of a single learning task by running multiple learning processes. Both boosting [Shapire, 1990] and stacked generalization [Wolpert, 1992] improve function generalization by combining the results of several generalizers or several runs of the same generalizer. These approaches contrast with layered learning in that the layers in layered learning each deal with different tasks. Boosting or stacked generalization could potentially be used within any given layer, but not across different layers.
More in line with the type of hierarchical learning discussed in this paper are hierarchical reinforcement learning algorithms. Because of the well-known ``curse of dimensionality'' in reinforcement learning RL researchers have been very interested in hierarchical learning approaches. As surveyed in [Kaelbling et al., 1996], most hierarchical RL approaches use gated behaviors:
There is a collection of behaviors that map environment states into low-level actions and a gating function that decides, based on the state of the environment, which behavior's actions should be switched through and actually executed. [Kaelbling et al., 1996]In some cases the behaviors are learned [Mahadevan & Connell, 1991], in some cases the gating function is learned [Maes & Brooks, 1990], and in some cases both are learned [Lin, 1993]. In this last example, the behaviors are learned and fixed prior to learning the gating function. On the other hand, feudal Q-learning [Dayan & Hinton, 1993] and the MAXQ algorithm [Dietterich, 1998] learn at all levels of the hierarchy simultaneously. A constant among these approaches is that the behaviors and the gating function are all control tasks with similar inputs and actions (sometimes abstracted). In the RL layer of our layered learning implementation, the input representation itself is learned. In addition, none of the above methods has been implemented in a large-scale, complex domain.
In all of the above RL approaches, like in layered learning, the task decomposition is constructed manually. However, there has been at least one attempt at the challenging task of learning the task decomposition. Nested Q-learning [Digney, 1996] generates its own hierarchical control structure and then learns low-level skills at the same time as it learns to select among them. Thus far, like other hierarchical RL approaches, it has only been tested on very small problems (on the order of 100 states in this case).