As in any RL approach, the reward function plays a large role in determining what policy is learned. One possible reward function is based entirely upon reaching the ultimate goal. Although goals scored are the true rewards in this domain, such events are very sparse. In order to increase the feedback from actions taken, it is useful to use an internal reinforcement function, which provides feedback based on intermediate states towards the goal. Without exploring the space of possible such functions, we created one reward function R.
R gives rewards for goals scored. However, players also receive rewards if the ball goes out of bounds, or else after a fixed period of time based on the ball's average lateral position on the field. In particular, when a player takes action in state s such that , the player records the time t at which the action was taken as well as the x coordinate of the ball's position at time t, . The reward function R takes as input the observed ball position over time (a subset of ) and outputs a reward r. Since the ball position over time depends also on other agents' actions, the reward is stochastic and non-stationary. Under the following conditions, the player fixes the reward r:
In case 1, the reward r is based on the value as indicated in Figure 1(b): . Thus, the farther in the future the ball goes out of bounds (i.e. the larger ), the smaller the absolute value of r. This scaling by time is akin to the discount factor used in Q-learning. We use and .
In cases 2 and 3, the reward r is based on the average x-position of the ball over the time t to the time or . Over that entire time span, the player samples the x-coordinate of the ball at fixed, periodic intervals and computes the average over the times at which the ball position is known. Then if , where is the x-coordinate of the opponent goal (the right goal in Figure 1(b)). Otherwise, if , where is the x-coordinate of the learner's goal. Thus, the reward is the fraction of the available field by which the ball was advanced, on average, over the time-period in question. Note that a backwards pass can lead to positive reward if the ball then moves forward in the near future.
Figure 1: (a) The black and white dots represent the players attacking the right and left goals respectively. Arrows indicate a single player's action options when in possession of the ball. The player kicks the ball towards a fixed set of markers around the field, including the corner flags and the goals. (b) The component of the reward function R based on the circumstances under which the ball went out of bounds. For kick-ins, the reward varies linearly with the x position of the ball.
The reward r is based on direct environmental feedback. It is a domain-dependent internal reinforcement function based upon heuristic knowledge of progress towards the goal. Notice that it relies solely upon the player's own impression of the environment. If it fails to notice the ball's position for a period of time, the internal reward is affected. However, players can track the ball much more easily than they can deduce the internal states of other players as they would have to do were they to determine future team state transitions.
As teammates learn concurrently, the concept to be learned by each individual agent changes over time. We address this problem by gradually increasing exploitation as opposed to exploration in all teammates and by using a learning rate (see Equation 1). Thus, even though we are averaging several reward values for taking an action in a given state, each new example accounts for 2% of the updated Q-value: rewards gained while teammates were acting more randomly are weighted less heavily.