TD(λ)

The temporal credit assignment problem revisited: in standard reinforcement learning, passing reinforcement information back to previous states (or state-action pairs) requires many time steps.
How TD(λ) solves it
- When a state value in value iteration changes as a result of new information, we can pass this information back to values updated on the way to this state. When a Q value in Q learning changes as a result of new information, we can pass this information back to Q values updated on the way to this state.
- We can control how much of the change is passed back with a parameter, λ, that varies between 0 and 1.

Value iteration

The ordinary learning rule for a neural network:
`Delta w_{ji,t} = eta [ max_{a_t} [r(vec s_t, a_t) + gamma V(vec s_{t+1})] - V(vec s_t)] (del V(vec s_t))/ (del w_{ji})`
The TD(λ) learning rule is
`Delta w_{ji,t} = eta [ max_{a_t} [r(vec s_t, a_t) + gamma V(vec s_{t+1})] - V(vec s_t)] sum_{k=1}^t [lambda^{t-k} (del V(vec s_k))/ (del w_{ji})]`
The lambda term, Λ can be calculated incrementally (only a single value is stored for each weight):
`Lambda_{t+1} = sum_{k=1}^{t+1} [lambda^{t+1-k} (del V(vec s_k))/ (del w_{ji})]`
`= (del V(vec s_{t+1}))/ (del w_{ji}) + sum_{k=1}^{t} [lambda^{t+1-k} (del V(vec s_k))/ (del w_{ji})] = (del V(vec s_{t+1}))/ (del w_{ji}) + lambda sum_{k=1}^{t} [lambda^{t-k} (del V(vec s_k))/ (del w_{ji})]`
`= (del V(vec s_{t+1}))/ (del w_{ji}) + lambda Lambda_t`

Example

Ordinary value iteration, γ = .9, η = 1.0, linear output unit

t	s	x₁	x₂	x₃	w₂	w₃	V	max[r+γV_t+1]	a
1	1	1	0	0	0	0	0	0	right
2	2	0	1	0	0	0	0	0	right
3	3	0	0	1	0	0	0	1	right
4	G	0	0	0	0	1.0	0	0	stay
1	1	1	0	0	0	1.0	0	0	right
2	2	0	1	0	0	1.0	0	.9	right
3	3	0	0	1	.9	1.0	1	1	right
4	G	0	0	0	.9	1.0	0	0	stay

TD(lambda) value iteration, λ = .9

t	s	x₁	x₂	x₃	w₁	w₂	w₃	max[r+γV_t+1]	∑V'₁	∑V'₂	∑V'₃	a
1	1	1	0	0	0	0	0	0	1	0	0	right
2	2	0	1	0	0	0	0	0	.9	1	0	right
3	3	0	0	1	0	0	0	1	.81	.9	1	right
4	G	0	0	0	.81	.9	1.0	0	.72	.81	.9	stay

The calculation of the update for `w_1` at time 3:
`Delta w_{1,3} = eta [ max_{a_3} [r(vec s_3, a_3) + gamma V(vec s_4)] - V(vec s_3)] sum_{k=1}^3 [lambda^{3-k} (del V(vec s_k))/ (del w_1)]`
`= 1.0 ((1 + 0.9*0) - 0.0)(.81*1.0 + 0.9*0.0 + 1.0*0.0) = 0.81`

TD-Gammon (Tesauro): a program that learns to play tournament-level backgammon using TD(λ) value iteration while playing against itself

TD(λ): Q-learning

The ordinary rule for a neural network:
`Delta w_{ji,t} = eta [ (r(vec s_t, a_t) + gamma max_{a_(t+1)} Q(vec s_{t+1}, a_{t+1})) - Q(vec s_{t}, a_t) ] (del Q(vec s_t, a_t))/ (del w_{ji})`
The TD(λ) rule
`Delta w_{ji,t} = eta [ (r(vec s_t, a_t) + gamma max_{a_(t+1)} Q(vec s_{t+1}, a_{t+1})) - Q(vec s_{t}, a_t) ] sum_{k=1}^t [lambda^{t-k} (del Q(vec s_k, a_k))/ (del w_{ji})]`
An example from biomorph; λ = .5, η = .1

Perceptual aliasing

With very large numbers of world states, it is not feasible to have the learner distinguish all of them in its model states
Perceptual aliasing arises when two or more world states correspond to a single model states
Perceptual aliasing is good when it treats different world states that don't need to be distinguished as the same, that is, when it handles invariance
Perceptual aliasing is bad when different actions in the model state result in different outcomes: reinforcements and next states.
One solution: figuring out which model states are "bad" by keeping track of the variance of the outcomes (reinforcements and next states)
- Learning to avoid particular states; there must be alternate paths to the goal
- Using perceptual actions to control states: attention; a single world state may correspond to more than one model state
Another solution: using sequences of "instances" (previous instance, state, action, reinforcement tuples) as states
- Storing all previous instances in a history sequence, comparing the current subsequence to similar subsequences in the history
- But how long do the subsequences need to be?
- Utile Suffix Memory (McCallum, 1995) answers this question
  - Store the instances in a "suffix tree" with leaves representing states (sequences of percepts and actions) and a "fringe" of nodes below the leaves representing possible ways of extending the lengths of the sequences (and adding new states)
  - A Q-value is stored (and updated) for each suffix tree leaf
  - To add a new level to the tree from the fringe, compare the nodes in the fringe with the node that was just updated (had a new instance added) for whether their predicted future reinforcements (reinforcement + discounted best value for next state) for the same action differ. If they do, make the fringe nodes state nodes and extend the fringe.
  - Example: room navigation with perceptual aliasing
    - Reinforcements: G: 5, bumping into barrier: -1, otherwise: -.1. Four possible actions.
    - The agent needs to distinguish two different 5 states (requiring a one-step memory) and two different 10 states (requiring a two-step memory). The final suffix tree:
    - A sequence of instances:
    - Extending the suffix tree:
    - Actions from state 10 would have high variance for predicted future reinforcement (because E and W moves have opposite effects for different instances of the state).
    - But actions from different sequences of states followed by actions followed by state 10 would have less variance and different distributions from each other (2-E-10 is different from 2-W-10, for example).
    - These differences cause the suffix tree to be extended one level so that state 10 is now replaces by states 2-E-10, 2-W-10, etc.