Making decisions

How is the agent to pick an action?
The exploration-exploitation tradeoff revisited
- Exploitation: pick the action that has the highest Q-value for the current state
- But the agent can only learn about the value of actions that it tries; it should try a variety of actions, including actions other than the ones it thinks are best: exploration.
- Exploration makes more sense early on in learning when the agent doesn't know much; the probability of exploration should decrease with experience.
One simple possibility for selecting an action
- Pick the "best" action with probability `P_{exploit} = 1 - e^{E*A}`, where A is the number of training samples (the "age" of the agent).
- With E = 0.1 and E = 0.01,
A smarter possibility
- Have the probability of picking an action depend on how high its value is relative to the values of all of the other possible actions, that is, to use the Luce choice rule.
- One way:
  `P(a_t^i) = (e^{E * A * Q(s_t, a_t^i)}) / (sum_a e^{E * A * Q(s_t,a_t)})`
- But as learning progresses, the variance among the Q-values should increase, increasing the probability of exploitation, even if not a function of "age".

Implementing reinforcement learning

Lookup table
- As the number of possible states increases, the likelihood of a given state actually occurring in a sequence of trials decreases.
- The agent knows nothing about states that it has never experienced.
Neural network
- States can be represented in a distributed fashion on the input layer of a neural network; each unit represents a state feature.
- Value iteration
  - The output layer represents the value of the input state, normally with a single unit but possibly also in a distributed fashion (as in Neurogammon).
  - For a state vector `vec s_t` and a selected action `a_t`, we can calculate a target for the output unit.
  - It's just the estimate of the value for `vec s_t` based on the best combination of reinforcement received and value of the next state for the different actions in `vec s_t`:
    `max_{a_t} [r(vec s_t, a_t) + gamma V(vec s_{t+1})]`
  - To calculate the right-hand term, we have to run the neural network with state `vec s_{t+1}` and read off the value on the output unit(s).
  - We can use the delta rule, or backpropagation if the network has a hidden layer, to update the weights.
  - The learning rate built into these rules replaces the η term in the update rule for a lookup table.
  - Using gradient descent, the weights into the output layer are updated according to this rule (assuming one output unit):
    `Delta w_{ji,t} = eta [ max_{a_t} [r(vec s_t, a_t) + gamma V(vec s_{t+1})] - V(vec s_t)] (del V(vec s_t))/ (del w_{ji})`
  - The partial derivative in the learning rule depends on the activation function of the output unit(s). For linear units, it is just `s_i`, that is, the activation of the ith input unit.
- Q learning
  - The output layer has a unit for each action (a localist representation).
  - Presented with a state vector on the input layer `vec s`, the activation of output unit j represents the network's current estimate of `Q(vec s, a_j)`.
  - For a state vector `vec s_t` and a selected action `a_t`, we can calculate a target for the output unit that corresponds to `a_t`.
  - It's just the estimate of the relevant Q-value, given the new reinforcement received and the best estimate of the other Q-values:
    `r(vec s_t, a_t) + gamma max_{a_(t+1)} Q(vec s_{t+1}, a_{t+1})`
  - To calculate the right-hand term, we have to run the neural network with state `vec s_{t+1}` and find the highest activation in the output layer.
  - As with value iteration, we can use the delta rule or backpropagation to update the weights. The learning rule is:
    `Delta w_{ji,t} = eta [ [r(vec s_t, a_t) + gamma max_{a_(t+1)} Q(vec s_{t+1}, a_{t+1})] - Q(vec s_{t}, a_t) ] (del Q(vec s_t, a_t))/ (del w_{ji})`
  - As with value iteration, the partial derivative in the learning rule depends on the activation function of the output unit(s). For linear units, it is `s_i`, that is, the activation of the ith input unit.