Introduction to reinforcement learning

Reinforcement learning: the problem of how an agent figures out how it can act in an environment so as to maximize long-term reward
The simplest case: all reinforcements are immediate.
- Like supervised learning in the reward case
- In the punishment case, there is no information about what should have been done
The more realistic case: reinforcements happen only sometimes
- The temporal credit assignment problem: when was/were the action(s) taken that led to the reinforcement received?
- In extreme cases, reinforcement only at the "end" (board games like backgammon and chess)
Markov decision processes
- States and actions
  - The "real" state of the agent and what it "thinks" its state is
  - Actions take agents from one state to another.
  - In each state there is a set of actions to choose from.
- Discrete time
- Reinforcements
  - Rewards and punishments
  - Reinforcements depend only on the current state and the action taken.
  - The reinforcement function is built into the environment and is not known to the agent; it may be non-deterministic.
- The next-state function
  - Function of the current state and the action.
  - The next-state function is built into the environment and is not known to the agent; it may be non-deterministic.
  - The actual value of a state and what the agent thinks the value is
  - The actual value of an action in a given state and what the agent thinks the value is (Q-values)
- Optimal substructure and overlapping subproblems: dynamic programming
Policies
- A policy maps states into actions the agent ought to take in those states
- The goal of RL is to learn a policy that maximizes the sum of future reinforcements
- The problem of delayed reinforcement (the temporal credit assignment problem): how to choose an action in a state where there will be no reinforcement
Learning
- Long-term memory: the agent's record of what it has learned about the environment
- Learning: updating the long-term memory on the basis of new information from the environment
- The importance of learning in small steps, controlled by the learning rate
How the agent uses its knowledge
- Exploitation: relying on what's been learned
- Exploration: behaving stochastically; trial-and-error

Value iteration

Value iteration tries to find a value for each state: `V(s)`
- Value needs to be based on reinforcement the environment gives the agent when the agent reaches that state
- But sometimes there's no reinforcement; instead reaching a particular state may bring the agent closer to a state with a reinforcement.
- So the value needs to somehow take into account what next states are possible (that is, what the environment's next-state function returns with different actions as inputs).
The trick
- Take a state's value to be the sum of all future reinforcements received by the agent when it has reached that state and then follows its policy.
- The agent can't know the sum of all future reinforcemenets, so it has to estimate it.
- To estimate future reinforcements, it treats the value of a state recursively. The value is (a function of)
  - the immediate reinforcement received and
  - (what the agent currently believes to be) the value of the best state that can be reached next.
- At any given point during learning, the agent has an estimate of the values for states.
- On a given learning trial, the agent is in state `s_t`. It tries out different actions, for each summing:
  - The reinforcement received: `r(s_t, a_t)`
  - The current estimated value of the next state that it ends up in: `V(s_{t+1})`
  It then picks the best of these sums to update its estimated value for `s_t`.
Problem with using the bare next state value estimate
- Just adding the reinforcement to the next state value creates problems if future reinforcements are infinite (for example, if there's no obvious end to the series of actions)
- So usually it is the discounted best next state value that is added, that is, that estimate times a discount rate γ, ranging between 0 and 1.
- When γ is 0, only immediate reinforcement matters; when it is 1, all reinforcements are equally weighted.
- So we could update the value for a state pair using this equation:
  `V(s_t) = max_{a_t} [r(s_t, a_t) + gamma V(s_{t+1})]`
Learning rate
- But the information we get on a given learning trial may be misleading, so this update equation moves too quickly.
- Instead we usually move relatively slowly in the direction indicated by the current evidence; there is a learning rate (η), which controls the step size of the learning.
- With this, the update equation is:
  `V^{t+1}(s_t) = (1-eta) V^t(s_t) + eta [max_{a_t} [r(s_t, a_t) + gamma V^t(s_{t+1})]]`
  
  This equation combines the old knowledge that the agent has with the new information coming from the current experience of receiving a reinforcement.

Q learning

Value iteration has the problem that the agent has to try all possible actions in a state to find the best possible next state.
Q learning tries to find a value for each pair of a state (s) and an action (a).
- Each Q-value is a function of a state and an action: `Q(s, a)`.
- The Q-value for a given s, a pair needs to be based on the reinforcement that the environment gives the agent when the agent performs a in s (that is, what the environment's reinforcement function returns).
- But sometimes there's no reinforcement; instead an action may make a future reinforcement possible.
- So the Q-value also needs to somehow take into account what happens in the next state (that is, what the environment's next-state function returns).
The trick
- The basic idea: take the Q-value to be the sum of all future reinforcements received by the agent when taking that action and then following its policy.
- The agent can't know this, so it has to estimate it, treating the Q-value of a state-action pair recursively. The value is (a function of)
  - the immediate reinforcement received and
  - (what the agent currently believes to be) the value of the best action in the new state that is reached.
- So at any given point during learning, the agent has an estimate of the values for all state-action pairs. On a given learning trial, the agent is in state `s_t`. It does the following.
  - Selects an action `a_t` and executes it.
  - Notes whatever reinforcement it receives as a result: `r(s_t, a_t)`
  - Notes what new state it is in: `s_{t+1}`
  - Finds what it believes to be the value of the best action in the new state: `Q^{best}(s_{t+1}, a)`
  - Combines 2 and 4 to update its estimated value for the pair `s_t`, `a_t`
Discounting the next state estimate
- Just adding the reinforcement to the estimated next state value causes problems if the future reinforcements are infinite.
- Usually it is the discounted best value of the next state that is added, that is, that estimate times a discount rate γ,
- So we could update the Q-value for a state-action pair using this update equation:
  `Q(s_t, a_t) = r(s_t, a_t) + gamma max_{a_(t+1)} Q(s_{t+1}, a_{t+1})`
  The expression beginning with "max" means for all of the possible actions in the next state (a_t+1), the highest value of those that agent has stored for that state in its LTM.
Learning rate
- But the information we get on a given learning trial may be misleading, so this update equation moves too quickly.
- Instead we usually move relatively slowly in the direction indicated by the current evidence; that is, there is a learning rate (η), which controls the step size of the learning. With this, the update equation is:
  `Q^{t+1}(s_t, a_t) = (1-eta) Q^t(s_t,a_t) + eta [r(s_t, a_t) + gamma max_{a_(t+1)} Q^t(s_{t+1}, a_{t+1})]`
  This equation combines the old knowledge that the agent has with the new information coming from the current experience of receiving a reinforcement and ending up in a new state.