The delta rule and backpropagation

Activation functions

The simplest activation function is the identity function: the activation is just the input.
Another possibility, used in a perceptron, is a threshold function which converts inputs above some threshold to a maximum value (usually 1.0) and inputs below the threshold to a minimum value (usually -1.0 or 0.0).
A third possibility is a "soft" threshold function which smooths out the region near the threshold. The following function, the sigmoid, has a minimum of 0.0 and a maximum of 1.0 (note that these are never reached):
g(h) = 1 / (1 + e^-h)
(h is the input to the unit.)

The delta rule

Supervised learning (pattern association) in feedforward networks with multiple output units and continuous activation functions
The delta rule (least mean squares rule) for supervised learning when the activation function of the output unit is the identity function
Δw_ji = η (t_j - x_j) x_i (1)
(The xs represent activations, t represents a target, and η is a learning rate between 0.0 and 1.0.)
The Perceptron Learning Rule is a special case of this rule (t_j and x_j are either 0 or 1, so a fraction of the input pattern is either added to or subtracted from the weight).
A formal way to derive the delta rule for the more general case, when the activation function is a differentiable function
- Gradient descent learning: learning by moving in the direction which looks locally to be the best
- For supervised neural network learning, the best direction to move in "weight space": for each weight, how the global error changes with respect to that weight
- A global error function: for each pattern the sum of the errors over all of the output units
  E = ∑_j ½ (t_j - x_j)²
- We want to move in "weight space" in a direction which is opposite that of the slope of the error function with respect to each weight because this will move us toward a region with a lower error. The size of the move should be proportional to the magnitude of the slope.
  1. To find the slope, we take the partial derivative of the error with respect to the weight. But the only element in the sum of error terms that depends on the weight is the one for the output unit where that weight ends (j in what follows).
    ∂E/∂w_ji = ∂ [½ (t_j - x_j)²] / ∂w_ji
  2. Using the chain rule, we can decompose this derivative into two that are easier to calculate:
    (∂[½ (t_j - x_j)²]/∂x_j) (∂x_j/∂w_ji)
  3. The first derivative is easy to figure; it's just
    -(t_j - x_j)
  4. The second derivative can be decomposed using the chain rule again if we remember that the activation of unit j is a function of the input to the unit, h_j, which is in turn a function of the weights into the unit.
    ∂x_j/∂w_ji = (∂x_j/∂h_j) (∂h_j/∂w_ji)
  5. Since the activation of an output unit is the activation function g applied to the input h, the first derivative on the right-hand side of (4) is just
    g′(h_j)
    that is, the derivative of whatever the activation function is at the value of the current input to unit j.
  6. The second derivative on the right-hand side of (4) can be derived as follows:
    ∂h_j/∂w_ji = ∂(∑_kx_kw_jk)/∂w_ji = x_i
    because none of the other weights or input activations depend on w_ji.
  7. Putting all of the parts together, we get
    ∂E/∂w_ji = -(t_j - x_j) g′(h_j) x_i
  8. Remember that we want the weight change to be proportional to the negative of the derivative with respect to the weight. So with a learning rate to control the step size for weight changes, we get the more general delta (least mean squares) learning rule
    Δw_ji = η (t_j - x_j) g′(h_j) x_i (2)
- For the identity activation function, the derivative is 1, so (1) is just a special case of (2)

Backpropagation

Problems that are not linearly separable can't be solved by a perceptron (or a network learning with the delta rule).
How hidden units can solve non-linearly separable problems.
Backpropagation: a gradient descent algorithm for learning the weights into hidden units as well as output units
A network with hidden units with linear activation functions is equivalent to a network with no hidden units, at least the hidden units must have non-linear activation functions, and these must be differentiable for backpropagation to apply: usually the sigmoid function.
The learning rule:
Δw_ji = ηδ_jx_i
- For output units:
  δ_j = (t_j - x_j) g′(h_j)
- For hidden units (k indexes the units in the next highest layer):
  δ_j = [∑_k δ_k w_kj] g′(h_j)
A famous example: NETTALK, the text-to-speech problem
Some questions and concerns
- Does BP get stuck in local minima?
- Does it take forever to learn the weights?
  - Faster as number of hidden units increases (assuming parallel update)
  - Faster with higher learning rate, within limits
- How does the network solve the problem? What sort of hidden-layer representations does it build? Using statistical techniques to analyze hidden-layer representations.
- Does it generalize? Does the network behave appropriately on patterns which it has not been trained on?
  - More local and more distributed (greater generalization) hidden-layer patterns
  - Effect of too many trainable connections: overfitting, the network "memorizes" individual patterns rather than generalizing over them
Optimization: setting the learning rate, other parameters
Incremental training: learning a simpler task which enables the learning of a more complex task
Multiple tasks in a single network
- Catastrophic forgetting: does the network unlearn one set of patterns when trained on a second?
- Does the network fail to learn two interfering tasks which it is trained on simultaneously? Example: the what-where vision problem
- Modularity as a solution to problems of interference

Sequential problems and simple recurrent networks

Sequence processing: inputs consist of sequences of patterns; the network's output depends on previous patterns as well as the current one
- Prediction: given a partial sequence, predict the next element (pattern)
- Sequence classification
- Parsing
Sequence processing requires some form of short-term memory (in addition to unit activations)
Simple recurrent network (Elman net): recurrent connections on the hidden layer with a time delay of one sequence event, usually implemented with a context layer that maintains a copy of the hidden-unit activations on the last time step
Training an SRN on prediction
- The input and output layers represent a single sequence event.
- During training, sequences of inputs are presented repeatedly.
- On a single training trial, an event is presented to the input layer, and the network is run in the usual fashion, with the context layer treated as another input layer.
- The target is the next event in the sequence. Error is back-propagated, and weights are updated using the backpropagation rule, with context-to-hidden weights treated exactly as input-to-hidden weights.
- Finally the activations on the hidden layer are copied to the context layer.