The simplest architecture for supervised pattern classification
Architecture
Input units + bias (threshold) unit
Binary output unit; each output unit a separate perceptron
Input and activation rules
`y = delta (sum_(i=1)^N w_ix_i + b)`
`delta(x) = {(1,if x>0),(0,text(otherwise)):}`
That is, the activation function is a simple linear threshold function.
Learning
For each input pattern p, there are three cases:
The pattern is classified correctly.
In this case, no changes are made to the weights.
The target is 1, but the network yielded 0.
In this case, we need to change each weight so that the output will be higher.
We can achieve this by adding the input vector (or a fraction of the input vector, the
learning rate η)
to the weight vector (the superscript represents the particular pattern).
`Delta vec w = eta vec x^p`
The target is 0, but the network yielded 1.
In this case, we need to change each weight so that the output will be lower.
We can achieve this by subtracting the input vector (or a fraction of the input vector)
to the weight vector.
`Delta vec w = -eta vec x^p`
The three cases can all be expressed with this general rule:
`Delta vec w = eta vec x^p(t^p - y^p)`
The Perceptron Convergence Theorem (Rosenblatt)
Assume that there is a set of weights that solves the problem.
Then there is weight vector `vec w text(*)` that never yields a sum
lying in a region around 0 of width `2 epsilon`.
That is, for inputs that are supposed to yield a positive output,
`vec w text(*)` yields an output greater than `epsilon`,
and for inputs that are supposed to yield a negative output, `vec w text(*)` yields an output less than `-epsilon`.
Show that the angle between the current weight vector and `vec w text(*)`
lies within an envelope that shrinks whenever the weights are updated.
Here is the proof for the case where the target for a given input pattern vector
`vec x^p` is 1 and the network's output is too low
(the proof is similar for the case where target is 0 and the output is too high).
Assume a learning rate of 1.0.
We'd like to show that the cosine of the angle between `vec w text(*)` and
the current set of weights gets larger as the pattern is repeatedly presented.
From the formula for the cosine between two vectors (n is the number
of presentations of the pattern),
`cos theta = (vec w text(*) * vec w_n) / (|\|vec w text(*)|\| |\|vec w_n|\|)` [1]
Now we'll look for an upper bound for the denominator and a lower bound for the numerator of this fraction.
Because for input `vec x^p`, the output was too low:
`vec w_(n-1) * vec x^p le 0`
By the perceptron learning rule,
`vec w_n = vec w_(n-1) + vec x^p`
Then the numerator of [1] is
`vec w text(*) * vec w_n =`
`vec w text(*) * (vec w_(n-1) + vec x^p) =`
`vec w text(*) * vec w_(n-1) + vec w text(*) * vec x^p`
Because of our assumptions about `vec w text(*)` and `vec x^p`,
`vec w text(*) * vec x^p > epsilon`
Because the target for input vector `vec x^p` is 1, on each previous presentation,
either the weights didn't change or the input vector was added to the weight vector,
`vec w text(*) * vec w_n`
`> vec w text(*) * vec w_(n-1) + epsilon`
`ge vec w text(*) * vec w_(n-2) + 2 epsilon`
`ge vec w text(*) * vec w_0 + n epsilon`
Assuming the initial weights were all 0,
`vec w text(*) * vec w_n > n epsilon` [2]
So far we have a lower bound for the numerator of [1].
For the denominator, first consider the square of the length of `vec w_n`.
Since all input values are between 0 and 1, the square of the length of any
input vector cannot be greater than the total number of input units N.
`|\|vec x^p|\|^2 le N`
So now we have an upper bound on the denominator of [1]:
`|\|vec w text(*)|\| |\|vec w_n|\| le |\|vec w text(*)|\| sqrt n sqrt N` [3]
Putting [2] and [3] together,
`cos theta > (sqrt n epsilon)/(|\|vec w text(*)|\| sqrt N)`
As the pattern is presented, only n changes, so the angle must decrease.
The theorem does not guarantee that the angle between the
current and final weight vectors will decrease monotonically, only that
the envelope within which this angle is found decreases with the number
of updates.
That is, the error may sometimes rise for a given run through the training
patterns.
The theorem only guarantees convergence if there is a set of weights that solves
the problem.
Perceptrons: what they can and can't do
One input
The input patterns fall along a line; all points on one side of a given
value are in, the other outside the category.
The trainable bias establishes the threshold, the sign of the single
weight the direction of category membership on either side of the
threshold.
If the points in the category are broken up by points not in the category, there
is no way for the network to solve the problem.
Two inputs
The two weights and the bias define a line
`w_1x_1 + w_2x_2 + b = 0`
with slope `-w_1/w_2` and y-intercept `-b/(w_2)`.
Points on one side of the line turn on the output unit; points on the
other side turn it off.
Because the perceptron defines an inequality (two possibilities for each
line), three values are required rather than the two required to define a
line.
The line defined by the weights and bias divides the input space
into two regions.
A perceptron in 2-space can only learn to separate two sets of points that
are on either side of a line.
In general, a perceptron can only solve a pattern classification problem
if the sets of patterns are linearly separable;
that is, if in N-space, there is a hyperplane of N-1 dimensions which separates
the two sets of points.
N values (N-1 weights and the trainable bias) are needed to specify
the desired behavior because in addition to the hyperplane, we need
to say on which side of the hyperplane points are in the category.
Problems which perceptrons can't solve
Examples of non-linearly separable sets of patterns
Exclusive OR: (0, 0), (1, 1); (1, 0), (0, 1)
Connectivity
Solving the problems with additional input dimensions, for example, for
exclusive OR an input unit that codes for whether the other two inputs are the same
Solving the problems with hidden units
Networks with hidden units with linear activation functions
are equivalent to networks without hidden units
Hidden units must have non-linear activation functions,
for example, a simple threshold function (like perceptron output
units) or sigmoidal or gaussian functions.
One possibility: provide "enough" (non-linear) hidden units connected
by random, hard-wired weights to the input units: example
Another possibility: train the input-to-hidden weights.
But how?
