Supervised constructivist networks: shrinking

Getting rid of weights
- Weight decay: penalizes large weights
  - Penalty added to cost function
    `E = E_0 + half epsilon sum_i sum_j w_(ij)^2`
  - Weight update; in addition to the usual update, do:
    `w_(ij)^(text(new)) = (1 - epsilon) w_(ij)^(text(old))`
- Weight elimination (Weigend, Rumelhart, Huberman, 1991)
  - Minimum description length (Rissanen, 1978)
    - Models are preferred which maximally compress the data
    - Compression is formalized in terms of description length: the sum of the description of the code and the description of the data in terms of the code
    - For a neural network, description length can be formalized as the sum of network complexity and error.
  - Penalty added to cost function, representing network complexity
    `E = E_0 + half lambda sum_i sum_j (w_(ij) / w_0)^2 / (1 + (w_(ij) / w_0)^2)`
    Relatively large w₀ results in a preference for many small weights; relatively small w₀ a preference for fewer large weights.
  - Learning starts with `lambda = 0` so that the network can use all of its resources.
  - Applied to time series analysis, prevents overfitting that results from networks
Getting rid of units
- Relatively useless units are those with small outputs or small incoming weights
- Penalize these units by increasing weight decay into them

Supervised constructivist networks: growing

Cascade Correlation (Fahlman and Lebiere, 1991)
One reason back-propagation is so slow: the moving target problem
Attempt to solve the task, adding hidden units as needed, each time doing as well as possible with the current set before adding more.
Begin with no hidden units.
Train the network using a single-layer learning rule.
- The learning algorithm is usually quickprop (Fahlman, 1988).
- Quickprop embellishes back-propagation with information about the second derivative of the error with respect to the weights, allowing the algorithm to estimate the appropriate step size to reach a local error minimum.
When no significant error reduction has occurred for some number of training cycles, measure error over training set.
If satisfied, stop. If not, create-unit, and again train all weights to output layer.
Create-unit
- Create a new hidden unit with a set of random weights from input units and existing hidden units (or only the input layer and lower hidden layers in sibling-descendant CC)
- Train these weights (again a single layer) to maximize the correlation between the unit's activation and the error over the output units for each pattern.
- Install the unit in the network, freezing its trained connections.
A pool of candidate hidden units, which may differ in activation function, may also compete with each other for a spot in the network ("tenure"). The candidate units may be trained in parallel. After training, the candidate unit with the highest correlation is selected to join the network.
Requires only a forward pass because weights are trained only between adjacent layers
Test: two spirals problem
Exhibits fast learning, good generalization, incremental learning, and complex, higher-order feature detectors
Recurrent cascade correlation (Fahlman, 1991)
- Complete recurrent connections within the hidden units would mean that "frozen" hidden units would get new trained connections as new units are added.
- Instead there are only self-recurrent connections, created when a unit's initial connections are created and trained in the correlation phase.
- Test: finite-state grammar
Knowledge-based cascade correlation (Shultz)
- In addition to new hidden units, new hidden networks may be added to the growing network.
- Test: primality testing
  - Primality testing of an integer `n` can decomposed into tests for divisibility by prime numbers up to `sqrt n`.
  - Subnetworks trained on divisibility tests were recruited as hidden modules by a successful KBCC network.
  - KBCC faster and superior in generalization to simpler CC networks.

Competitive constructivist networks

ART
Constructive competitive learning (Gasser, 2004)
- Add a new output unit when none of the current units is closer to the input pattern than a threshold μ.
- Decrease μ during training.
- Remove output units that do not win within some number of training patterns ζ.

Constructivist self-organizing maps

Goal: a topology-preserving map
- Similar input patterns should be mapped onto neighboring or identical map units.
- Neighboring map units should have similar input patterns mapped onto them.
Problems with SOMs
- Map configuration and size must be specified beforehand
- Decaying σ and η
Three constructivist models (Fritzke, 1996)
- Growing Cell Structures (Fritzke)
  - Maintain error totals for all winning map units.
  - Use separate step size for winning unit and its immediate neighbors.
  - Every λ input presentations, create a new map unit by splitting the longest connection out of the unit with the highest error and adding a new unit, and decrease the error totals of neighbors of the new unit.
  - Decrease the error totals of all units, to emphasize recent error.
- Competitive Hebbian learning (Martinetz and Schulten)
  - Start with a set of unconnected output units.
  - For each input pattern find the two closest units, and if they are not already connected, create a connection between them.
  - Can be combined with an algorithm that moves the weight vectors too: "topology-representing networks".
- Growing Neural Gas (Fritzke): like Growing Cell Structures, except
  - As with Competitive Hebbian Learning, create a connection (if one doesn't already exist) between winner and runner-up
  - Connections have ages, which are zeroed each time the connected units are winner and runner-up and incremented for neighbors of winner.
  - Connections with ages over some threshold are eliminated (and unconnected units are as well).
  - A new unit is added between the unit with the highest total error and its neighbor with this highest error, but this new unit is not automatically connected to its neighbors' neighbors.
  - Only the error totals of the two neighbors of the new unit are decreased.
  - Unlike other models, the dimensionality of the map need not be pre-specified and can differ in different regions.
  - All parameters have constant values; there is no decay schedule.