Hopfield networks with delays (Kleinfeld): architecture
- Goal
- Given a sequence of patterns, learn to respond like a sequential content-addressable memory
- A network with a set of quasi-attractors, which it settles into temporarily in a pattern sequence
- In addition to static weights, a set of transition weights
- Directional, asymmetric (`w_(ij)^T` is not necessarily equal to `w_(ji)^T`)
- Delays significantly longer than the update step size
- Input function (`tau` the delay on the transition weights)
`h_i(t) = sum_(j=1)^N[w_(ij)^0x_j(t) + w_(ij)^Tx_j(t - tau)]`
- Learning rule (`Q` individual patterns, `P` patterns within the sequence)
`w_(ij)^0 = sum_(p=1)^Q x_i^p x_j^p`
`w_(ij)^T = lambda sum_(p=1)^P x_i^(p+1) x_j^p`
Hopfield networks with delays: behavior
- The input to a unit is a function of the overlap between each of the patterns and the current state and between each of the patterns and the state before the delay `tau`
`h_i(t) = sum_(j=1)^N[w_(ij)^0x_j(t) + w_(ij)^Tx_j(t - tau)]`
`= sum_(j=1)^N[sum_(p=1)^Q x_i^p x_j^px_j(t) + lambda sum_(p=1)^P x_i^(p+1) x_j^px_j(t - tau)]`
`= sum_(p=1)^Q x_i^p sum_(j=1)^N x_j^px_j(t) + sum_(p=1)^P x_i^(p+1) sum_(j=1)^N x_j^px_j(t-tau)`
where the second summation in each term represents the overlap
- If the network is started in pattern `q`, and there is little overlap between the patterns and little overlap with other patterns at earlier times, the input is just
`h_i(0) ~~ N x_i^q`
- Following an interval of length `tau`, the input becomes
`h_i(tau) ~~ N(x_i^q + lambda x_i^(q+1))`
- If `lambda` is greater than 1, the network will now move toward pattern `q+1`
- Stability
- The network does not settle into fixed-point attractors.
- But for periods less than that of the delays on the transition connections, the network behaves as if it had a fixed-point attractor.
- Problem: handling overlapping sequences
