Solutions to the binding problem

A binding dimension
- Synchronization
  - Units representing the same object are in phase with one another
- Alignment
Conjunctive coding
- Units represent conjunctions of features along two or more dimensions
- Solution to the problem of the large number of units: coarse coding (O'Reilly et al.)

The relational binding problem

Representing a relation between two objects
- a red circle near a blue square
- a red circle above a blue square
Representing relations between relations
a red circle above a blue square
RED_CIRCLE ABOVE BLUE_SQUARE
Symbolic representations
- Argument
```
above(circle1, square3)
```
- Role-filler pairs
```
[rel=above, higher=circle1, lower=square3]
```
- Directed acyclic graphs

One kind of symbolic structure consists of unordered collections of role-filler pairs:

((lover = lois) (lovee = clark))
To represent and make use of such a structure, we need
- a way to represent symbols, such as lover and john
- a way to bind role and filler symbols together
- a way to combine the role-fillers that make up a structure
- a way to unbind a pair, given a structure (trace) and a role or filler (cue)
Convolution memories: a basic reference (Plate, 2002)

Symbols: vectors
Binding: tensor product (generalized outer product) of filler and role vectors.
Combination: tensor addition
Unbinding: dot product of role or filler vector and tensor representing structure, yielding vector representing a noisy version of the other element of the pair
Error retrieving a filler from a trace `vec s`, given a cue `vec r_i`
`vec s * vec r_i = (sum_j vec f_j ox vec r_j) * vec r_i`
`= sum_j vec f_j (vec r_j * vec r_i)`
`= (vec r_i * vec r_i)vec f_i + sum_(j!=i)(vec r_j * vec r_i) vec f_j`
Ratio of incorrect to correct filler
`(vec r_j * vec r_i)/(|\|r_i|\|^2) = cos theta_(ji) |\|r_j|\|/|\|r_i|\|`
Problem 1: size of trace increases exponentially with the depth of the structure being represented
Problem 2: we need to know beforehand how many dimensions (ranks) are required (because they are built into the hardware)

Symbol representation and combination: as for Smolensky's approach
Binding: circular convolution, resulting in a vector, rather than a tensor of rank greater than the elements in the pair
Unbinding: circular correlation

Constraints, limitations
- For correlation to decode convolution, elements of each vector must be independently distributed with mean zero and variance 1/n
- Convolution trace stores pairs only with enough information to discriminate item from others; a cleanup memory (e.g., a Hopfield net) is required if an accurate output is needed
Error retrieving a filler from a trace, given a cue
`vec (f) o/ vec r = [(f_0r_0+f_1r_2+f_2r_1 ),(f_0r_1+f_1r_0+f_2r_2),(f_0r_2+f_1r_1+f_2r_0)]`
`vec (r) o- (vec(f) o/ vec r) = [(f_0R_1+f_1R_2+f_2R_2 ),(f_1R_1+f_0R_2+f_2R_2),(f_2R_1+f_0R_2+f_1R_2)] ~~ [(f_0 + n_0),(f_1 + n_1), (f_2 + n_2)]`
where `o/` represents circular convolution, `o-` circular correlation, `R_1 = r_0^2 + r_1^2 + r_2^2 ~~ 1`, `R_2 = r_0r_1 + r_0r_2 + r_1r_2 ~~ 0`, and `n_i` are noise terms.
Size of trace is constant as depth of structure increases
Implementing circular convolution with sigma-pi units
Encoding and decoding complex propositions with HRRs
- Politicians tell stories.
  P_tell = tell + politicians + stories + tell_agt ⊘ politicians + tell_thm ⊘ stories
- Bill knows politicians tell stories
  P_know = know + bill + P_tell + know_exp ⊘ bill + know_thm ⊘ P_tell
- What does Bill know?
  know_thm ⊖ P_know ≈ P_tell