Some applications of GAs

Iterated Prisoner's Dilemma (Axelrod)
- Prisoner's Dilemma: two-person game which is an idealized model of the development of strategies of cooperation and competition
  
  Player B
  
  Cooperate Defect
  
  Player A Cooperate A:3, B:3 A:0, B:5
  
  Defect A:5, B:0 A:1, B:1
- Iterated Prisoner's Dilemma:
  - Pair of players plays multiple times; score is the total payoff for each.
  - Each player has a limited memory of previous moves made by the other player.
  - Each player has a strategy, which maps previous moves of the other player onto a choice of a move.
  - A tournament pits particular strategy entrants against one another for a fixed set of games.
    - The success of a strategy depends heavily on which competing strategies are involved.
    - Round-robin tournament: every strategy plays an equal number of games against every other strategy.
    - Evolutionary tournament: members of a population of players with given strategies play against other players in the population on each generation, and the resulting scores provide the fitness for a genetic algorithm. The proportion of different strategies changes as the population evolves.
  - Some strategies: Random, Always Cooperate, Always Defect, Tit-for-Tat, Tit-for-Two-Tats
- Strategy encoding
  - Each genome locus represents an action to take (Defect or Cooperate) for a particular sequence of previous game outcomes (3 in Axelrod's simulation)
  - Extra loci encode imaginary games to get the strategy started
- Fitness: two possibilities
  - Have the strategy compete against a fixed set of representative strategies
  - Have the strategy compete against the other strategies in the current population
Sorting networks (Hillis)
- Goal: to find the minimum number of comparisons for a correct sorting network with a given n
- Encoding of comparisons
- Fitness
  - Evaluate the network on randomly generated test cases
  - Evaluate the network on a co-evolving population of test cases
Evolving neural networks
- Evolving weights in a fixed network
- Evolving architectures
  - Direct encoding: each locus in the genome represents the presence or absence of a connection between two units
  - Graph generation grammars and "development" of phenotypes
- Evolving a learning rule (Chalmers)
  - Encoding:
    `Delta w_{ji} = k_0(k_1w_(ji) + k_2x_i + k_3y_j + k_4t_j + k_5w_(ji)x_i + k_6w_(ji)y_j + k_7w_(ji)t_j + k_8x_iy_j + k_9x_it_j + k_10y_jt_j)`

	Player B
	Cooperate	Defect
Player A	Cooperate	A:3, B:3	A:0, B:5
Defect	A:5, B:0	A:1, B:1

Genetic programming

Encoding
- Individual is a program in the form of a tree, consisting of functions and terminals.
```
(SQRT (* A (* A A)))
```
Basic algorithm
- Choose a set of functions and terminals for the problem (hard!).
- Generate an initial population of random trees using the set of functions and terminals with the constraint that the programs have to be correct.
- Calculate the fitness of each program in the population by running it on a set of test cases. Fitness is a function of the number of cases on which it performs correctly.
- Apply selection, crossover, mutation, and architecture altering operations to form some proportion of the new population.
  - Crossover consists of choosing a random point in each parent and exchanging the subtrees beneath those points.
  - Mutation consists of replacing a randomly selected function symbol with another function symbol or a terminal symbol with another terminal symbol.
  - Architecture altering operations
    - Subroutine duplication
    - Argument duplication
    - Subroutine creation
    - Subroutine deletion
    - Argument deletion
Examples
- Orbital period of a planet
  - Problem: evolve Kepler's Third Law from data relating the period of the planets to their distance from the sun (`P^{2} prop d^{3}`)
  - Terminal: d, function set: {+, -, *, /, SQRT, SIN, COS}
  - In early generations the algorithm found Kepler's early solution: (* d d)
  - These correct solutions found eventually:
```
(SQRT (* d (* d d)))
(* d (SQRT d))
(- (* d (SQRT d)) (SIN 0.0))
(* d (+ (- d d) (+ (- d d) (SQRT d))))
```
- Block stacking
  - Task: to get some blocks, some stacked, some on the table, into a particular order on the stack.
  - Terminals (sensors): CS (top block of current stack), TB (top correct block), NN (next needed block)
  - Functions: MS(x) (move to stack), MT(x) (move to table), DU(exp1, exp2) (do exp1 until exp2 is true), NOT(exp), EQ(exp1, exp2)
  - Fitness: how well a program does on a set of 166 test initial configurations
  - Generation 1
```
(EQ (MT CS) NN)
(MS TB)
(EQ (MS NN) (EQ (MS NN) (MS NN)))
```
  - Generation 5
```
(DU (MS NN) (NOT NN))
```
  - Generation 10
```
(EQ (DU (MT CS) (NOT CS)) (DU (MS NN) (NOT NN))) [a correct program]
```

Learning classifier systems

Basic features
- Rule and message system: type of production system with
  - Fixed-length representations
  - Parallel rule activation
  - Value determined through competition among rules
- Apportionment of credit system: exchange of internal currency among classifiers
- Genetic algorithm: creates new rules (classifiers)
- Finite-length message list: environmental messages and messages posted by classifiers

Classifiers: condition pattern recognition device with wildcard (#) added to alphabet

When condition of a classifier is matched (by a message from the environment or another classifier), it becomes a candidate to post its message to the message list

0 1 # #:0 0 0 0
0 0 # 0:1 1 0 0
1 1 # #:1 0 0 0
# # 0 0:0 0 0 1
Environmental message:     0 1 1 1,  matches classifier 1
Message from classifier 1: 0 0 0 0,  matches classifiers 2 and 4
Message from classifier 4: 0 0 0 1
Message from classifier 2: 1 1 0 0,  matches classifier 3
Message from classifier 3: 1 0 0 0,  matches classifier 4

Apportionment of credit (the bucket brigade)
- Classifiers have strength.
- Auction: matched classifiers make bids in proportion to their current strength.
- Clearinghouse: Once classifiers are selected for activation, the bids are distributed to classifiers responsible for the matched message.
  - Each matching classifier makes a bid that depends on its current strength and the specificity of its condition.
  - The n highest bidders are selected.
    - Winners get to post their messages on the message list.
    - If there is immediate input from the environment, all winners receive it.
    - Winners lose their bids (they get subtracted from their strength).
  - The bid of each winner is distributed among the classifiers that posted the message that matched the winner's condition.
  - There may also be noise and taxation.
    `strength_i(t+1) = strength_i(t) - payments_i(t) - taxes_i(t) + receipts_i(t)`
    `bid_i = C_(bid) * spec_i * strength_i`
    `actualBid_i = bid_i + noise(sigma_(bid))`
    `tax_i = C_(tax) * strength_i`
Genetic algorithm
- Only a fraction of population is replaced.
- GA not called on every time step.
- Wild cards are included in mutation.
- Crowding may be used to help maintain a diverse population of rules.
How the bucket brigade works
- Classifier C1 fires, resulting in reinforcement from the environment. C1's strength increases.
- Later classifier C2 fires, and its action (message) causes C1 to fire on the next time step. C1 gives its bid, which is proportional to its (increased) strength, to C2.
- Classifier systems implement a form of reinforcement learning (the next unit)
Example: learning to control a robot (Dorigo, 1995)
- AutonoMouse: robot which learns to play (follow a light), to hide from a predator, coordinate its behaviors
- Each classifier has two conditions (both must be matched) and one action
- Sensory message: presence of predator, lair distance and position, light distance and position
- Message to motors: 4 bits specifying forward move (1 or 2 steps) backward move, or not moving
- Monolithic architecture: a single CS learns the global behavior
- Hierarchical (switch) architecture: separate CSs learn the play, hide, and coordination behaviors
- Modular shaping: the basic CSs are trained first and frozen; then the switch CS is trained