Q learning examples

• Maze
  

  s	a	r	Q
  			1,r	1,s	2,r	2,l	3,r	3,l	G,s	G,l
  1	r	0	0	0	0	0	0	0	0	0
  2	r	0
  3	r	1
  G	s	0
  . . .
  1	r	0
  2	r	0
  3	r	1
  G	s	0

  γ = .8, η = .5 and all values initialized at 0. In the chart, "new" means the reinforcement received plus the discounted maximum value of the next state. The "new" value is combined with the "old" using the learning rate to give the updated Q-value appearing in the next line of the chart.

  s	a	r	Q								new
  			1,r	1,s	2,r	2,l	3,r	3,l	G,s	G,l
  1	r	0	0	0	0	0	0	0	0	0	0
  2	r	0	0	0	0	0	0	0	0	0	0
  3	r	1	0	0	0	0	0	0	0	0	1
  G	s	0	0	0	0	0	.5	0	0	0	0
  . . .
  1	r	0	0	0	0	0	.5	0	0	0	0
  2	r	0	0	0	0	0	.5	0	0	0	.4
  3	r	1	0	0	.2	0	.5	0	0	0	1
  G	s	0	0	0	.2	0	.75	0	0	0	0

• ALife
  • States (texture of cell in front of critter)
    • Empty (black)
    • Soft (green)
    • Hard (yellow)
  • Actions
    • MOVE
    • TURN
    • EAT
  • Reinforcements
    • Moving: -1
    • Turning: -1
    • Bumping into something: -5
    • Eating a plant: 10
    • Eating a rock: -10
    • Eating nothing (emptiness): -1
  • Q values
    

    		a
    		MOVE	TURN	EAT
    s	EMPTY
    	SOFT
    	HARD

    
    
    

    TURN

    MOVE

    MOVE

    EAT

    EAT

    MOVE

    TURN

    EAT

    EAT

    MOVE

    EAT

    

© 2009. Indiana University and Michael Gasser. www.cs.indiana.edu/classes/b553/Notes/reinf_ex.html 2009-02-05