Exercises

To help you get familiar with the program, the following exercises are suggested. For each one, set up the world and critters appropriately. Then try to predict what you think will happen: Will the critters all die off quickly? Will one group do better than another because of a sensor/effector/learning difference?

Run the simulation for around 5000 time steps for each experiment (until the critters seem to have learned the task or they are obviously not learning it and dying off), and watch what happens. Record the results. You should run each experiment several times (ideally, 5 to 10) so that you get a good idea of what generally happens and also so that you don't make generalizations based on a possibly unusual outcome.

When analyzing the results of a run, you can look at things from the individual level and the population level. From a population-level standpoint, look at how many critters of each species survived and the average age and strength of those species. If there is just one species, you might want to run the simulation several times (resetting the world each time) and compare the results across those runs. If there are several species, compare the measurements for each species to each other.

From an individual-level standpoint, look at the reinforcement and STM windows, and observe critter actions in the world view while the simulation is running. What reinforcement values predominated during the run? Are these values the same across most critters? While watching the STM window, did any outputs or sensor-output combinations seem relvant to a critter doing well?

Overall, you'll want to be able to characterize the strategy that the critters learned (if they learned anything). You may also notice that they learn easier things first, like avoiding bumping into things, then they begin to start eating plants or other critters (watch the reinforcement window along with the STM window).

The importance of the environment

World size

Files:

Load in this file, run the simulation for a while. Then, edit the setup file, size.setup. There are two sets of world sizes commented out. Just uncomment one of the other sizes and comment out the current one. Then run the world again (reload the environment to make the changes take effect). Do this for all of the sizes there and any other reasonable sizes (not too big!) you want.

Compare how well the critters did in each world. Why did they differ?

Movement, world size, food density

Files:

There are two environments and two critter types. The A critters can move up to 3 spaces (they can jump) while the B critters can only move 1 space at a time. Does one of the critter species do better or worse than the other in each environment? Why or why not?

Sensors

Different sensor abilities

Files:

There are 6 species of critters in this world. Each has different sensor capabilities (some have only touch, others have several sensors). There are 2 of each kind. Which ones survive and which ones die off? Why? Do any sensors work well on their own? Do any combinations seem much better? Why might having all sensors not necessarily be optimal?

One-sensor critters

Files:

Try running the above setup after you've disabled the critter's touch and sonar sensors ("comment out" all of the the touch and sonar parameters). What happens? Does this go against your predictions? Why isn't vision alone sufficient for critters to survive long?

Try the same experiment but use only sonar (disable vision and touch). Then try this with only touch. What happens?

Two-sensor critters

Maybe critters need to have several senses working together. Using the same environment, run another experiment with critters that have vision and touch. Do your results match your predictions?

Then, try critters that have vision and sonar. By the way, you can simultaneously compare these new critters to the ones you just tested if you simply edit the critters file and Load critters instead of reloading the whole setup. Do the ones with vision and sonar fare better than those with vision and touch? Do your results seem consistent with the fact that sonar gives more information about distance while touch gives more information about identity? What seems to be the key to solving things in this world?

Critters with all 3 sensors

Now enable all 3 sensors and run the experiment. Did you get what you expected?

Back to one sensor

Try running the same experiment using vision-only critters, but edit the environment file so that the plantEatingIncr is 120. What happened? Can you explain it in terms of what vision does? (Hint: you might want to try other experiments in this world.)

The usefulness of sonar

It may seem that sonar on its own isn't useful. You'll explore that here.

Files:

There are 2 species in this world. One has Sonar, the other has Touch and Vision. Try letting the critters die instead of reincarnating them. You may need to run this world more than once to see noticeable results. Which one fares better? Compare this to the experiments you just performed above -- do these results suggest anything to you about the relationship between species design and the nature of the environment?

Effectors

Critters can eat, rest (do nothing), move, turn and hit. The following exercises explore some of these actions in more detail, especially moving and hitting.

Files:

First run this environment/critter setup as-is. This will serve as a point of comparison (a control) for the following experiments with effectors.

Turning

Critters need to be able to turn to explore their world, avoid being eaten, and orient towards plants and away from obstructions. Edit generic.crit to give the critters a turn value of 5. This means they can instantly move to another heading. Observe their behavior and compare their performance to the control run you did above. Did it differ? Why or why not?

Set turn back to 1.

Hitting

You may have noticed that the critters never eat plants (just each other). All of the plants in this environment are hard, so they need to be hit first. Set hit to 1 and run the simulation. How much of their reinforcement now comes from plants? Do they still eat their friends? Since plants are harder to eat than other critters, why would critters ever bother to hit and then eat plants? Does this make sense from what you know of the rules of this world?

Leave hit set to 1.

Moving

Critters sometimes need to jump over rock boundaries, or move quickly through their world. Currently, move is 1. What if the world were larger? Make this world 30x40 (modify worldRows and worldColumns). Run the critters. Then, change their move to be 3. Run them again and compare your results. Did the critters with longer strides do better?

Reinforcement learning parameters

Learning rate

Files:

There are four critters in this experiment, only differing by their learning rate. A's learning rate is .01, B's is .05, C's is .1 and D's is .25. Which one does best? How does learning rate relate to this particular world and the task that the critters must solve?

Discount rate

Exploitation rate

Files:

Two critter types compete in this world. They differ only in that A's have exploitation rate of .01 and B's have a 0 exploitation rate. What happens to each, and how is exploitation rate helping? Can the learning rate be changed to make the B's more successful?

Challenge problems

For each of the following environments, try to design a critter species that can survive and even prosper. You'll need to consider which sensors and effectors the critters will need. Realize that it is not always best to give a critter every sensor and effector and just let learning do the work. You will also need to consider what to set the learning parameters to. Learning rate, exploitation rate, and discount rate may be crucial to performing a given task well.

For each of the challenge worlds, try to examine the weights in the LTM window if you can get some critters to be successful. You should be able to determine the general "strategy" that the critter has learned; is it similar to a strategy that you thought might be successful?

Gridworld

gridded.env

This world is a lattice of small walled-in rectangles: to navigate successfully in the world, the critters have to learn to jump over the walls separating the cells. Fertile cells are placed in the middle of each rectangle, but there is generally not enough food being eaten and replaced for critters to survive by staying in one rectangle and waiting for new food.

Labyrinth

labyrinth.env

As the name suggests, this world is like an irregular maze, with passages that the critters must learn to follow in order to find new food. Since the grid is irregular and there are a lot of rock walls, it is highly unlikely that critters will be able to jump from passage to passage, so they'll need to learn to follow passages to find the food that placed throughout the maze.

Columns

column.env

This world has vertical columns rocks placed in every other column of cells, so critters have two options for navigation in this world: either moving straight up and down in the same column, or jumping (2) over the rock walls. Critters should be able to learn to jump walls, as the empty rows should be sparse enough to allow jumping in general. Critters are also allowed to eat each other, so they can't be "blocked" in a particular row.

Hexgrid

hexgrid.env

This world is like a hexagonal grid of alternating rocks and fertile cells; because the number of plants is high, fertile cells almost always have plants in them. Because plants are so plentiful, the critters have to learn to navigate successfully in order to quickly eat large numbers of plants.

Hexpath

hex.env

This world has a single hexagonal path, one cell wide, with fertile cells at the six corners of the hexagon. Three plants appear in the fertile cells. A single critter should learn to survive in this world.