Tutorial

This section is a tutorial on using Salsa. You will learn how to start the program, how to set up various parameters for critters and for the world, and then you will learn how to run an experiment and observe the results.

The world

The world is a hexagonal "grid" where each hexagonal space can hold one thing. The things in the world are: critters, rocks, plants. Critters are red, and look like a pie with a slice cut out of it. In their center, each critter is labeled with its species name. Rocks are blue. Plants are light or dark green. If a space is empty, it is black.

When the simulation is running, critters each get one action per time step. Critters cannot pass through objects or move into a cell that contains another object. However, they can jump over objects if they have been given the ability to jump.

Critters are updated one at a time, so they will never try to move into the same space as another critter simultaneously. However, if a critter tries to move (or jump) into a space that is occupied, its movement will simply be ignored. Critters have a heading, so when they move, it is always in the direction they are facing. A critter can turn, however, to change its heading.

The world is wraparound, which means that instead of being a rectangular space bounded by walls on all four sides, moving off of one side brings a critter to the other side (left<->right, top<->bottom). The world is, in effect, a torus.

When critters eat a plant, it disappears. That plant will then be replaced by another plant since the number of plants in the world is constant. Rocks cannot be affected by critters (neither movable nor breakable). Some plants are hard to eat. In fact, plants are either hard or soft. Hard plants can't be eaten, but if they are hit then they turn into soft plants. Only soft plants can be eaten. Critters can also eat each other. If a critter eats another, it really just takes a bite out of the other. The eater gets some strength while its "meal" loses some strength.

If a critter ever dies (its strength goes to 0 or less), it is removed. If the simulation is set so that critters "don't die", another critter will take the place of the dead one, with full strength, 0 age, and a random neural net. If a critter dies, then that critter is simply removed, and all of the data about that critter are blacked out.

A world is specified in a text file that describes attributes about that world, such as size, numbers of plants and rocks, where plants and rocks can appear, and how much food is worth and how costly it is to move. New worlds can be created by modifying existing world files or by starting from scratch and using a sort of "world painter" to draw where rocks and plants will go.

Critters

Critters are the organisms that act in a given world in Salsa. They can move, eat, and hit (these are done via effectors). They can see, use sonar, and touch things next to them (these are done via sensors). Each one behaves according to its inputs and its neural net. The neural net is initially random, but after each action in the world, a reinforcement value allows it to modify its network. A reinforcement learning technique known as Q-learning is used to modify the critter's net. After many actions in the world, the critter may be able to learn the right things to do (e.g., eat plants and/or other critters, don't eat rocks, don't bump into things) and thus it may live a long time. However, some critters never learn, and they just die. That's life.^TM

Critters are not all the same. Each critter "species" (or type, or kind) is specified in a special text file, much like the world files; a list of parameters defines a critter. Some critters will only have vision, some will only have touch and vision, and still others may have vision and sonar. Some critters will only be able to eat, move and turn. Others may also be able to hit. Any combination is possible, though some combinations are silly for any world (a critter that can't eat won't last long at all!).

Critters have 3 kinds of sensors: touch, vision and sonar. Touch is a way to detect the softness/hardness of an adjacent object. For things that are farther away, sonar and vision are useful. Vision contains no depth information (no distances, sizes, etc., just like in real visual systems). Instead, it gives detailed information about what is out there. This information is in the form of colors that are like those in the world. Thus, far-away plants or rocks or critters can be detected, although no information about how far away is available. Sonar, on the other hand, is like vision in that it can sense things that are far away. It can't tell what it finds, but how far away it is.

Critters can have any combination of 5 effectors: eat, hit, move, turn, rest. Critters can eat plants and other critters, assuming that plants and critters in their world are edible (these are world parameters that you can modify for each world). Critters can hit, which allows them to convert inedible, hard plants into edible, soft plants. Critters can move, meaning that they can advance forward in whatever direction they are facing. They can also turn, allowing them to change their heading. Finally, critters are able to rest, which means that they can sit and do nothing. This is sometimes a good idea, especially if food is a long way away and some is likely to grow near you soon.

Critters can get benefits for eating plants and critters (assuming these are edible). Critters incur costs for all other actions: turning, moving, hitting, eating (just doing this requires energy). Critters can also incur costs for colliding with anything else, being eaten by another critter, or by simply existing.

Each critter has a neural net that tells it what to do given some inputs. Initially, the net contains random weights. Thus, critter actions at the beginning of a run will seem random and silly; they are. However, after each action, a reinforcement value is given, the sum of all costs and rewards from that action. E.g., if a critter ate a plant but was itself being eaten, the total reinforcement for the eat action is (eating reward - eat action cost - being eaten cost). This value is used to adjust the weights from the critter's input units to the eat neuron. Only the weights from the inputs to the eat neuron are adjusted, so learning is somewhat slow. Many actions must occur before all the output neurons have had time to be adjusted.

The critters use Q-learning (a reinforcement learning algorithm) to adjust their neural nets.

Getting Started

The program is a Java application, so you will have to have the software to run Java applications on your computer. This differs depending on what operating system you are using. To download the program, click on this link. The link points to a zipped (compressed) file, and if your browser is set up right, it should download and decode the file. If not, you may have to explicitly copy the link to your hard disk and then run the program that unzips files. Once you have unzipped the file, you should have one folder (directory) called Src and another called Classes. You will only need the files in Src if you want to see how the program was written. To start the program, go to the Classes folder and run java on the class called Salsa. (If you don't know how to do this on your operating system, ask someone else familiar with it.) Several windows should appear. Arrange them on your screen and activate the window labeled Salsa. You're ready to go!

Try it out

Start up Salsa. Then, try the following exercise. You will learn how to load a world, add some critters to it and run the simulation.

Load a world

You should also see several critters in the world you just loaded. Critters are red, with a letter in the center identifying their particular "species" (given by name in their critter file). Theere are two species present: the critters in this environment are labelled "A" and "B".

Start the simulation

You can stop the run by pressing interrupt. The run will stop after a certain number of time steps. You can resume an interrupted or stopped run by pressing run again.

You can observe your critters aging and losing strength (gaining strength if they're learning useful actions). You can watch the proportions of their reinforcement signals that come from rewards and punishments. If a critter is receiving more reward than punishment, its strength will fall. You can also watch the STM to find out what the creature can see, touch and detect with sonar, as well as what its action outputs are (green is positive and red is negative activation for actions). As time progresses, some critters may die; they will be replaced by new critters. The success of a run can be measured by how many old critters are in the population. If critters can get old, then they have been able to solve their problems effectively.

When you are done, you can try out another world or critter species by loading the appropriate files and running again.

Creating worlds

You can also load in certain environment files that contain a world file which allows you to decide where rocks and plants will be placed. One of these files is tutorial2.env. Load this file in and a window will appear labeled, Place rocks and fertile areas. The world window will be black with lots of dark grey dots evenly spaced around it. If you select Rocks from the pull-down menu in the placement window, you can click on the dots in the world window to put rocks there. If you select fertile cells from the pull-down menu, you can click on the world dots to place fertile cells. These cells are where plants can grow. Once you have placed rocks and fertile cells to your heart's content, click OK and the world will be filled with rocks and plants as you have specified. You can save this world if you like from the Save environment option under the File menu (which actually saves a world file, not an environment file, since you don't care what critters were in the world, just the world itself). Saved files go to the Data folder in the Salsa folder.

Aside: rock and plant placement

Simple experiments

Running critters in an environment

You'll want to run the critters for 2500 time steps, so go to the Options file menu and select Set display parameters. A window will pop up. Change the "# time steps per run" to 2500 and press OK.

Start the simulation (press Run). Watch the reinforcement window. You should see that the rewards for the B critters (the bars on the bottom) are much smaller than those for the A critters (the bars on the top). This is because the As have very limited sensory abilities as compared with the Bs. (Actually, the As have no sensors at all.)

Click on a bar in one of the STM, reinforcement or strength windows. You should see the critter light up that corresponds to this bar.

After a few hundred time steps, most of the original As should have died off while most of the original Bs have survived. However, for each of the As that died, a new A took its place. You can see this by looking at the ages of the various critters in the strength&age window. Click on one of the bars which shows an older critter (longer age indicator) and it should be one of the Bs. By contrast, all of the As should be young (since they're being "reincarnated" whenever another A dies).

You might want to Interrupt the run in order to examine the highlighted critter more closely. However, interrupting a run means that the next time you press Run, the number of time steps will be reset (2500 again in this case).

You can turn off the reincarnation of critters by selecting critters die from the Options menu. When this option is on, critters who die are not replaced, and their corresponding strength, reinforcement and STM bars turn grey.

Modifying environment files

How many plants and rocks?

How will rocks and plants be placed?

Edit the parameter plantPlacement. It is currently 0, indicating that plants are to be placed randomly. Make it 3, so that you can place "fertile cells" where plants will grow. Load the setup file, and you'll notice that you are now told to place fertile cells and rocks. Select from the popup menu which kind you want to place and click away. When you've put enough rocks and fertile cells into the world, click OK and run your world. Plants will now only appear where your fertile cells were placed (indicated by dark grey hexagons). You must place at least as many fertile cells as nPlants is set to (otherwise, it can't fill up the world properly).

plantPlacement can also be set to 1 or 2, indicating that plants will be placed randomly but always next to rocks (1) or randomly but never next to rocks.

Reinforcement values

Edibility

World size

Modifying critter files

Modifying setup files

How many critters from some critters file (load as many files this way as you like):

critters  critters-file #critters

environment environment-file

Copy the original intro.env and make a few changes, saving it under the name intro2.env. Then, modify the setup file to read this new environment. Load the setup file into Salsa to see this work.