Flynn's Taxonomy of Parallel Systems

Flynn did this before machines of either category existed, and it did clarify thought and bring to a stop arguments that had at their basis differing ideas of what a "parallel machine" was.

SIMD performs the same single instruction on multiple pieces of data in parallel. So, for example, a vector update operation y = y + alpha*x could be performed by having each processor simultaneously handling a different range of elements in the vectors. The mental model is of a master or control or host processor which at every cycle sends a common instruction for all of the processors to execute.

MIMD means performing arbitrary operations (e.g. multiple instructions) on different data at the same time. In a MIMD parallel machine each processor may be programmed as a standard sequential machine, running sequential jobs. Nowadays machines can exhibit both SIMD and MIMD parallelism at different levels, such as a multiprocessor Cray T90. MIMD parallel is more flexible than SIMD parallelism. SIMD now usually appears within functional units on a processor (e.g., floating point, integer, and branch units). or memory systems.

Flynn's taxonomy also includes SISD (single-instruction-single-data) machines, which are standard sequential machines, and MISD (multiple-instruction-single-data), a category that essentially does not exist. Some architectures, such as systolic arrays, could be considered as MISD. However, systolic arrays have not been commerically viable because they tend to be special purpose, circumventing the raison d'etre of programmable computers. Their main payoff has been for generating academic publications. O.K., sometimes they work well for a set of computations that have to be done repeatedly over a period of years - e.g., rendering.

SPMD is a form of MIMD, where each processor is executing the same program but on different data. However, in this style of programming each processor runs the same program; in MIMD each processor can be running a different program. This is more of a programming model distinction than a hardware model distinction.

Course taxonomy:

• p will always represent the number of processes.

Note that I use the term processes, not processors. The typical case is where those two are equal to each other, but it's never safe to assume that.

• The term "simultaneous" will refer to operations that can in principle be executed at the same time without affecting the end results.

• processor will refer to a single processor, or a single core in a multi-core processor. As with processes, multiple processors can in principle run simultaneously and indendently.