Testing - I

Slides by Topic

Testing is a process of inferring certain behavioral properties of a product based on the results of executing the product in a known environment with selected inputs.

- J. B. Goodenough

Importance of Testing:

* «Checkout» and testing often consume 50% of software effort.

- Barry Boehm

* The cost of fixing an error increases by an order of magnitude at each stage in development.

- Barry Boehm

Verification:

«Are we building the product right?»

Validation:

«Are we building the right product?»

Testing:

«Are there failures?»

«What are the failures?»

Debugging:

«Why do they occur?»

Utility/Correctness
Meets users needs
Meets functional specifications

Reliability
Avoids unacceptable behavior under normal conditions

Robustness
Behaves properly across a variety of conditions and inputs, normal and abnormal

Availability
Recovers gracefully and quickly from anomalous behavior or conditions

Maintainability

Efficiency/Performance

Security

Many of the above seem quite similar, but that's why it's important to recognize all the variations

* Can't test-in quality

* Program testing can be used to show the presence of bugs, but never their absence.

- Edsger Dijkstra

* Combinatorial explosion

Exhaustive testing isn't possible in most cases

* Concurrency makes it worse!

Difficult to anticipate concurrent behavior and to control testing in concurrent environment

* Theoretical limitations

* Test cycle mirrors life cycle:

* Integration & testing happen inside to out

Test Strategy <--> Integration Strategy

Top-down

entire layer

module at-a-time

Bottom-up

Big-bang

Vertical slice

The following are less integration-dependent:

«Sandwich» testing

Testing by ADT

way to do controlled bottom-up testing in otherwise top-down context

Testing by (global) dataflow

I/O, traversal of program tree

Unit testing phase

Regression testing

retest units after changes in them or related units
part of CCCM

System testing phase

Critical systems require comparison testing

two or more completely independent (redundant) systems are developed and the results compared

Process & Methodology

Appropriate tools and techniques

Organization and planning vital

Testing testing

Tests must be evaluated for adequacy

Driven by:

* Inputs

* Output

* Internals

Test Set vs. Test Execution

Black Box Testing

* Ignores internals

* Driven by specifications

* Suitable for separate V&V team

* Unit testing up to entire system

Glass Box Testing

* Recognizes internals
(i.e. uses source code)

* Driven by design and implementation

* Unit testing

* Can apply tools

* Suitable for test evaluation

* Techniques for test set definition

* Driven by specifications

Equivalence Partitioning:

O specifications divide input into valid and invalid classes

O class usually a range or set of values

O if complex conditions (e.g. regular expressions), treat classes as intersection or union of simple cases

O tests should cover classes independently Boundary value analysis (for range classes):

O ask: «What causes [or changes] condition?»
«What is dividing line?» «Why?»

O test just above and just below range endpoints

O treat number-of-items like a range

O attempt to generate output at boundaries too

Heuristics for menus and commands:

* decide which fields interact

O try both blatant and subtle incorrect forms

O try input that is correct except for order

O try partial correct commands

O try null entries

O try over-constrained commands or too much qualifying data

O try interchanged, mismatched, and wrong delimiters

O try interrupts, reset, other unusual timing problems

Heuristics for forms:

O check functionality as above

* check format and appearance too!

O try all form modes

O try all button modes
visible/invisible, active/inactive, etc.

O try help for every item

* functionality of UI platform typically beyond scope

* usability tests done at unit test time give better chance for correction

* Driven by design and implementation

Techniques
Develop appropriate sets of test inputs in order to cover some aspect of a module

* Condition testing

o Node testing

o Edge testing

* Path testing

* Dataflow testing

Test-Set Evaluation

* Path/condition/flow coverage

* Mutation testing

* Indicates where to test, not how to test

e.g. GB => test both branches of

 if (X<0)

context   instance definition
X is input               trivial/automatic
X is computed            test designer
x<0 is exception         ? ? ?

* cover logical conditions, such as in if statements

* each condition must be evaluated yielding TRUE and FALSE if «significant» test

* treat «A > 0 AND B > C» as two conditions

- result of «B > C» irrelevant if «A > 0» is FALSE
- hence, other conditions in AND must be TRUE
- similarly, other conditions in OR must be FALSE