Software Metrics

              |
              | Product          Quality       Process
              | (Technical)
--------------+----------------------------------------
              |
Size-         | lines of         test          work
  oriented    | code             coverage      effort
              |
Function-     |                  failure
  oriented    |                  rate
              |
Human-        |                  usability
  oriented    |

! «metric» is not used in its mathematical sense

Characteristics of Measures & Measurements

Measure: e.g. yardstick Measurement: e.g. 6.5 inches

Quantitative
* execution time * time of completion of tasks * always numeric, may be continuous or fit regular intervals * if a true zero value exists, then ratios make sense * accuracy considerations are meaningful

Qualitative
* completeness of documentation * portability * tool adequacy * may be nominal («yes»-«no» or other discrete values) * may be ordered («better than»)

Direct

Indirect

How is Software Measured?

=> measurement itself involves products and processes.

hence a software metrics project should itself be evaluated and directed.

* software metrics are environment dependent

hence metrics must be done over a long time as part of a coherent project

How is Software Measured?

* requires substantial knowledge of statistics

* difference between fitting and predicting data

expand this figure

* accuracy improves with knowledge

expand this figure

«confidence envelope»

How is Software Measured?

Establishing a Measure

* distinguish quantitative and qualitative

* define measurement methodology

direct or indirect

* establish scale

X not a metric in mathematical sense

Develop a Model

* analytic or experimental

* descriptive and mathematical

linear model => additive measure => evaluation by pieces => better reliability

Estimation

Most common & sophisticated estimation model:

COnstructive COst MOdel

Barry Boehm, Software Engineering Economics, Prentice-Hall, 1971; summarized in IEEE Trans. on Software Engineering, Jan. 1984.

Also, additional COCOMO course notes

COCOMO evaluation based on:

* size estimation

based on specifications and design sketch
product specific

* environment information

type of project, experience of staff, . . .
organization specific

Hence, cost estimation requires:

* experience

* specifications or preliminary design

Process for COCOMO Estimation

expand this figure

COCOMO Project Classification

Boehm gives many aids for deciding how a project fits within his various classifications.

Major development modes:

organic

* stand-alone application

semi-detached

* utility or application in larger system

embedded

* tight hardware connection, high-concurrency e.g. O.S., ignition control in automobile

Other considerations:

* organizational understanding of project objectives

* tightness of specification conformance requirements

* concurrent development of hardware or operational environment

* need for innovation

Basic COCOMO

Input variables:

KDSI - Kilo Delivered [1mS[22mource [1mI[22mnstructions

mode - determines constants c_i

Output variables:

WM - Work Months

T_dev - minimum calendar Time for development

Basic model:

WM = c₁ × KDSI^c₂

T_dev = 2.5 × WM^c₃

# programmers = WM / T_dev

Basic COCOMO

Effort and Schedule Equations:

mode effort schedule

WM = T_dev =

organic 2.4×KDSI^1.05 2.5×WM^0.38

semidetached 3.0×KDSI^1.12 2.5×WM^0.35

embedded 3.6×KDSI^1.20 2.5×WM^0.32

Effort Multipliers

* An effort multiplier is a numeric value associated with an observable characteristic of a project

* characteristic may be either product or process

* value is 1 for """nominal""" state of characteristic

* also called cost driver

e.g. Program Control Operations


rating      description       value
very low                  straight-line code, non-nested con-            0.7
                          ditionals
low                       straightforward nesting                        0.85
nominal                   mostly simple nesting; some inter-             1.0
                          module control
high                      highly nested; complex  predicates;            1.10
                          considerable intermodule control
very high                 reentrant code; interrupts                     1.20


extra high                dynamic scheduling with multiple               1.35
                          resources

Intermediate COCOMO

Input variables/observations:
KDSI - Kilo Delivered Source Instructions
mode - determines constants c_i
characteristics - determine effort multipliers em_i
Output parameters:
WM - Work Months
T_dev -minimum calendar Time for development
Intermediate model:
WM = c₁ × KDSI^c₂ × PRODem_i
T_dev =2.5 × WM^c₃
# programmers = WM / T_dev

Intermediate COCOMO

Same development modes:
* organic * semi-detached * embedded
Effort multipliers based on characteristics of:
* product * environment (development & operational) * personnel * project (resource/management)
PROD product of all effort multipliers
Effort and Schedule Equations:

mode effort schedule
WM = T_dev =
organic 3.2×KDSI^1.05×PROD 2.5×WM^0.38
semidetached 3.0×KDSI^1.12×PROD 2.5×WM^0.35
embedded 2.8×KDSI^1.20×PROD 2.5×WM^0.32

COCOMO Effort Multipliers

Cost Driver Ratings
____________________________________ very very extra low low nom high high high
product attributes required reliability .75 .88 1.00 1.15 1.40 database size .94 1.00 1.08 1.16 product complexity .70 .85 1.00 1.15 1.30 1.65
system attributes time constraint 1.00 1.11 1.30 1.66 space constraint 1.00 1.06 1.21 1.56 platform volatility .87 1.00 1.15 1.30 environment friendliness .87 1.00 1.07 1.15
personnel attributes designer capability 1.46 1.19 1.00 .86 .71 programmer capability 1.42 1.17 1.00 .86 .70 application experience 1.29 1.13 1.00 .91 .82 platform experience 1.21 1.10 1.00 .90 prog. lang. experience 1.14 1.07 1.00 .95
project characteristics use of good practices 1.24 1.10 1.00 .91 .82 use of good tools 1.24 1.10 1.00 .91 .83 schedule flexibility 1.23 1.08 1.00 1.04 1.10

****** OMITTED OVERLAY ******

Example

Inputs:
PROD = 1.07
KDSI = 73
mode semi-detached
Results:
WM = 3 × 73^1.12 × 1.07 = 392
T_dev =2.5 × 392^.35 = 20
# programmers = 392 / 20 = 20

COCOMO Validity

Q: Is COCOMO any more than just a lot of numbers and a few formulas?
A: COCOMO has undergone a great deal of empirical verification.
The parameters for mode-equations and effort modifiers were derived from fitting experimental data.
Q: Is COCOMO out-of-date?
A: Many of the original parameters are relevant but the general model is still valid.
BUT
* COCOMO can only provide relative results.
* COCOMO still depends on design and other project knowledge.

COCOMO Information Model

expand this figure

Changes Over Time

Outdated/Depreciated Environment Characteristics:
* batch turn-around time for debugging
* primary memory constraints
* ratio of data volume to program size
New/Appreciated Environment Characteristics
* multi-platform targeting
* type of user interface
* tools - now subdivided into many categories:
design aids application generators CCCM
* security
COCOMO II
* expanded characteristics
* begin with «function points» COCOMO II web site

Overview of Function Point Analysis

Goal:
a repeatable methodology for estimating size/complexity of modules based only on their specifications
Function Point (from Functionality Requirements)
General approach:
size = c × SUM( FP_i×w_i )
where
sum is over modules (at all levels)
FP_i is the number of FP's of type i
w_i is a weight associated with type i
presentation on FPA by Hunt, Gottschalk, and Grobstein

Original FPA

Oriented toward functional requirements
Original function point categories:
(with modern terminology)
1. data sources
2. reports
3. user inquiries
4. files
5. external interfaces (to other systems)

Modern FPA

Must incorporate
* new specification methods
* new tools and technologies
Some potential candidates
1. entities
2. relationships
3. data flows
4. forms
subforms, fields, . . .
5. ? ? ?
Where do qualitative requirements fall?
e.g. concurrency constraints

V&V Software Metrics

Estimations of Complexity
correlate to:
Techniques

* related to «glass box» methodologies measure program complexity based on syntax * should be integrated with CM tools
* Cyclomatic complexity
based on structure of control flow graph #edges - #nodes + 2
* Variable usage
* «Software physics»
still controversial size = c × #operands × log(#operands)

Metrics & Quality Assurance

Goals
* Evaluate testing
related to coverage conditions
* Improve reliability
identify weak points
* Improve usability
find awkward places
* Facilitate maintenance
especially corrective and adaptive track «problem» modules

Metrics & Project Management

Goals
1. compare various alternatives
2. evaluate risks
3. monitor
Management must understand value of information
hence must be willing to «invest in information»
project experience database prototype simulate design for measurement

Software Metrics Top 10 List

1. Finding and fixing a software problem is 100 times more expensive after delivery than during requirements and early design phases.
2. A software development schedule can be compressed up to 25%, but no more.
T_dev = 2.5 × WM^0.38
3. For every dollar spent on development, two will be spent on maintenance.
4. Software development and maintenance costs are primarily a function of the number of source instructions.
5. Variations between people account for the biggest differences in software productivity.
6. The overall ratio of software to hardware costs has gone from 15:85 in 1955 to 85:15 in 1985 and is still growing.
7. Only 15% of software development effort is devoted to programming.
8. Software systems and software products typically cost three times per instruction to fully develop as does an individual software program. Software-systems products cost nine times as much.
9. Walkthroughs catch 60% of the errors.
10. Many software phenomena follow a Pareto distribution: 80% of the contribution comes from 20% of the contributors.

- Barry Boehm, 1987