1. Requirements Volatility
Mauricio E. Peña
February 28, 2011

2. Agenda Introduction Definitions
Requirements development and management
Requirements metrics
Causes of requirements volatility
Impacts of requirements volatility
Mitigation approaches
Summary 5

3. Importance of Understanding Requirements Volatility
Requirements volatility has been identified by numerous research studies as a risk factor and cost-driver of systems engineering projects1
Requirements changes are costly, particularly in the later stages of the lifecycle process because the change may require rework of the design, verification and deployment plans2
The Government Accountability Office (GAO) concluded in a 2004 report on the DoD’s acquisition of software-intensive weapons systems that missing, vague, or changing requirements are a major cause of project failure3
System developers often lack effective methods and tools to account for and manage requirements volatility
Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO

4. Requirements Volatility is Expected
Changes to requirements are a part of our increasingly complex systems & dynamic business environment
Stakeholders needs evolve rapidly
The customer may not be able to fully specify the system requirements up front
New requirements may emerge as knowledge of the system evolves
Requirements often change during the early phases of the project as a result of trades and negotiations
Requirements volatility must be anticipated and managed
Sources; Kotonya and Sommerville (1995); Reifer (2000)

5. Requirements Definitions
“Requirements are intended to change vague desires into explicit and unambiguous statements of what the customers want” 1
“A requirement is a capability that a system must supply or a quality that a system must possess in order to solve a problem or achieve an objective within the system’s conceptual domain” 2
“A requirement is: (1) A condition or capability needed by a user to solve a problem or achieve an objective (2) A condition or capability that must be met or possessed by a system or system component to satisfy a contract, standard, specification, or other formally imposed documents ” 3
Source: 1- Weinberg, G. (1983), 2- Costello and Liu (1995); 3- IEEE (1990)

6. Requirements Categories
Customer or operational requirements define the expectations of the system in terms of operational scenarios and environments, mission objectives, and measures of effectiveness
Functional requirements specify what has to be done in terms of tasks or activities
Performance requirements define how well or to what extent the activities must be accomplished
Design requirements dictate how to build products and what manufacturing processes to follow
Derived requirements are spawned from higher-level requirements; and
Allocated requirements: high level-requirements are partitioned into multiple lower-level requirements
Source: DoD Systems Management College (2001)

7. Requirements Volatility Definitions
Requirements volatility is typically defined as the change in requirements (added, deleted, and modified) over a given time interval
Also known as:
Requirements creep: An increase in scope and number of system requirements
Requirements churn: Instability in the requirements set – requirements are modified or re-worked without necessarily resulting in an increase in the total number of requirements
Source: MIL-STD-498

8. Requirements Management
The Systems Engineering Capability Model (EIA, 2002) defines five activities that influence the generation and evolution of requirements1
Problem refinement
Requirements analysis
Requirements quality
Requirements evolution and maintenance
Feedback and verification
Requirements definition and analysis are sometimes collectively referred to as requirements engineering:
The use of systematic and repeatable techniques to identify, document, develop and maintain a complete and consistent requirement set2
Source: 1- EIA (2002); 2- Kotonya and Sommerville (1998)

9. Context Diagram – Requirements Analysis Process
Source: Systems Engineering Handbook. INCOSE Version 3.2

10. Requirements Development
The customer’s needs and objectives are developed into operational concepts to satisfy them
System functional and performance requirements are developed to define what the system must do and under what environments and constraints it must perform
Once the baseline system requirements and system architecture are established, they are decomposed through successively lower levels.
Source: DoD Instruction (2008)

11. Requirements Baseline and Change Control
A Requirements baseline is established when the functional requirements and characteristics of the system are formally documented and placed under configuration control
After the System Requirements Review or equivalent milestone
Modifications to the baseline are requested through Engineering Change Proposals (ECP)
Typically, a formal management approval structure is established to prioritize the ECPs, evaluate their impact, and make decisions regarding their approval and implementation
Other change documents used to control the requirements baseline are Requests for Deviation and Requests for Waivers
RDWs propose a departure from the baseline by allowing the acceptance of products that are non-conformant or do not meet the requirements as stated
Source: DoD Instruction (2008)

12. Systems Engineering Reviews and Acquisition Lifecycle Phases
Source: DoD Instruction (2008)

13. Requirements and Evolutionary Acquisition Process
Source: DoD Instruction (2008)

14. Requirements Base Measures
Measurement Method
# of Requirements
Count the # of requirements in the applicable specification (# of shalls/wills/musts)
# of requirements defects
Count the # of defects
# of requirements changes
Count the # of changes per category (added, deleted or modified) from Engineering Change Proposals (ECP), requirements database, etc.
Impact of each change
Estimate effort-hours per change (sometimes estimated in ECPs, budgeted hours vs. actual hours)
Source: Systems Engineering Leading Indicators Guide, Version 2.0, 2010

15. Requirements Derived Measures
% Requirements Growth =
((# of requirements in current baseline - # of requirements in previous baseline)
( # of requirements in previous baseline) X 100
% Requirements Modified =
(# of requirements modified / total # of requirements)
Source: Systems Engineering Leading Indicators Guide, Version 2.0, 2010

16. Requirements Volatility Metrics (1 of 2)
Volatility expressed as a % of the total number of requirements
Systems Engineering Leading Indicators Guide, Version 2.0, 2010

17. Requirements Volatility Metrics (2 of 2)
Volatility metrics track the number and type of changes over time
Source: Hammer, T., Huffman, L., and Rosenberg, L. (1998).

18. Periodic and Cumulative Measures

19. Expected Level of Volatility Across Lifecycle Phases (Survey)
Most respondents expect >20% volatility during the conceptualize phase of the project, decreasing to <5% in the transition to operation phase
Source: LAI Knowledge Exchange / CSSE ARR Survey (2010)

20. Impact of Hardware/Software Project Breakdown on Expected Volatility
Operational Test & Evaluation Lifecycle Phase
Transition to Operation Lifecycle Phase
Respondents from S/W intensive projects tend to expect more volatility later in the lifecycle
Source: LAI Knowledge Exchange / CSSE ARR Survey (2010)

21. Sample Volatility Profiles
Source: Practical Software & Systems Measurement Conference Survey (2010)

22. Requirements Trends as a Leading Indicator of Project Performance
Requirements Growth Trends
Evaluates trends in the growth, change, completeness and correctness of the system requirements.
It helps to determine the stability and completeness of the system requirements which could potentially impact project performance
Corrective Action Taken
Number of Requirements
Planned # of Requirements
Actual # of Requirements
Projected # of Requirements
SRR PDR CDR
Time
Source: Systems Engineering Leading Indicators Guide, Version 2.0, 2010

23. Causes of Requirements Volatility
External factors
Shifting customer priorities and needs
Changes in the political and business environment
Addition or change of stakeholders
Development of new technologies
Changes in co-dependent systems
Internal factors
Deficient requirements development processes
Lack of experienced systems engineering resources applied to requirements analysis
Poor initial understanding or interpretation of customer needs by the development team
Changes in organizational structure and policies
Sources: Kotonya and Sommerville (1995); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)

24. Systems Dynamics View of the Causes of Requirements Volatility6 7 8 9
Contextual / Environmental Changes
Changes in Org / structure/ Process
Experienced Staff
Changes in COTS Products 1 2 3 4 5
Poor Understanding of the System & Customer Needs
Changes in Co-dependent Systems
Requirements Process Maturity
Customer-driven Scope Change
Technology Maturity
Requirements Volatility
Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009) 7

25. Impacts of Requirements Volatility
Several research studies have found that requirements volatility is positively correlated with increase in
The functional size of the project (due to discarded lines of code and additional requirements)
Engineering effort
Cost (often >20%) and schedule duration
Changes in requirements result in additional rework and increased defect density
The impact of changing requirements is much greater the later the change occurs in the project lifecycle
Removing a requirement may not necessarily result in a net decrease in engineering effort and cost
Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)

26. Systems Dynamics View of the Impact of Requirements Volatility
Quality
Rework / Defects
Project Cost
+/-
Requirements Volatility
+/-
Customer Satisfaction
Project Effort
+/-
+/-
Number of System Requirements
+/-
+/-
Project Schedule
Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009) 7

27. Cost Commitment on Projects
Changes to the System are more difficult to implement the later they occur in the lifecycle
Source: Blanchard and Fabrycky (1998)

28. Expected Cost Penalty due to Requirements Volatility (Survey Results)
Source: COCOMO Forum Survey Exercise (2010)

29. Requirements Volatility: A Sizing and Cost Estimation Challenge
Our dynamic competitive environment leads to rapid changes in objectives, constraints and priorities
Requirements are often emergent instead of pre-specifiable
These trends have lead to the increasing use of incremental and evolutionary development strategies
Counting the number of source lines of code (SLOC) delivered may underestimate the size of the system
Portions of the code may have been modified or deleted due to changing requirements
These modified or discarded lines would not result in an increase in the final SLOC count
Source: AFCAA Software Cost Estimation Metrics Manual Version 0.82 (draft)

30. Options for Handling Cost and Size Estimation Challenges
Improve ability to estimate requirements volatility
Data collection and analysis: use of code counters able to determine numbers of SLOC added, modified, and deleted during development
If data are not available, estimate expected ranges of requirements volatility
For incremental and evolutionary development:
Treat the earlier increments as reused software and apply reuse factors
Include an Incremental Development Productivity Decline (IDPD) factor
A decrease in producibility is observed in later increments
Previous-increment breakage, usage feedback, and increased integration and test effort lead to higher unit cost
Source: AFCAA Software Cost Estimation Metrics Manual Version 0.82 (draft); Nguyen (2010);

31. Requirements Volatility and Parametric Cost Estimation
The Constructive Cost Model (COCOMO II) incorporated the impact of requirements volatility on the effective size of software systems using an adjustment factor called REVL (Requirements Evolution and Volatility)
REVL is also known as the “breakage parameter,” defined as the % of code discarded due to changes in requirements.
Where,
SizeD is the initial equivalent size of the software product adjusted for reuse
Source: Boehm et al. (2000).

32. Requirements Volatility Expected Ranges
Source: AFCAA Software Cost Estimation Metrics Manual Version 0.82 (draft)

33. Incremental Development Productivity Decline (IDPD) Example
SLOC 1M
Source: AFCAA Software Cost Estimation Metrics Manual Version 0.82 (draft)

34. Actions to Mitigate Requirements Volatility
Improve communication between stakeholders and development team
Improve the capability and maturity of the requirements development process
Additional training / more experienced personnel
Increase the use of prototypes and models to validate requirements
Improve technology insertion processes
Increase management oversight (i.e. metrics, reviews)
Establish a more strict change control process

35. Mitigation Factor Effectiveness: Survey Results
Source: CSSE ARR Survey (2010)

36. Characteristics of Good Requirements (1 of 3)
Necessary
Avoid specification of design details & redundant requirements
Implementation Independent
Specify “what” is to be done, not “how” to do it
Clear and Concise
Don’t use ambiguous terms; avoid compound requirements with multiple “shall statements” and conjunctions such as “and”, “or”
Example – is the following a good requirement?
“The car shall accelerate from 0 to 60 MPH in 5 seconds and break to a standstill in 10 seconds under extreme conditions”
Source: Systems Engineering Handbook. INCOSE Version 3.2

37. Characteristics of Good Requirements (2 of 3)
Complete
Provide all the information necessary for the developer to understand what you mean
Do you have any TBXs (to be determined, to be confirmed)?
Consistent
Is the requirement self-consistent?
Do you have conflicting statements?
Achievable
Engage the developer/designer in the requirements definition to ensure they concur that the requirement can be met
Source: Systems Engineering Handbook. INCOSE Version 3.2

38. Characteristics of Good Requirements (3 of 3)
Traceable
Are there any orphan requirements (i.e. they don’t trace to a higher level requirement)?
Verifiable
Avoid subjective terms such as “provide support,” and “user friendly”
Look out for conditions that are not clearly achieved such as “minimize,” “to the degree possible,” etc.
Example:
The aircraft should be as light as possible
The wet weight of the aircraft shall not exceed 2000 lbs.
Source: Systems Engineering Handbook. INCOSE Version 3.2

39. Requirements Monitoring & Control
Org Improvement Actions + +
+/-
SE Process
Maturity
Staff
Experience - -
Volatility
Metrics /
Thresholds
Requirements
Volatility
+/- +
+/-
Rework
Number of
Requirements
+/- +
SE Project
Effort
+/-
+/-
+/-
SE Cost
Project
Schedule
Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)

40. Summary
Requirements changes are unavoidable in today’s rapidly changing environment
Some level of volatility is healthy and is part of the system development process
However, excessive volatility and late requirements changes can have a very detrimental effect on the project – this is a risk that must be addressed and mitigated
Requirements volatility should be anticipated, managed, and accounted for in cost and size estimates

41. References
Air Foce Cost Analysis Agency Software Cost Estimation Metrics Manual, Draft Version 0.82 (2011)
Boehm, B. (1991). “Software Risk Management: Principles and Practices.” IEEE Software. Vol 8 (No.1).pp 32-41
Boehm, B., Abts, C., Brown, A.W., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer, D.J., and Steece, B. (2000). Software Cost Estimation with COCOMO II. Prentice Hall
EIA (2002). EIA Systems Engineering Capability Model
Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009). “Understanding the effects of requirements volatility in software engineering by using analytical modeling and software process simulation.” The Journal of Systems and Software. Vol. 82, pp
General Accounting Office (2004). Stronger Management Practices are Needed to Improve DOD’s Software-intensive Weapon Acquisitions (GAO ). Defense Acquisitions.
Hammer, T., Huffman, L., and Rosenberg, L. (1998). “Doing requirements right the first time.” Crosstalk, the Journal of Defense Software Engineering. Pp
Houston, Dan X. (2000). A Software Project Simulation Model for Risk Management, Ph.D. Dissertation, Arizona State University
IEEE (1990). IEEE Standard Glossary of Software Engineering Terminology (ANSI), [ X] [SH13748-NYF], IEEE
INCOSE (2010). Systems Engineering Handbook. INCOSE Version 3.2
Kotonya, G., Sommerville, I., (1998). Requirements Engineering: Processes and Techniques. John Wiley and Sons, Ltd.
MIL-STD Software Development and Documentation. U.S. Department of Defense.
Nguyen, V. and Boehm, B. (2010). A COCOMO Extension for Software Maintenance. 25th International Forum on COCOMO and Systems/Software Cost Modeling
Reifer, Donald J “Requirements Management: The Search for Nirvana.” IEEE Software. Vol 17 (No. 3), pp 45-47
Roedler, G. and Rhodes, D. (2007). Systems engineering leading indicators guide. Version 1. Massachusetts Institute of Technology, INCOSE, and PSM
Weinberg, G. (1993). Quality Software Management: Volume 2 First-Order Measurement. Dorset House.
Zowghi, D. and Nurmuliani, N. (2002). A Study of the Impact of Requirements Volatility on Software Project Performance. Proceedings of the Ninth Asia-Pacific Software Engineering Conference