Software Engineering Processes

Programming History

1960’s 60’s 1968 “Cowboys” wrote software anyway that they could Difference between best programmers and worst as high as 28:1 (many sources) Start of the “software crisis” 1968 Edsger Dijkstra, “GOTO Statement Considered Harmful” (CACM) Recognition that rules can improve the average programmer

Structuring Software Development Few rules helped immensely Good rules and practices developed over the 70’s and 80’s If a few rules are good, more are better… Late 80’s, major focus on process as a key to quality ISO 9000 (first published 1987) Malcolm Baldrige National Quality Award (just celebrated 25th anniversary) ISO: International body 150 national standards organization (US: ANSI) Originally technical standards Has broadened its scope: e.g., quality ISO 9000: family of standards Generic management system standard Not the process but the management of the process Compendium of best practices Continues to be updated ISO 9001 key standard Business customer requirement based: communication and validation internal audits: evaluation and improvements problem mgmt & effectiveness monitoring: non-conformances, bad product Quality Policy formal statement from management understood and followed at all levels by all employees used to establish employee measurable objectives Quality System regularly audited and evaluated for conformance and effectiveness decisions based on recorded data records that trace raw materials and products to the source

Why not apply to software development? Companies started codifying their practices Large documents and people to manage them Rise of the project manager “Honored in the breach” More large projects and more late or failed projects 1995 Standish Group Study Jerry Saltzer SOSP 1999 1995 Standish Group Study: 50% software projects challenge d 2x budget 2x completion time 2/3 planned function 30% impaired Scrapped 20% success Who is Jerry Saltzer? Early time sharing (CTSS) Multics Operating System (“inspired” Unix) Project Athena Thin client computing Kerberos LDAP Instant messaging

1995 Standish Group Study 50% software projects challenged 2x budget 2x completion time 2/3 planned function 30% impaired Scrapped 20% success

Jerry Saltzer Presentation Who is Jerry Saltzer? Early time sharing (CTSS) Multics Operating System (“inspired” Unix) Project Athena Thin client computing Kerberos LDAP Instant messaging

Processes

The Process Customer Described Lead Understood Analyst Designed Programmer Built Customer Needed

Project noise level Anarchy Complex Requirements Complicated Simple Far from Agreement Anarchy Complex Requirements Complicated Source: Strategic Management and Organizational Dynamics by Ralph Stacey in Agile Software Development with Scrum by Ken Schwaber and Mike Beedle. Simple Close to Agreement Technology Close to Certainty Far from Certainty

Software Engineering Processes Differ by how often you do the steps Points on the spectrum Differences in overhead Three fundamental models Waterfall Spiral Iterative Widely used models Integrated Product Development Unified Software Development Process Extreme Programming

All models address the 4 P’s of Software Engineering People: those doing it Product: what is produced Process: manner in which it is done Project: the doing of it

Fundamental Steps Requirements Design Implementation Test Deployment Maintenance

Processes Waterfall Spiral Iterative (Agile) Differ by how often you do the steps Focus and emphasis Points on the spectrum Differences in overhead Three fundamental processes Waterfall Spiral Iterative (Agile)

Waterfall Do it once Traditional model Used for large next version releases, especially when well understood product tightly coupled changes

Waterfall 1970s Built on 1950’s stage-wise process Recognized the need for feedback Limited Heavy process

Waterfall Pros Cons Simple documentation management Clean design phase Least flexibility No early feedback

Iterative (a.k.a. Agile) Many iterations Each iteration is on a fixed cycle Typically biweekly Used for projects with lots of small independent, but well understood, changes small development team strong client involvement

Iterative Reaction to waterfall Derived from “evolutionary” process Requirements and specs evolve over time Two well-known models Extreme programming SCRUM

Iterative (a.k.a. Agile) Pros Cons Fast feedback on problems Very adaptable to any changes Lots of versions to work with Heavy user involvement Cons Document maintenance Code maintenance Requires good automation

Agile Manifesto Extreme Programming SCRUM February 2001 Representatives from Extreme Programming SCRUM DSDM Adaptive Software Development Crystal Feature-Driven Development Pragmatic Programming

Spiral Few iterations Each iteration adds new requirements Used often for projects with less well defined requirements

Spiral Risk based Barry Boehm 1988 “A Spiral Model of Software Development and Enhancement”

Spiral Pros Cons Adaptation to changes based on risks Good customer interaction Early version Limited iterations provide phase structure Cons Document maintenance

Unified (Software Development) Process spiral variant Iterations within phases 4 phases and core workflows for each Identifies that iterations differ Inception Elaboration Construction Transition Requirements Analysis Design Implementation Test Also known as Rational Unified Process (Rational products)

Historical Recap Waterfall: 1970, built on 1950’s stage- wise processes Recognized need for feedback Iterative (agile): late 70s,modeled on evolutionary model Didn’t work well for large products Spiral: 1988, risk-based

Software Generations (Rational View) Speaker Notes Software Generations (Rational View) 1960s-1980s 1990s-2000s 2005+ Complexity 30% Reused Assets 70% Custom 70% Reused Assets 30% Custom 100% Custom Process Managed and Measured Ad-hoc Repeatable Distributed Systems/Software Professionals Team Collocated On the Job Training Collocated Software Skills Mix of Proprietary and Commercial Not Integrated Proprietary Not Integrated Commercial Integrated Processes-Tools Tools This slide shows three generations of basic technology advancement in tools, components, and processes. The levels of quality and personnel required are assumed to be constant. The three generations of software development are defined as follows: Conventional: 1960s and 1970s, craftsmanship. Organizations used custom tools, custom processes, and virtually all custom components built in primitive languages. Project performance was highly predictable in that cost, schedule, and quality objectives were almost always under-achieved. Transition: 1980s and 1990s, software engineering. Organizations used more repeatable processes, off-the-shelf tools, and mostly (>70%) custom components built in higher level languages. Some of the components (<30%) were available as commercial products, including the operating system, database management system, networking, and graphical user interface. Modern best practices: 2000 on, software production. Today’s philosophy is rooted in the use of managed and measured processes, integrated automation environments, and mostly (70%) off-the-shelf components. Perhaps as few as 30% of the components need to be custom built. With advances in visual modeling and integrated production environments, these custom components can be produced very rapidly. Technologies for environment automation, size reduction, and process improvement are not independent of one another. In each new era, the key is complementary growth in all technologies. For example, the process advances could not be used successfully without new component technologies and increased tool automation. For more information, see Royce pages 21-24. Predictable Unpredictable Predictable Project Performance over budget, over schedule Infrequently on budget, on schedule Frequently on budget, on schedule

Four Patterns of Success Scope management  Asset based development Solutions need to evolve from user specifications AND user specifications need to evolve from candidate solutions. As opposed to getting all the requirements right up front. Process management  Rightsize the process Process and instrumentation rigor evolves from light to heavy. As opposed to the entire project’s lifecycle process should be light or heavy depending on the character of the project. Progress management  Honest assessments Healthy projects display a sequence of progressions and digressions. As opposed to healthy projects progress through a monotonically increasing and predictable plan. Quality management  Incremental demonstrable results Testing needs to be a first class, full lifecycle activity. As opposed to a subordinate, later lifecycle activity.