Productive Engineering Teams Carnegie Mellon University Software Engineering Institute Productive Engineering Teams Watts S. Humphrey Software Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213-3890 Sponsored by the U.S. Department of Defense © 2000 by Carnegie Mellon University 1

The Improvement Strategy Management comes first: without sound management cost and schedule performance is unsatisfactory quality engineering work is unlikely With the CMM, management performance improves. Cost and schedule commitments are met. Processes are defined and data gathered. Quality engineering work may or may not be done. The PSP and TSP provide disciplined engineering. Engineers use defined processes and data. With sound management and disciplined practices, quality engineering is standard.

Objectives Engineering organizations strive to produce quality products on aggressive schedules for lowest cost To do this, they need motivated and productive teams. The methods for building such teams are well known but not obvious. This talk describes how to build, manage and motivate productive engineering teams.

Characteristics of Productive Teams are attacking important problems have aggressive but realistic goals work to defined plans and processes The team members are skilled and trained for the job know their roles and responsibilities are personally committed to the work The Team Software Process (TSP) builds productive teams. SM SM Team Software Process, and TSP are service marks of Carnegie Mellon University.

Costs and Benefits - 1 The benefits of the TSP are substantial. Productivity has more than doubled. Teams have been consistently on or ahead of schedule. Testing time was cut by 10 or more times. Delivered products have been defect free. Engineering turnover has been zero.

Costs and Benefits - 2 Results from three projects in one company show that Test defects were reduced from an average of 20/KLOC to 1/KLOC. At an average cost of 12 hours per defect, the engineers paid for the TSP investment with the first 1000 LOC written. By using TSP, the company estimates they have saved an estimated $5.3 Million in two years. Customer acceptance test time was reduced from many months to a few weeks.

Costs and Benefits - 3 Engineers like working on productive teams: This really feels like a tight team. This process forces you to design, to think the whole thing out. Design time is way up but code time decreased to compensate. Tracking your time is an eye opener. Really good teamwork on this project - no duplication of effort. I’m more productive. Gives you incredible insight into project performance. Wonderful to have team members assigned specific roles. Team really came together to make the plan. I feel included and empowered.

Costs and Benefits - 4 The introduction costs for TSP are significant. Seminars and courses executives - 1 1/2 days managers - 4 days engineers - 125 to 150 hours instructors/coaches - engineer training plus 2 weeks team launches, relaunches, and coaching Culture change executives - leadership and participation managers - coaching and support engineers - commitment and ownership

The CMM and the TSP The SEI has addressed process improvement from two perspectives. The CMM® establishes the management framework. policies and practices systems, methods, and facilities The TSP shows engineers and their managers how to use a defined, measured, and planned process establish the conditions for effective teamwork manage, track, and report on the work ® Registered with the U.S. Patent and Trademark Office.

CMM, TSP & PSP Relationship CMM - Builds organizational capability TSP - Builds quality products on cost and schedule PSP - Builds individual skill and discipline

The CMM KPAs and TSP/PSP Level Focus Key Process Areas (KPA) Software Project Planning Software Project Tracking Defect Prevention Technology Change Management Process Change Management Quantitative Process Management Software Quality Management Organization Process Focus Organization Process Definition Integrated Software Management Software Product Engineering Peer Reviews Indicates the CMM KPAs that are fully or partially addressed at the personal level in the PSP 5 Optimizing Continuous Process Improvement Requirements Management Software Project Planning Software Project Tracking Software Quality Assurance Software Configuration Management Software Subcontract Management Defect Prevention Technology Change Management Process Change Management Quantitative Process Management Software Quality Management Organization Process Focus Organization Process Definition Training Program Integrated Software Management Software Product Engineering Intergroup Coordination Peer Reviews 4 Managed Product and Process Quality 3 Defined Engineering Process Indicates the CMM KPAs that are fully or partially addressed at the team level in the TSP Requirements Management Software Quality Assurance Software Configuration Management Intergroup Coordination 2 Repeatable Project Management

TSP Teams Need Proper Training The PSP Course The TSP Launch The TSP Process Project goals Team roles Team process Project plan Balanced plan Risk analysis Team communication Team coordination Status tracking Project reporting Personal measures Process discipline Estimating & planning Quality management Team Management Engineering Skills Team Disciplines A Productive Engineering Team

Introducing Engineering Disciplines Until they try it, most people do not believe disciplined engineering will work for them. They won’t try it without evidence. They can’t get evidence without trying it. The TSP engineering course is called the PSP (Personal Software Process). Engineers gather data while writing 10 programs. They start with their current methods. They advance in steps to the full PSP. They see from their personal data that sound engineering works. SM SM Personal Software Process and PSP are service marks of Carnegie Mellon University.

The Engineering Course Overview The course focuses on building engineering disciplines. to consistently meet commitments to measure and manage quality to understand personal performance The course instructors have all completed the engineering course themselves. The students must be competent software engineers.

Building the Team Environment Engineers take the PSP/TSP training as teams. This starts the teambuilding process. Shortly after completion, the trained teams launch their projects.

The Project Launch TSP projects can start on any phase. Each phase starts with a launch or relaunch step. The strategy is to overlap phases develop in increments balance team workload accelerate tasks wherever possible Postmortem Requirements Launch High-Level Design Relaunch Implementation Integration and Test

The TSP Launch Every TSP project starts with a launch. The launch takes 4 days. It is part of the project. It is led by a trained team coach. It immediately follows PSP training. In the launch, the engineers do essential project work.

The Launch Process Meetings Day 1 Day 2 Day 3 Day 4 1. Establish Product and Business Goals 4. Build Top- down and Next-Phase Plans 7. Conduct Risk Assessment 9. Hold Management Review 2. Assign Roles and Define Team Goals 5. Develop the Quality Plan 8. Prepare Management Briefing and Launch Report Launch Postmortem 3. Produce Development Strategy 6. Build Bottom- up and Consolidated Plans

Launch Products During the launch, the teams produce important products. documented team goals defined team-member roles an overall development strategy a defined team process a list of the project’s planned products size estimates for the planned products an overall project schedule a quantitative quality plan a detailed and balanced next-phase plan a prioritized evaluation of the project’s key risks

Building Motivated Teams While these products are important, the key launch objective is to build a motivated team. Motivated teams believe their goals are challenging, important, and achievable. To believe the goals are achievable, they must have a plan they are committed to meeting the skills and resources to do the job When teams make their own plans, they are committed to meeting them.

The Team Roles The roles distribute team management among the engineers. These roles establish responsibility for managing the teamworking environment. The members select their roles during the team launch. The standard roles cover planning, process, quality, support, user interface, design, implementation, and test.

Planning a TSP Project In the TSP, planning is done at three levels. The team first makes overall project and quality plans. Then, the team makes a detailed plan for the next project phase. The engineers next produce detailed personal plans for the following phase. Finally, the engineers balance their plans to evenly distribute the work and minimize the schedule.

TSP Planning Define Process and Tasks Estimate Defects Injected Produce Conceptual Design Make the Engineers’ Plans Balance Team Workload Estimate Tasks Make Size Estimates Estimate Yields Estimate Weekly Time Balanced Team Plans Conceptual Design Overall Project Plan Quality Plan Detailed Engineer Plans

Tracking a TSP Project The detailed team and individual plans facilitate precise project tracking. The team members regularly reassess project risks and devise mitigation actions. In weekly team meetings, the engineers report on task status review the key risks rebalance the workload The team produces weekly management status reports.

Engineer Task, Schedule, TSP Project Tracking Enter Time by Task Enter Week Task Completed Product Summary Task Status Engineer A Updated Team and Engineer Task, Schedule, and Quality Plans Team Task and Schedule Summary Enter Defects by Component and Phase Enter Size by Component Schedule Status Engineer A Quality Summary

Team Leader Support The Team Leader periodically reports to management on project status and risks. The team leader supports the team by obtaining staff and arranging for training protecting team resources interfacing with other groups resolving problems and issues maintaining process discipline overseeing process and product quality sustaining the team’s energy and drive

Management Support The PSP/TSP will not work unless the engineers are fully supported by all management levels. During the project launch the team reviews its plan with management. answers questions resolves issues explores alternatives To sustain the team’s motivation, management must periodically review the project probe the team’s data focus on quality

Engineer and Team Performance By learning TSP skills, engineers improve their personal performance. More important, however, the teams also build a defined process and working framework. They now have a common language and the ability to manage their work with facts and data. This combination produces extraordinary team performance.

TSP Experience Many TSP projects have been launched. Project size has ranged from 2 up to 50 team members. Products have been imbedded systems, real-time controllers, and commercial applications. Program size has ranged from a few hundred LOC to nearly half a million LOC. The following charts show some of the results to date.

PSP/TSP Change Behavior Many engineers have now been taught the required PSP/TSP skills. These skills both provide the foundation for effective teamwork and improve personal performance. The following data show the results from 298 engineers in PSP courses. In the course, the engineers write 10 programs. For program 1, they use their current practices. By program 10, they are practicing the PSP/TSP disciplines.

The PSP Course for Engineers Cyclic development Team Software Process Teambuilding Risk management Project planning and tracking PSP2.1 Design templates PSP2 Code reviews Design reviews PSP1.1 Task planning Schedule planning PSP1 Size estimating Test report PSP0.1 Coding standard Process improvement proposal Size measurement PSP0 Current process Basic measures

Effort Estimation Results Effort Estimation Accuracy Trend 0.2 0.3 0.4 0.5 0.6 0.7 Estimated Minutes – Actual Minutes/Estimated Minutes Mean Time Misestimation PSP Level Average This chart illustrates the change in average errors for estimating effort across the 10 PSP programs. The figure shows gradual improvement during the course of the training. The bold line, again, tracks the change across the 10 programs, and the dotted line is the pooled average for each PSP level. 11 10 9 8 7 6 5 4 3 2 1 Program Number © 2000 by Carnegie Mellon University Version # Productive Engineering Teams - page 32

Design Time Results Time Invested Per (New and Changed) Line of Code 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Mean Minutes Spent Per LOC Design Code Compile Test 11 10 9 8 7 6 5 4 3 2 1 Program Number

Quality Results Defects Per KLOC Removed in Compile and Test 10 20 30 40 50 60 70 80 90 100 110 120 Mean Compile + Test PSP Level Mean Comp + Test Mean Number of Defects Per KLOC We are using the number of defects found and removed in the Compile and Test Phases as one indicator of product quality. The bold line tracks the changes in average defects per KLOC for the 298 students across each of the 10 PSP assignments. The sample size varies from program to program (not shown in the chart yet). but there is a minimum of 100+ data points reflected in each mean. The dotted line was drawn by pooling all the individual defect densities by PSP level (program 10 is included in PSP level 2) and then computing an average. In this chart we see a dramatic decrease in the number of defects per thousand (new and changed) lines of code. These differences are statistically significant at the PSP level and across the 10 assignments (using a repeated measures Analysis Of Variance). 11 10 9 8 7 6 5 4 3 2 1 Program Number © 2000 by Carnegie Mellon University Version # Productive Engineering Teams - page 34

Productivity Results Lines of (New and Changed) Code Produced Per Hour of Total Development Time 20 22 24 26 28 30 Mean LOC/Hour Mean LOC/Hour PSP Level Average In this first productivity chart, we see that there is no CLEAR AND CONSISTENT change in productivity, compared to the previous figures. Take note of the scale on the vertical 11 10 9 8 7 6 5 4 3 2 1 Program Number © 2000 by Carnegie Mellon University © 2000 by Carnegie Mellon University Version # Version # Productive Engineering Teams - page 35 Productive Engineering Teams - page 35

Engineering Performance in Industry From the PSP/TSP course, engineers learn how to plan and track their personal work measure and manage product quality This significantly improves their performance when working alone. It also has a positive impact on their performance when working on a team.

Predictable Schedules Schedule Deviation Individual Value Control Chart - Commercial Systems 350 300 250 200 150 % Deviation 100 50 Overrun 112% to 37% to 5%. Industry data -- average overrun in 1998 was 121%. -50 01/88 01/89 01/90 01/91 01/92 01/93 01/94 01/95 01/96 01/97 01/98 -100 Pre-CMM CMM PSP -150 Date of Project Start Individual Data Points Mean Upper Natural Process Limit Lower Natural Process Limit One Standard Deviation

With Less Test Time, Productivity Improves with PSP

Team Performance in Industry Even with the proper training, engineers do not generally practice the TSP methods. To obtain the benefits of the TSP, organizations need to provide their engineers with professional training a defined teambuilding process a supportive team environment consistent management and coaching Under these conditions, engineering teams can be highly productive.

TSP Benefits (Boeing Pilot #1) # of Defect Software Size 2.36X more Sloc count # of Defect Detected 75% lower Defect Release # 6 Release # 7 Release # 8 Release # 9 PSP/TSP trained

TSP Benefits (Boeing Pilot #1) System Test time 2.36X more Sloc count 41 days System Test time 32 days 28 days 2.36X more Sloc count 94% less time 4 days Release # 6 Release # 7 Release # 8 Release # 9 PSP/TSP trained

TSP Team Task Hours Task hours directly produce products. Average Task Hours Per Week 18 16 15.1 14 13.3 12.6 12 Task Hours 10 9.6 8 6 4 Avg. Task Hours - Week Avg. Task Hours - Phase 2 04/20/1998 04/27/1998 05/04/1998 05/11/1998 05/18/1998 05/25/1998 06/01/1998 06/08/1998 06/15/1998 06/22/1998 06/29/1998 07/06/1998 07/13/1998 07/20/1998 07/27/1998 08/03/1998 08/10/1998 08/17/1998 08/24/1998 08/31/1998 09/07/1998 09/14/1998 09/21/1998 09/28/1998 10/05/1998 10/12/1998 10/19/1998 10/26/1998 11/02/1998 11/09/1998 11/16/1998 11/23/1998 11/30/1998 12/07/1998 12/14/1998 12/21/1998 12/28/1998 01/04/1999 01/11/1999 01/18/1999 01/25/1999 02/01/1999 02/08/1999 02/15/1999 02/22/1999 03/01/1999 03/08/1999 03/15/1999 03/22/1999 03/29/1999 04/05/1999 04/12/1999 04/19/1999 04/26/1999 05/03/1999 05/10/1999 05/17/1999 05/24/1999 05/31/1999 06/07/1999 06/14/1999 06/21/1999 06/28/1999 Task hours directly produce products. Task hours initially averaged under 10 per week. Task hours were improved through quiet times, process documentation, more efficient meetings, etc. © 2000 by Carnegie Mellon University Version # Productive Engineering Teams - page 42

Unbalanced Team Workload Engineer E G I 20 40 60 Schedule Weeks

Balancing Team Workload Unbalanced Engineer E Balanced G I 20 40 60 Schedule Weeks

A TSP Government Project Air Force flight planning system 6 software engineers 5 were PSP trained Integration and system test time was reduced from an average of 22% of schedule to 2.7%. Productivity was 2.33 times the same team’s prior project. Projects A B C TSP Size - LOC 67,291 7,955 86,543 25,820 Test Days 63 23 92 6 System test defects/KLOC 2.21 4.78 2.66 0.54 Acceptance test defects/KLOC N.A. 1.89 0.07 0.15

A TSP Industrial Project - 1 Communications test equipment 12 software and 3 hardware engineers not all software engineers PSP trained Plan Actual Size - KLOC 110 89.9 Effort - hours 16,000 14,711 Schedule - weeks 77 71 Defects per KLOC Integration test 1.0 0.2 System test 0.1 0.4 Field trial 0.0 0.02

A TSP Industrial Project - 2 Comparison with a prior project of 9,500 LOC. Engineers not PSP trained. Communications system controller TSP non-TSP Size - KLOC 89.9 9.5 Field testing duration engineering trial 2 weeks 3 months acceptance trial 3 weeks 6 months Trial defects/KLOC 0.02 5+

A TSP Maintenance Project A company applied TSP with 2-engineer teams on 14 maintenance projects. Average results showed. TSP non-TSP Time estimate errors +- 7.5% Not tracked Size estimate errors +- 10.0% +-23.4% Defects/page 0.14 0.35 Review yield 57.1% Not tracked Productivity - pages/hour 2.28 2.08

Getting Started with TSP Sprinkling a few PSP-trained engineers around an organization will not produce noticeable results. Installing TSP in an organization requires a team-based improvement focus careful planning senior management involvement and sponsorship a change in the behavior of both the engineers and their managers

The TSP Introduction Strategy The TSP introduction strategy has these steps: identify key areas for initial introduction hold executive kickoff and planning seminar identify and train affected managers train the engineers in teams train and authorize PSP instructors and TSP coaches establish needed support for pilot projects conduct PSP/TSP pilot projects plan and initiate rollout across the organization

Introducing the TSP Installing TSP in an organization can be done in a few months. It requires substantial training support from all involved managers management emphasis on completing PSP course work before the first team launch Training and transition support PSP courses are available from the SEI and its transition partners. Some university PSP courses are now offered. The TSP is only available from the SEI.

Preparation Activities An executive kickoff and planning seminar takes 1 1/2 days. Management training is 4 days. Engineer training: 125 - 150 hours of PSP training. teaches personal project management engineers learn to measure and manage quality required for effective team participation PSP/TSP coaches need additional training.

Example Introduction Timeline Example timeline based on 9-12 month pilot projects initial results are available in the first 6-12 months final results available within 12-18 months demonstrates costs and benefits with PSP/TSP rapidly builds high-performance teams Task Q1 Q2 Q3 Q4 Q5 Q6 Executive training/kickoff session X Select participants, develop schedule X Train managers, engineers, instructors X X X Conduct TSP pilots, train coaches X X Plan and initiate roll-out X

Conclusion The TSP shows engineers how to plan and direct their own work meet aggressive schedules produce superior products The TSP shows management how to build world-class teams lead and support these teams When properly trained and led, engineers can do extraordinary work.

For More Information Visit the PSP or TSP web sites http://www.sei.cmu.edu/psp/ http://www.sei.cmu.edu/tsp/ Contact a PSP transition partner http://www.sei.cmu.edu/collaborating/partners/trans.part.psp.html Contact SEI customer relations Software Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213-3890 Phone, voice mail, and on-demand FAX: 412/268-5800 E-mail: customer-relations@sei.cmu.edu

TSP Applications - Domain The PSP/TSP has been used in a wide range of application domains. TSP programs have also been developed for Windows, Unix, and imbedded systems. Automotive Avionics Communications control Display systems Financial Imbedded Internet Office systems Real time Spacecraft

TSP Applications - Size The size of TSP systems has varied widely. The smallest new programs of several hundred to a few thousand LOC have been developed by students. The largest programs have been for imbedded communications systems and integrated financial applications of 100,000 LOC up to about 500.000 LOC. Both small and large modifications have been made to very large legacy systems.

TSP Applications - Users Some of the organizations that are introducing the TSP are AIS Allied Signal Bahn Boeing Comnet EBS EDS Erickson Honeywell Iomega JPL Kaiser Litton SAIC SDRC Teradyne Trilogy United Defense USAF: Hill AFB USN: China Lake USN: NAVOCEANO Xerox