1. CS 577b Software Engineering II -- Introduction
8 September 2018
Technical Debt
CS 577 Software Engineering
Barry Boehm
Supannika Koolmanojwong
© USC Center for Software Engineering

2. What are the cost of a software?
Think about the whole life cycle

3. Total cost of ownership
a financial estimate whose purpose is to help consumers and enterprise managers determine direct and indirect costs of a product or system.
including the costs to research, develop, acquire, own, operate, maintain, and dispose of a system

4. Potential total cost of ownership
Computer hardware and programs
Operation expenses
Long term expenses
HW and SW
Network
Server
Workstation
Installation & integration
Purchasing research
Warranties and licenses
License tracking - compliance
Migration expenses
Risks: susceptibility to vulnerabilities, availability of upgrades, patches and future licensing policies, etc.
Infrastructure
Electricity
Testing costs
Downtime, outage and failure expenses
Diminished performance
Security (including breaches, loss of reputation, recovery & prevention)
Backup and recovery process
Technology training
Audit
Insurance
IT personnel
Replacement
Future upgrade or scalability expenses
Decommissioning

5. Relative* Total Ownership Cost (TOC) For single system life cycle (TOC-SS)
~5% architecture investment
~5% architecture investment
~25% architecture investment
* Cumulative architecting and rework effort relative to initial development effort

6. Technical Debt Code Debt Deficit programming Software Decay CodeSmell
coined by Ward Cunningham
Design Debt
Technical
Inflation

7. Examples of Technical Debt

8. What is Technical Debt?
"Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite... The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise.”
— Ward Cunningham, 1992

9. Outline Examples of technical debt situations
Major causes of technical debt
Ways to identify technical debt
Ways to address and avoid technical debt

10. Technical Debt Example 1
“I don’t know what happened, I just changed one line”
“We can’t upgrade, It will break”
“We can’t upgrade the code, we don’t have time”
“We can’t upgrade the code, no one understands it”
“Just put in the comment XXX, we will do it later”
“Just put in the TODO comment”

11. Technical Debt Example 2 Wrong architecture or NDI choices
Lack of needed scalability, reliability, performance
Incompatible Non-Developmental Items (NDI)
Architectural style clashes cause crashes
You don’t find out until too late (operations phase)
No choice but to start again or rewrite big chunk to keep it working
Read more: 

12. Technical Debt Example 3 Error-prone code
“ .. If I change X, it is going to break Y, I think ..”
“ Don’t touch that code, last time we did, we spent a week fixing it…”
20% of the code where 80% of bugs are found
Hard to understand
Dangerous to change because done poorly one in the first place
Not rewriting this code is one of the most expensive mistakes that developers make
Read more: 

13. Technical Debt Example 4 Not easily tested
“ .. I thought we had a test for that ..”
Don’t have good automated tests
Tests keep falling apart when you change the code
Testing costs tend to go up over time as you write more code
Read more: 

14. Technical Debt Example 4 Code that mysteriously works
Nobody is sure how or why
Might be written by the geek who left the company
If nobody on the team understands it, it’s a time bomb
Read more: 

15. Technical Debt: More Examples
Forward and backward compatibility
Short term debt
Duplicate, copy-and-paste code
How many ? Trackable ?
Hard coding
Out of date documentation
“We just lost the drive, where are the backups?”
If nobody is using it, get rid of it. If people are using it, why isn’t it up to date?
Read more: 

16. Development Cost(Perfect World)
% Effort per Phase
Analysis
Design
Implementation
Test
Integration

17. Classification of Defects
Analysis
Design
Implementation
Test
Integration
Defect Insertion
Defect detection & Removal
Defects
Typical Defect Profiles
Implementation
Defects

18. Development Cost(Real World)
% Effort per Phase
Analysis
Design
Implementation
Test
Integration

19. Development Cost % Effort per Phase Analysis Design Implementation
Test
Integration
Real world
Perfect World

20. Development Cost % Effort per Phase
Technical
Debt?
Technical
Debt?
% Effort per Phase
Technical
Debt?
Not enough information in this diagram to determine whether these are technical debt. The dip in the middle could represent technical debt from trying to move things along, but paying for it in integration and test.
Technical
Debt?
Analysis
Design
Implementation
Test
Integration
Real world
Perfect World

21. Development Cost % Effort per Phase Technical Debt? Technical Debt?
Analysis
Design
Implementation
Test
Integration
Real world
Perfect World

22. COTS Integration % Effort per Phase
This one shows that a decision must have been made that there was no real need to perform the System Definition activities. This represents Technical Debt.
Technical
Debt?
Analysis
Design
Implementation
Test
Integration
Real world
Perfect World

23. Outline Examples of technical debt situations
Major causes of technical debt
Ways to identify technical debt
Ways to address and avoid technical debt

24. Major Causes of Technical Debt
Conspiracy of Optimism
Neglect of ICSM Principles
Stakeholder value-based guidance
Incremental commitment and accountability
Concurrent system engineering
Evidence and risk-driven decisions
Business pressures
Easiest-first; neglecting rainy day use cases
Delayed Refactoring

25. The Conspiracy of Optimism and The Cone of Uncertainty
Copyright © USC-CSSE
October 16, 2012

26. Assumption of Stability vs
Assumption of Stability vs. Rapid Change – Need evolutionary/incremental vs. one-shot development
Uncertainties in competition, technology, organizations, mission priorities
There is Another Cone of Uncertainty: Shorter increments are better
Uncertainties in competition and technology evolution and changes in organizations and mission priorities, can wreak havoc with the best of system development programs. In addition, the longer the development cycle, the more likely it will be that several of these uncertainties or changes will occur and make the originally-defined system obsolete. Therefore, planning to develop a system using short increments helps to ensure that early, high priority capabilities can be developed and fielded and changes can be more easily accommodated in future increments.
Copyright © USC-CSSE
October 16, 2012

27. Major Causes of Technical Debt
Conspiracy of Optimism
Neglect of ICSM Principles
Stakeholder value-based guidance
Incremental commitment and accountability
Concurrent system engineering
Evidence and risk-driven decisions
Business pressures
Easiest-first; neglecting rainy day use cases
Delayed Refactoring

28. http://pkruchten. files. wordpress

29. Cost Growth of Delaying Rework
06/25/07
©USC-CSSE

30. Outline Examples of technical debt situations
Major causes of technical debt
Ways to identify technical debt
Ways to address and avoid technical debt

31. Ways to Identify Technical Debt
Top 10 inhibitors
Not addressing major causes of technical debt
Cost, schedule overrruns
Neglected stakeholders
Lack of feasibility evidence
Delayed refactoring
Neglecting rainy day use cases
Technical debt tools: SONAR, CAST, SQALE

32. Sonar Mainly manage the software quality
SQALE – Software Quality Assessment based on Lifecycle Expectations

33. Nemo - An open source tool – by sonar http://nemo.sonarqube.org

34. Sonar tool Bad Distribution of Complexity Duplications

35. Sonar tool Lack of Unit Tests No Coding Standards

36. Sonar tool
Not Enough or Too Many Comments (time consuming maintenance)
Documented complex or not straightforward pieces of code
Readable unit tests scenarios that will help the developers understand the code
Documented API that will help external developers understand how to use those API
Respect of coding standard such as naming conventions
Potential Bugs

37. Top 10 enablers – Similarities causes of technical debt
A New Single System
An Existing Single System
A System of Systems
Requirements Volatility
Lack of Interoperability
Unprecedentedness
High numbers of external interfaces
Lack of / incompatible standard & protocol
Delayed authority to proceed/start with fixed milestone
Infeasible schedule/staffing profile
Vague Requirements
Lack of Domain Experience
Embedded poor quality software
Technology Volatility
Conflicting Stakeholders
Inability to test across systems
Under average people / Personnel Capability
Technical debt
Technology Immaturity
Interoperability / compatibility

38. Top 10 enabler– Differences
A New Single System
An Existing Single System
A System of Systems
Requirements Volatility
Lack of Interoperability
Unprecedentedness
High numbers of external interfaces
Lack of / incompatible standard & protocol
Delayed authority to proceed/start with fixed milestone
Infeasible schedule/staffing profile
Vague Requirements
Lack of Domain Experience
Embedded poor quality software
Technology Volatility
Conflicting Stakeholders
Inability to test across systems
Under average people / Personnel Capability
Technical debt
Technology Immaturity
Interoperability / compatibility

39. Technical Debt Sources, Cost: SONAR
(in man days)
cost_to_fix_duplications +  cost_to_fix_violations +  cost_to_comment_public_API +  cost_to_fix_uncovered_complexity +  cost_to_bring_complexity_below_threshold +  cost_to_cut_cycles_at_package_level
Average $3.61 to $5.42 per line of code.

40. Technical Debt Observations “Agile Project Management”, Jim Highsmith, second edition

41. Outline Examples of technical debt situations
Major causes of technical debt
Ways to identify technical debt
Ways to address and avoid technical debt

42. Fixing technical debt Big Bang Dedicated Team Boy Scout
no new features for a year? Really? 
spend some time cleaning up the mess
Good ?
Dedicated Team
Have another team dedicated
Good ? 80/20 rule ?
Boy Scout
remove technical debt little and often
If no tests, add some. If poor test, improve them. If bad code, refactor it
The boy scout rule – leave the camp cleaner than you found it

43. Ways to Address and Avoid Technical Debt
Top 10 enablers
Following ICSM Principles
Stakeholder value-based guidance
Incremental commitment and accountability
Concurrent system engineering
Evidence and risk-driven decisions
Use technical debt tools: SONAR, CAST, SQALE

44. Top 10 Inhibitors- Similarities Help mitigating technical debt
A New Single System
An Existing Single System
A System of Systems
Rapid Prototyping
Target hardware lab / test like you fly & simulation
Customer /tech requirements flexibility
Incremental test and feedback
Incremental Delivery & feedback
Flexible / Tailorable rules
Agile/lean approach
Common standard and protocol
Decision making authority
Reusing assets
Best people / personnel capability
Common standard, interface
Tools and automation
Less context switching when doing multiple projects
Domain knowledge
Best people / Personnel Capability
Understanding of the existing system and interfaces
Team cohesion

45. Top 10 Enablers – Differences Help mitigating technical debt
A New Single System
An Existing Single System
A System of Systems
Rapid Prototyping
Target hardware lab / test like you fly & simulation
Customer /tech requirements flexibility
Incremental test and feedback
Incremental Delivery & feedback
Flexible / Tailorable rules
Agile/lean approach
Common standard and protocol
Decision making authority
Reusing assets
Best people / personnel capability
Common standard, interface
Tools and automation
Less context switching when doing multiple projects
Domain knowledge
Best people / Personnel Capability
Understanding of the existing system and interfaces
Team cohesion

46. Addressing Enablers and Inhibitors
Best people
Under average people
Decision making authority
Lack of decision making authority
Team Cohesion
Conflicting stakeholders
Incremental Test and Feedback
Inability to test across systems

47. The Bottom Line Technical debt often a good practice
Meeting market windows
Prototyping to determine user needs, satisfaction
Short-term fixes, targets of opportunity
But needs pay-down later
To avoid mounting debt and interest
Need balanced investment in fixes, new features

48. FCR ARB
Post all materials to your team website before your presentation
All team members must be available
Don’t forget to dry run
Print out 2 copies of your presentation for your clients
(2 slides per page)
Check EC-17 for more info
No class on 10/12, 10/14, and 10/16
Mon 10/12
Wed 10/14
Fri 10/16
Mon 10/19
12:30 pm – 1:50 pm
Team 06
2:00 PM - 3:20 PM
Team 04
3:30 PM - 4:50 PM
Team 05
Team 07
5:00 PM - 6:20 PM
Team 02
Team 03
Team 01 (5:30PM)

49. References

50. Technical Debt Cost Estimation
Principal = Repair Effort
Interest = Maintenance Effort
[Nugroho et al 2011]
[Curtis et al 2012]
Debt
(in man days)
cost_to_fix_duplications +  cost_to_fix_violations +  cost_to_comment_public_API +  cost_to_fix_uncovered_complexity +  cost_to_bring_complexity_below_threshold +  cost_to_cut_cycles_at_package_level
Ref: Nugroho et al 2011, An Empirical Model of Technical Debt and Interest, MTD 2011
Ref: Curtis e al 2011, Estimating the Principal of an Application's Technical Debt. IEEE Software

51. No sample less than 10KSLOC
700 applications
158 organizations
357 MLOC
No sample less than 10KSLOC
Conservative - Assume all violations would be fixed within one hour
Weighted - varied the hours needed for fixing within each severity category,
More realistic -
based on data observed in several IT
organizations.
Ref: Curtis e al 2011, Estimating the Principal of an Application's Technical Debt. IEEE Software

52. Average Technical Debt per LOC = $3.61
Ref: Curtis e al 2011, Estimating the Principal of an Application's Technical Debt. IEEE Software

53. Ref: Curtis e al 2011, Estimating the Principal of an Application's Technical Debt. IEEE Software

54. Technical Debt within each technology

55. Sonar Mainly manage the software quality
SQALE – Software Quality Assessment based on Lifecycle Expectations

56. Nemo - An open source tool – by sonar http://nemo.sonarqube.org

57. Sonar – SQALE plug-in http://www. sonarsource

58. Developers’ 7 Deadly Sins
Bad Distribution of Complexity
Duplications
Lack of Unit Tests
No Coding Standards
Not Enough or Too Many Comments
Potential Bugs
Spaghetti Design

59. Developers’ 7 Deadly Sins
Bad Distribution of Complexity
Duplications

60. Developers’ 7 Deadly Sins
Lack of Unit Tests
No Coding Standards

61. Developers’ 7 Deadly Sins
Not Enough or Too Many Comments (time consuming maintenance)
Documented complex or not straightforward pieces of code
Readable unit tests scenarios that will help the developers understand the code
Documented API that will help external developers understand how to use those API
Respect of coding standard such as naming conventions
Potential Bugs

62. Developers’ 7 Deadly Sins
Spaghetti Design
Hunting for cycles
Checking your own architecture rules
Managing your libraries
Checking the single responsibility principle with LCOM4 (Lack of Cohesion of Methods)
RFC (Response for Class) – checking coupling (complexity of class in terms of method calls)

63. x = = x

64. In-Class 5 From your team project Be Specific
Identify 1 technical debt that your team has
What is the enabler of this technical debt?
How do you plan to mitigate this technical debt?
Be Specific
Individual work, do not discuss with others
Off-campus : Due Monday Oct 12. D2L