1. Holistic Release Criteria
Sagar Bhagwat – Senior Manager

2. Holistic Release Criteria
Numerous meetings, go/no-go discussions
One question
What’s the takeaway – are we good to go / on-track?
Holistic Release Criteria
PTC’s in-house developed framework
Technical nitty-gritties off the view
Single slide summary of health of the release

3. Licensed under the Creative Commons Attribution-Share Alike 4
Licensed under the Creative Commons Attribution-Share Alike 4.0 International license or available under the Creative Commons CC0 1.0 Universal Public Domain Dedication.

4. Holistic Release Criteria
Identify quality parameters to measure
Identify the current state (start point) for every measure
Define the expected state / threshold (end point) to achieve for the release
Project upside/downside during the development cycle
Understand velocities of involved team/s
Involve stakeholders
Agree on start point, end point
Identify measurable plan for time periods (week / sprint) over the development cycle
Iteratively measure all the aspects of the release criteria
Use the reported measurements for risk assessment
Abiding to the release criteria is a team decision

5. Holistic Release Criteria
PTC Financial Analyst Event, March 8, 2004
Holistic Release Criteria
Product Quality
Performance
Scalability
Serviceability
Usability / Productivity
Compatibility & Interoperability
Security
Reliability
Production Readiness
© Copyright 2004 Parametric Technology Corporation

6. Holistic Release Criteria – Measures
Release criterion may contain one or more measures
Measures within a release criterion may have equivalent weightage or varying weightage
Measurement types
Test Progress: % completion / coverage and % pass
Trending to zero: end-point is zero, lower than plan (for time-periods) is better
Open defects, Pending defects verifications
Fixed (min.; higher the better): Fixed # as a minimum necessity
Reliability (more than 1000 hrs. up time with user activity)
Fixed (max.; lower the better): Fixed # as a maximum threshold
Performance (less than 10 seconds for every UI action)

7. Holistic Release Criteria – Example
Visit
2-May-19
Holistic Release Criteria – Example
Measure
Unit
End-point
Open Unresolved Bugs
Count
Open Critical / Urgent Bugs
Open Bug Verifications
Testing (New Feature)
Completion %
100%
Pass %
Testing (Regression)
Documentation Review
© Copyright 2000 Parametric Technology Corporation

8. Holistic Release Criteria – Example
Week 04
Measure
Unit
Plan
End-point
Open Unresolved Bugs
Count 12
Open Critical / Urgent Bugs 4
Open Bug Verifications 10
Testing (New Feature)
Completion %
50%
100%
Pass %
80%
Testing (Regression)
20%
98%
Documentation Review 0%

9. Holistic Release Criteria – Example
Week 04
Measure
Unit
Plan
Actual
End-point
Open Unresolved Bugs
Count 12 14
Open Critical / Urgent Bugs 4 6
Open Bug Verifications 10
Testing (New Feature)
Completion %
50%
30%
100%
Pass %
80%
85%
Testing (Regression)
20%
15%
98%
90%
Documentation Review 0%

10. Holistic Release Criteria – 100 point score
Consistent scale to depict release criteria progression over the release cycle
Easy to understand without needing to know technical details
Measurement weightage factored in calculating the score on 100-point scale
100-point score calculation formulae differ based on type of measurement
*Credits – Kevin Hedrick & Amol Patil, PTC, Inc. USA

11. Holistic Release Criteria – 100 point score
100 point score calculation formulae by measurement type
Test Progress: % completion / coverage and % pass
Score = (% completion / coverage) X (% pass) = (aka success rate)
Trending to zero: end-point is zero, lower than plan is better
If count is zero, then score = 100
If count >= max planned count across time periods, then score = 0
Else, score = [(max planned count – current count) / (max planned count)]*100
Fixed (min.; higher the better): Fixed # as a minimum necessity
If count is >= planned end-point, then score = 100
If count is zero, then score = 0
Else, score = (current count / planned end-point)*100
Fixed (max.; lower the better): Fixed # as a maximum threshold
If count is <= planned end-point, then score = 100
If count is > planned end-point, then score = 0

12. Holistic Release Criteria – 100 point score
Calculate 100 point score for every measure of the release criterion
Apply weightage of every measure / use simple average to decide the overall score for the release criterion
Calculate the overall score for plan and actual for every time-period

13. Holistic Release Criteria – 100 point score
Week 04
Measure
Unit
Plan
Actual
Score
End-point
Projection
Open Unresolved Bugs
Count 12 14 40 30 20
Open Critical / Urgent Bugs 4 6 60 10
Open Bug Verifications 50 70
Testing (New Feature)
Completion %
50%
30% 26
100%
Pass %
80%
85%
Testing (Regression)
20%
15%
98%
90%
Documentation Review 0%
Release Criteria Score 35
100

14. Holistic Release Criteria – Colors
On Track
Mitigated
At Risk
Not Started
Ready for Release
Recommend to Release
Release criteria tiles depiction
Green
On track
No roadblocks to meet the plan (for time period / end-point) as expected
Yellow
Issues / roadblocks and solutions / approaches for those are identified
Behind plan but can be back on track with mitigation plans in place
Red
Issues / roadblocks identified with no immediate mitigation plan in place
Risk to schedule milestones and possibly release date
Blue
Measurement not started

15. Holistic Release Criteria – Colors
On Track
Mitigated
At Risk
Not Started
Ready for Release
Recommend to Release
Release criteria tiles depiction
Measurement meets the end-point definition & ready for release
Measurement does not meet the end-point definition due to external factors
Recommend to release with known issues

16. Holistic Release Criteria – Dashboard
On Track
Mitigated
At Risk
Not Started
Ready for Release
Recommend to Release
Plan: 35
Actual: 30
Plan: 100
Actual: 100
Plan: 100 Actual: 90
Application
Product Quality
Security
Compatibility & Interoperability
Plan: 77
Actual: 52
Plan: 50 Actual: 30
Plan: 80 Actual: 63
Infrastructure
Performance
Scalability
Reliability
Plan: 90 Actual: 90
Plan: 100
Actual: 90
Plan: 90 Actual: 86
Customer Focus
Serviceability
Usability
Production Readiness

17. Holistic Release Criteria Continuous Improvement
Add new criteria based on release content
Tighten / strengthen criteria based on results
Update criteria based on customer feedback

18. @Sagar Bhagwat, PMP Senior manager from Quality Engineering practice
Working in this field for last 13+ years
Instrumental with delivery of Enterprise products
Currently leads R&D (QA) team of ThingWorx Platform (IoT) and ThingWorx Studio (AR) in PTC, Inc.

19. Thank You