Holistic Release Criteria Sagar Bhagwat – Senior Manager

Holistic Release Criteria Numerous meetings, go/no-go discussions One question What’s the takeaway – are we good to go / on-track? Holistic Release Criteria PTC’s in-house developed framework Technical nitty-gritties off the view Single slide summary of health of the release

Licensed under the Creative Commons Attribution-Share Alike 4 Licensed under the Creative Commons Attribution-Share Alike 4.0 International license or available under the Creative Commons CC0 1.0 Universal Public Domain Dedication.

Holistic Release Criteria Identify quality parameters to measure Identify the current state (start point) for every measure Define the expected state / threshold (end point) to achieve for the release Project upside/downside during the development cycle Understand velocities of involved team/s Involve stakeholders Agree on start point, end point Identify measurable plan for time periods (week / sprint) over the development cycle Iteratively measure all the aspects of the release criteria Use the reported measurements for risk assessment Abiding to the release criteria is a team decision

Holistic Release Criteria PTC Financial Analyst Event, March 8, 2004 Holistic Release Criteria Product Quality Performance Scalability Serviceability Usability / Productivity Compatibility & Interoperability Security Reliability Production Readiness © Copyright 2004 Parametric Technology Corporation

Holistic Release Criteria – Measures Release criterion may contain one or more measures Measures within a release criterion may have equivalent weightage or varying weightage Measurement types Test Progress: % completion / coverage and % pass Trending to zero: end-point is zero, lower than plan (for time-periods) is better Open defects, Pending defects verifications Fixed (min.; higher the better): Fixed # as a minimum necessity Reliability (more than 1000 hrs. up time with user activity) Fixed (max.; lower the better): Fixed # as a maximum threshold Performance (less than 10 seconds for every UI action)

Holistic Release Criteria – Example Visit www.ptc.com 2-May-19 Holistic Release Criteria – Example Measure Unit End-point Open Unresolved Bugs Count Open Critical / Urgent Bugs Open Bug Verifications Testing (New Feature) Completion % 100% Pass % Testing (Regression) Documentation Review © Copyright 2000 Parametric Technology Corporation

Holistic Release Criteria – Example Week 04 Measure Unit Plan End-point Open Unresolved Bugs Count 12 Open Critical / Urgent Bugs 4 Open Bug Verifications 10 Testing (New Feature) Completion % 50% 100% Pass % 80% Testing (Regression) 20% 98% Documentation Review 0%

Holistic Release Criteria – Example Week 04 Measure Unit Plan Actual End-point Open Unresolved Bugs Count 12 14 Open Critical / Urgent Bugs 4 6 Open Bug Verifications 10 Testing (New Feature) Completion % 50% 30% 100% Pass % 80% 85% Testing (Regression) 20% 15% 98% 90% Documentation Review 0%

Holistic Release Criteria – 100 point score Consistent scale to depict release criteria progression over the release cycle Easy to understand without needing to know technical details Measurement weightage factored in calculating the score on 100-point scale 100-point score calculation formulae differ based on type of measurement *Credits – Kevin Hedrick & Amol Patil, PTC, Inc. USA

Holistic Release Criteria – 100 point score 100 point score calculation formulae by measurement type Test Progress: % completion / coverage and % pass Score = (% completion / coverage) X (% pass) = (aka success rate) Trending to zero: end-point is zero, lower than plan is better If count is zero, then score = 100 If count >= max planned count across time periods, then score = 0 Else, score = [(max planned count – current count) / (max planned count)]*100 Fixed (min.; higher the better): Fixed # as a minimum necessity If count is >= planned end-point, then score = 100 If count is zero, then score = 0 Else, score = (current count / planned end-point)*100 Fixed (max.; lower the better): Fixed # as a maximum threshold If count is <= planned end-point, then score = 100 If count is > planned end-point, then score = 0

Holistic Release Criteria – 100 point score Calculate 100 point score for every measure of the release criterion Apply weightage of every measure / use simple average to decide the overall score for the release criterion Calculate the overall score for plan and actual for every time-period

Holistic Release Criteria – 100 point score Week 04 Measure Unit Plan Actual Score End-point Projection Open Unresolved Bugs Count 12 14 40 30 20 Open Critical / Urgent Bugs 4 6 60 10 Open Bug Verifications 50 70 Testing (New Feature) Completion % 50% 30% 26 100% Pass % 80% 85% Testing (Regression) 20% 15% 98% 90% Documentation Review 0% Release Criteria Score 35 100

Holistic Release Criteria – Colors On Track Mitigated At Risk Not Started Ready for Release Recommend to Release Release criteria tiles depiction Green On track No roadblocks to meet the plan (for time period / end-point) as expected Yellow Issues / roadblocks and solutions / approaches for those are identified Behind plan but can be back on track with mitigation plans in place Red Issues / roadblocks identified with no immediate mitigation plan in place Risk to schedule milestones and possibly release date Blue Measurement not started

Holistic Release Criteria – Colors On Track Mitigated At Risk Not Started Ready for Release Recommend to Release Release criteria tiles depiction Measurement meets the end-point definition & ready for release Measurement does not meet the end-point definition due to external factors Recommend to release with known issues

Holistic Release Criteria – Dashboard On Track Mitigated At Risk Not Started Ready for Release Recommend to Release Plan: 35 Actual: 30 Plan: 100 Actual: 100 Plan: 100 Actual: 90 Application Product Quality Security Compatibility & Interoperability Plan: 77 Actual: 52 Plan: 50 Actual: 30 Plan: 80 Actual: 63 Infrastructure Performance Scalability Reliability Plan: 90 Actual: 90 Plan: 100 Actual: 90 Plan: 90 Actual: 86 Customer Focus Serviceability Usability Production Readiness

Holistic Release Criteria Continuous Improvement Add new criteria based on release content Tighten / strengthen criteria based on results Update criteria based on customer feedback

@Sagar Bhagwat, PMP Senior manager from Quality Engineering practice Working in this field for last 13+ years Instrumental with delivery of Enterprise products Currently leads R&D (QA) team of ThingWorx Platform (IoT) and ThingWorx Studio (AR) in PTC, Inc.

Thank You