EMI INFSO-RI Software Metric Definitions, Reports and Analysis in EMI Authors: Eamonn Kenny (TCD), Gianni Pucciani (CERN) Date: Tuesday 12 th April

EMI INFSO-RI Measure quality/quantity of the software. Encourage conformance between the 4 middleware groups. Highlight previously unlooked at problems (e.g: using static analysers) 2 Why do we need metrics?

EMI INFSO-RI To measure: – Performance: e.g: How long to fix a bug? – Complexity: e.g: How many fixed bugs? – Support: e.g: How complex is a component? E.g: Cyclomatic Complexity and SLOC EMI is funded to provide quality software and to measure its quality With metrics one can see trends and understand what needs to be improved and how many defects a component has. 3 Why do we need metrics?

EMI INFSO-RI Metrics Role in EMI SA2 4 Area Leaders Product Teams (PT) SA2.3 Metrics SA2.2 QA Plan SA2.4 Tools SA2.5 QA Review Top 3 Metrics?Existing Middleware Metrics? Tool Definition Verification SA1 Quality Control Evaluation Definitions & Analysis EMT Weekly Meetings SA2 SA1 JRA1 Report Generation SA2.1 Coordination PEB link Survey Weekly report Define & Structure

EMI INFSO-RI Model for Metric Collection Practical metrics, not a theoretical set! Governed by the end goals of the project End-Goals Suggests a Question Leads to a Metric

EMI INFSO-RI Metric Template Metrics Id E.g: PriorityBugs Name Description Measurement Calculation Mathematical formula Input(s) & Units E.g: time range in days Output(s) & Units E.g: Average time in hours Scope Thresholds/Target Value Tools Availability Per middleware availability Goals Quality factor Follows the McCall factors Risks Special Notes

EMI INFSO-RI Metrics Description Example 7

EMI INFSO-RI Areas of CoverageTypes of Metrics Process Management (Bug-tracking related) Relates to priority, severity, open/closed bugs, state changes in bugs, improving turnaround times External:Quality in Use (Bug-tracking/EGI) 3 rd level GGUS related metrics (KPIs for SA1) Product Team software - Static Analysers Language specific analysers, SLOC, CLOC, Cyclomatic Complexity, etc. Product Team software - Testing, platform, etc. Unit tests, supported platforms, bug density Optional Add-on ToolsValgrind/Helgrind (Memory leak & thread checking) 8 Metric Categories

EMI INFSO-RI Average Time To Close a Bug (PriorityBugs, BugSeverityDistribution) Time To Fix a Bug (SA1-QC) Bug-tracking Process 9 Open Priority Bugs Open Is Accepted? YES Accepted Fixed Can I test it? Test successful? Not Tested Closed Rejected NO YES NO YES Suggested Metrics Untouched Open Bugs (DSA 1.1 requirement)

EMI INFSO-RI Exposed XML to ETICS (run anytime) Middleware Bug- Mapping (needs periodic intervention) Middleware Bug- tracking (Snapshot taken daily) 10 Handling Multiple Bug-Trackers ARC Bugzilla dCache Request Tracker (RT) gLite Savannah UNICORE SourceForge Multiple bug-trackers One common interface

EMI INFSO-RI Metric NameDesignedImplementedRecipient Open Priority Bugs (High/Immediate) EMT/Reports Successful Builds metric Open Untouched Bugs (> 14 days) Fixed Bugs SA1-QC Priority Bugs (High/Immediate) Periodic Reports/ Deliverables/ Product Team Reports Bug Severity Distribution Backlog Management Index Integration test effectiveness metric Delay on release schedule metric Up-to-date documentation metric 11 SQAP/Bug-tracking Metrics

EMI INFSO-RI Fixed Bugs (SA1-QC) Required for assessing the number of regression tests produced in association with each fixed bug.

EMI INFSO-RI Open Untouched Bugs (weekly EMT) Report from 10 th March 2011 Inter-quartile ranges used in visualization

EMI INFSO-RI Successful Builds Metric (for EMT) Highlight: product teams having failures due to their own internal issues Highlight: product teams causing other product teams to fail

EMI INFSO-RI Backlog Management Index Track whether product teams have an increasing backlog or decreasing backlog of bugs or defects?

EMI INFSO-RI Metric NameDesignedImplementedRecipient Unit test coverage metric Reports/ Deliverables/ Periodic PT reports Number of supported Platforms Total bug density Bug density per release Cyclomatic complexity C/C++ metrics – CCCC, cppcheck Java metrics - FindBugs, PMD, Checkstyle Python metrics - pylint Code commenting metrics 16 Product related Metrics

EMI INFSO-RI Static Analysers: Java – FindBugs (PT) Project Overview - Results per Product team Finer Grained Reporting for Product Teams

EMI INFSO-RI Source Lines of Code Source lines of code is particularly important when assessing the Bug Density Distribution (i.e: Bugs open per lines of codes)

EMI INFSO-RI More Static Analysers 19

EMI INFSO-RI The metrics are useful and we defined them for the whole of EMI: – There is now a common framework for producing reports for the EMT, deliverables and Product Teams (PTs). – The reporting structure is interpretable at the project level, activity level & Product Teams level. – Static analysers produce reports highlighting problems previously not seen by developers. 20 Current Status

EMI INFSO-RI We do have an objective way to measure performance/complexity/support. EMI can provide quality software and measure its quality. With the current metric reports one can see trends and understand what needs to be improved and how many defects a component has. 21 Conclusions

EMI INFSO-RI A few important metrics are currently being implemented. The metrics will be reassessed and evaluated after the EMI-1 release. 22 Future Work