(1) Cam Moore Collaborative Software Development Laboratory Communication & Information Sciences University of Hawaii, Manoa

Slides:



Advertisements
Similar presentations
Requirements gathering
Advertisements

Maintenance Forecasting and Capacity Planning
This material is approved for public release. Distribution is limited by the Software Engineering Institute to attendees. Sponsored by the U.S. Department.
Software Engineering Lab Session Session 4 – Feedback on Assignment 1 © Jorge Aranda, 2005.
PRO2 - 1 Introduction to the Personal Software Process SWENET PRO2 Module Developed with support from the National Science Foundation.
ISERN Meeting Tuesday, October 10, 2000 Facilitators: Carolyn B. Seaman Reidar Conradi Qualitative Methods in Software Engineering Research.
Copyright 2000, Stephan Kelley1 Estimating User Interface Effort Using A Formal Method By Stephan Kelley 16 November 2000.
Chapter 14: Usability testing and field studies. 2 FJK User-Centered Design and Development Instructor: Franz J. Kurfess Computer Science Dept.
Important concepts in software engineering The tools to make it easy to apply common sense!
USABILITY AND EVALUATION Motivations and Methods.
Software Quality Metrics
CSCD 555 Research Methods for Computer Science
Amirkabir University of Technology, Computer Engineering Faculty, Intelligent Systems Laboratory,Requirements Engineering Course, Dr. Abdollahzadeh 1 Effectiveness.
The reform of A level qualifications in the sciences Dennis Opposs SCORE seminar on grading of practical work in A level sciences, 17 October 2014, London.
Data Analysis Statistics. Inferential statistics.
How to Measure the Impact of Specific Development Practices on Fielded Defect Density.
Project Workshops Results and Evaluation. General The Results section presents the results to demonstrate the performance of the proposed solution. It.
درس :مهندسي نيازمندي ها استاد: دكتر عبداله زاده دانشجو: خيرالنسا مرچانت Effectiveness of Elicitation Techniques in Distributed Requirement Engineering.
Major Outcomes of Science Instruction
Formulating the research design
Science and Engineering Practices
12 Steps to Useful Software Metrics
Using A Defined and Measured Personal Software Process Watts S. Humphrey CS 5391 Article 8.
1 Prediction of Software Reliability Using Neural Network and Fuzzy Logic Professor David Rine Seminar Notes.
CS 4001Mary Jean Harrold 1 Using Evidence Effectively.
A Survey on Software Cost Estimation in the Chinese Software Industry Da Yang, Qing Wang, Mingshu Li, Ye Yang, Kai Ye, and Jing Du Institute of Software,
Business and Management Research
(1) Project LEAP: A “Personal Information Environment” for Software Engineers Philip Johnson Cam Moore Collaborative Software Development Laboratory University.
Copyright (c) 2003 CPTTM 1 Common fears of a software development manager Common fears of a software development manager: –Deadline.
Chapter 6 : Software Metrics
Understanding MYP Criteria
An-Najah National University Faculty Of Engineering Industrial Engineering Department Implementation Of Quality Function Deployment On Engineering Faculty.
SE 501 Software Development Processes Dr. Basit Qureshi College of Computer Science and Information Systems Prince Sultan University Lecture for Week 6.
Qualifications Update: Higher Accounting Crowne Plaza Glasgow 10 December 2013 Qualifications Update: Higher Accounting Crowne Plaza Glasgow 10 December.
CHAPTER 1 LESSON 3 Math in Science.
Data Collection. The first state of the PSM is analysis where an organisation and its systems are examined to determine the nature of the problem to be.
Research Project Statistical Analysis. What type of statistical analysis will I use to analyze my data? SEM (does not tell you level of significance)
Math in Science.  Estimate  Accuracy  Precision  Significant Figures  Percent Error  Mean  Median  Mode  Range  Anomalous Data.
CHAPTER 1 LESSON 3 & 4 MATH IN SCIENCE + GRAPHS. WHAT ARE SOME MATH SKILLS USED IN SCIENCE? SOME MATH SKILLS USED IN SCIENCE WHEN WORKING WITH DATA INCLUDE.
Lecture 4 Software Metrics
INFO 636 Software Engineering Process I Prof. Glenn Booker Week 9 – Quality Management 1INFO636 Week 9.
The Art of Estimating Programming Tasks Adriana Lopez Development Director Dragon Age II.
Achieving High Software Reliability Using a Faster, Easier and Cheaper Method NASA OSMA SAS '01 September 5-7, 2001 Taghi M. Khoshgoftaar The Software.
Copyright  2003 by Dr. Gallimore, Wright State University Department of Biomedical, Industrial Engineering & Human Factors Engineering Human Factors Research.
Disciplined Software Engineering Lecture #3 Software Engineering Institute Carnegie Mellon University Pittsburgh, PA Sponsored by the U.S. Department.
TESTING LEVELS Unit Testing Integration Testing System Testing Acceptance Testing.
This material is approved for public release. Distribution is limited by the Software Engineering Institute to attendees. Sponsored by the U.S. Department.
Chapter 8 Usability Specification Techniques Hix & Hartson.
By James Miller et.all. Presented by Siv Hilde Houmb 1 November 2002
SOFTWARE METRICS. Software Process Revisited The Software Process has a common process framework containing: u framework activities - for all software.
Chapter 3: Software Project Management Metrics
(1) Cam Moore Collaborative Software Development Laboratory Communication & Information Sciences University of Hawaii, Manoa
1 f02laitenberger7 An Internally Replicated Quasi- Experimental Comparison of Checklist and Perspective-Based Reading of Code Documents Laitenberger, etal.
Using Bayesian Nets to Predict Software Defects in Arbitrary Software Lifecycles Martin Neil Agena Ltd London, UK Web:
Advanced Software Engineering Lecture 4: Process & Project Metrics.
Software Project Management
Metrics "A science is as mature as its measurement tools."
(6) Estimating Computer’s efficiency Software Estimation The objective of Software Estimation is to provide the skills needed to accurately predict the.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 1 Research: An Overview.
Laboratory Investigations Each lab group will submit a single input. All members of the group will get THE SAME grade UNLESS... You are observed goofing.
If you have a transaction processing system, John Meisenbacher
Introduction Many problems in Engineering, Management, Health Sciences and other Sciences involve exploring the relationships between two or more variables.
Research Design
School of Business Administration
Achieving High Software Reliability
An assessment framework for Intrusion Prevention System (IPS)
Estimating with PROBE II
Human Factors Issues Chapter 8 Paul King.
S519: Evaluation of Information Systems
Estimating the number of components with defects post-release that showed no defects in testing C. Stringfellow A. Andrews C. Wohlin H. Peterson Jeremy.
Presentation transcript:

(1) Cam Moore Collaborative Software Development Laboratory Communication & Information Sciences University of Hawaii, Manoa Investigating Individual Software Development: An Evaluation of the Leap Toolkit

(2) Outline Why Is Quality Software Development Important? Solutions Thesis Evaluation Contributions & Future Directions

(3) Problem Every Software developer wishes they got home earlier and spent less of their weekend at work. Software is often: Late - Windows 2000 Over budget - DIA baggage control Full of defects - Therac-25 Problem Solutions Leap Evaluation Contributions

(4) Solutions Hope Based (Try Harder) 1960s Product Based (Testing, Review) 1970s Organization Based (CMM, ISO 9000) 1980s Developer Based (PSP, Leap) 1990s Problem Solutions Leap Evaluation Contributions

(5) Problems with traditional solutions Many software process improvement initiatives suffer from one or more of the following problems: Heavy weight development constraints Organization-level analysis and improvement Measurement Dysfunction Manual Data Collection Problem Solutions Leap Evaluation Contributions

(6) Supporting Software Developer Improvement with LEAP LEAP is our design philosophy. All LEAP tools must satisfy four major criteria: Light weight Empirical Anti-Measurement dysfunctional and Portable The Leap toolkit is a reference implementation Problem Solutions Leap Evaluation Contributions

(7) Leap Toolkit Started August 1997 Over 25 public releases As of 17 Apr. 2000, the size of the Leap Toolkit is: 44KLOC of Java, 2156 Methods, 311 Classes and, 14 Packages It is available for download from Problem Solutions Leap Evaluation Contributions

(8) Leap Toolkit

(9) General Thesis LEAP provides a more accurate and effective way for developers to collect and analyze their software engineering data than methods designed for manual enactment. Problem Solutions Leap Evaluation Contributions

(10) Evaluation To evaluate this thesis I broke it into 3 claims: I. Leap is able to prevent important classes of data error as identified in Disney98. II. Leap reduces the level of collection stage errors by reducing the overhead of data collection. III. Leap implements data collection and analyses that enable sophisticated analysis not available in manual methods such as PSP. Problem Solutions Leap Evaluation Contributions

(11) I: Preventing data error I. Leap is able to prevent important classes of data error as identified in Disney98. Disney’s research discovered the following problems with manual PSP data: Students had a 5% error rate in their data These errors were significant - Time estimation - Incorrect Yield Even 95% accuracy is not good enough Problem Solutions Leap Evaluation Contributions

(12) I: Preventing data error The design and implementation of the Leap toolkit address the data error categories: Problem Solutions Leap Evaluation Contributions

(13) I: Preventing data error Data error categories (cont): Problem Solutions Leap Evaluation Contributions

(14) II: Reducing collection error II. Leap reduces the level of collection stage errors by reducing the overhead of data collection. Case Study: 16 Students used Leap while developing 8 Java programs in ICS 613 Data collection: - 4 surveys conducted (# responses) - Interviews (11 conducted) - Students’ Leap data (16 data sets) Problem Solutions Leap Evaluation Contributions

(15) II: Reducing collection error The students were asked to collect three types of data: Defects - Students used many different collection methods Size - LOCC used for collecting size data Time - Two different time collection tools Problem Solutions Leap Evaluation Contributions

(16) II: Reducing collection error First method of time collection in Leap Naia Problem Solutions Leap Evaluation Contributions

(17) II: Reducing collection error Second method of time collection in Leap Io Problem Solutions Leap Evaluation Contributions

(18) II: Reducing collection error Using Naia for data entry makes data entry harder Students using Naia said data collection was neutral to hard Students using Io said data collection was easy to neutral Problem Solutions Leap Evaluation Contributions

(19) II: Reducing collection error Types of time collection errors Missing time Added time Rounding Reported use of Naia has high correlation to rounded times

(20) III: Empirical Investigation III. Leap implements data collection and analyses that enable sophisticated analysis not available in manual methods such as PSP. Leap allowed me to evaluate 14 different quantitative time estimation techniques. A BCActual Size Planned Size Average Linear Exponential LOC Methods PSP (A, B, or C) Problem Solutions Leap Evaluation Contributions Student’s

(21) III: Empirical Investigation I used Leap to calculate the prediction error for each estimation method. Prediction Error = | estimate - actual | Compared methods for each student Compared methods for the whole class Problem Solutions Leap Evaluation Contributions actual

(22) III: Empirical Investigation Individual Students’ Results: 3 of the 16 students had significantly better estimation methods 2 students, their own estimation methods were best (F = 3.93, p < 0.001), (F = 2.25, p = 0.019) 1 student, exponential regression using actual LOC was best (F = 2.34, p = 0.014) 5 students, own estimates best Problem Solutions Leap Evaluation Contributions

(23) III: Empirical Investigation Class’ Results: The students’ own estimates were the most accurate (F = 3.75, p <0.001) Ignoring the students’ estimates, the Exponential Regression using actual methods was the best (F = 2.40, p = 0.005) The PSP’s time estimation method was 6 th Problem Solutions Leap Evaluation Contributions

(24) Contributions The Leap toolkit An effective tool for software developer improvement - Reduces overhead - Flexible - Portable A novel software engineering tool for: - Education - Research Problem Solutions Leap Evaluation Contributions

(25) Contributions The results of the estimation experiment Insight into better software estimation techniques Experience may lead to over-confidence Problem Solutions Leap Evaluation Contributions

(26) Future Directions Replicated studies with more subjects including industrial software developers Industry adoption of Leap Toolkit More support for data analysis in Leap Toolkit Further study into effort estimation Personal Agents that “observe” the developer Problem Solutions Leap Evaluation Contributions

(27)

(28) Results: Case Study Estimation accuracy unaffected by rounding error No significant difference between the estimation accuracies of the class (F = 1.71, p = 0.07) No significant difference between Naia and Io users (F = 2.70, p = 0.10)

(29) Results: Case Study There was very little evidence of measurement dysfunction Grades not based upon estimation accuracy Students did not compare their Leap data Most students felt little pressure to make their data look “good” - Most said they felt less pressure - Two with increasing remained very low

(30) Results: Case Study Students showed traditional software developer improvements Increased Productivity Improved size estimation skills Improved time estimation skills

(31) Increased Productivity LOC Equal Slight improvement Methods Equal

(32) Productivity Size of methods increased Projects changed from Command line to GUI Last few project were more complicated Problem Solutions Leap Evaluation Contributions

(33) Improved Size Estimation LOC Steady Underestimation Methods Improvement Better Designs?

(34) Improved Time Estimation Class’ time estimation accuracy Under estimated Range consistent