1. Understanding the Human Estimator Gary D. Boetticher Boetticher@uhcl.edu Univ. of Houston - Clear Lake, Houston, TX, USA http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop Nazim Lokhandwala Lokhandwala@uhcl.edu Univ. of Houston - Clear Lake, Houston, TX, USA James C. Helm Helm@uhcl.edu Univ. of Houston - Clear Lake, Houston, TX, USA

2. Introduction Chaos Chronicles [Standish03]  300 billion dollars  250,000 new projects  1.2 million dollars per project http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

3. Boehm’s 4X http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

4. Types of Estimation [Jorgenson04] 63 - 86% Human-Based 7 - 16% Algorithmic and Machine Learners http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

5. Research Focus Number of Papers On Software Estimation in IEEE [Jorgenson02]  Human-Based Estimation (17%)  Other (83%) http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

6. Statement of Problem How do human demographics affect human-based estimation? Can predictive models be constructed using human demographics? http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

7. Investigation Procedure Collect demographics from participants Request participants to estimate software components Build models (Estimates vs. Actuals) Survey http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

8. Which Demographics? Basic Demographics Academic Background Work Experience Domain Experience http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

9. The Survey http://nas.cl.uh.edu/boetticher/EffortEstimationSurvey.html http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

10. Competitive Procurement Software Buyer Admin Buyer 1 Buyer n... Buyer Software Distribution Server Supplier 1 Supplier 2 Supplier n : Supplier Software http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

11. Sample Estimation Screenshots http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

12. Survey Results Screenshots http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

13. Data Collection Invitations Filtered Incomplete Records 122 Final Records http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

14. Participant Educational Background Most of the participants hold Bachelors or Masters Degrees MeanMaximum Standard Deviation Computer Science Undergrad Courses 8.85257011.6326 Grad Courses 2.4262153.2293 Hardware Undergrad Courses 3.5246648.0209 Grad Courses 0.5000101.3252 Management Information Systems Undergrad Courses 0.7705121.5892 Grad Courses 0.491891.3742 Project Management Undergrad Courses 0.295140.6886 Grad Courses 0.811561.1806 Software Engineering Undergrad Courses 0.918071.2958 Grad Courses 2.1557213.1202 http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

15. Participant Work Experience MeanMaximum Standard Deviation (Years) Years of Experience As Hardware Project Manager 0.6557151.9251 Software Project Manager1.3443102.0811 No of Projects estimated Hardware Projects0.8279202.6307 Software Projects2.9508284.4848 http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

16. Participant Domain Experience 2.2512200.7274 Process Industry 1.3818100.6209 Procurement and Billing Domain Experience Standard Deviation Maximum (Years) Mean (Years) http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

17. Data Preparation INPUT= 69% zeros…Needs Consolidation Courses, Workshops, Conferences, Programming Exp. 45 attributed reduced to 14 attributes Highest Degree Achieved…Need Transformation 942000100091000000000010110900200000030742400 000000000100000101100000000000000000005050520 000100000000000100000001200000000000040000001 000100000000000000000201110200100100040000510 OUTPUT= MRE=Abs (Total Actual – Total Est.)/(Total Actual) http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

18. Build Models Linear Regression (Excel) Non-Linear Regression (DataFit) Genetic Programming (GDB_GP) http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

19. GP Configuration 3 Settings 1000 Chromosomes 50 Generations 512 Chromosomes 128 Generations 1000 Chromosomes 128 Generations http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop 20 Trials each

20. Results: All Demographic Factors 1.87E-15 3.45E-17T-test 0.88470.55920.1550Mean Non-Linear Regression Genetic Programming Linear Regression 1.6470 0.8847 Non-Linear Regression Std. Error R Squared 1.38754.4580 0.91740.1550 Genetic Programming Linear Regression Best Values of R Squared with Min. Std. Error T-Test between Average R Square Values http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

21. Results: Educational Factors 0.0486 2.74E-13T-test 0.21360.19730.0373Mean Non-Linear Regression Genetic Programming Linear Regression 4.1667 0.2136 Non-Linear Regression Std. Error R Squared 3.97384.6101 0.27840.0373 Genetic Programming Linear Regression Best Values of R Squared with Min. Std. Error T-Test between Average R Square Values http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

22. Results: Work Experience 1.54E-11 2.73E-19T-test 0.36980.55640.0596Mean Non-Linear Regression Genetic Programming Linear Regression 4.0644 0.3698 Non-Linear Regression Std. Error R Squared 2.28554.5169 0.75720.0596 Genetic Programming Linear Regression Best Values of R Squared with Min. Std. Error T-Test between Average R Square Values http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

23. Results: Domain Experience 4.55E-16 3.27E-23T-test 0.32600.54050.0243Mean Non-Linear Regression Genetic Programming Linear Regression 3.9091 0.3260 Non-Linear Regression Std. Error R Squared 2.92834.5425 0.59110.0243 Genetic Programming Linear Regression Best Values of R Squared with Min. Std. Error T-Test between Average R Square Values http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

24. Summary of All Experiments R Square Values Linear Regression Best Case Genetic Prog. Avg. Case Genetic Prog. Non-Linear Regression All Factors0.15500.91740.55920.8847 Education Only0.03730.27840.19730.2136 Work Experience Only0.05960.75720.55640.3698 Domain Experience Only 0.02430.59110.54050.3260 http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

25. Best Equation: All Factors. r 2 = 0.9174 ((Log (TechGradCourses + (TechGradCourses ^ ((Log TotWShops)/(Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (Log (Log (Log SWProjEstExp))))))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (((ProcIndExp + (Log (Sin MgmtGradCourses)))/(Sin SWPMExp)) + (Sin ((Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Sin SWPMExp)))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Cos (TechGradCourses ^ ((Log SWProjEstExp) / (((Log (ProcIndExp + (Log (TechGradCourses ^ ((Log SWProjEstExp) / (Log SWProjEstExp)))))) - 3) / (ProcIndExp + (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos ((((Log SWProjEstExp) / ((ProcIndExp + (Log (TechGradCourses ^ (TechGradCourses ^ (Log SWProjEstExp))))) / (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (Log (Log (Log SWProjEstExp)))))))))))))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / (TechGradCourses ^ (Log SWProjEstExp))))))))))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))) + ((Log SWProjEstExp) / (Log SWProjEstExp)))))) / (Log (Log (Log (TechGradCourses + (Cos (Log (Log (TechGradCourses ^ (Cos (((((Log SWProjEstExp) / (TechGradCourses ^ (Log SWProjEstExp))) / ((ProcIndExp + (Log (Sin MgmtGradCourses))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp))))))))))))))))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Log ((((Log TotLangExp) / (Log SWProjEstExp)) / (Log SWProjEstExp)) / (Sin SWPMExp))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))))))) + (((((ProcIndExp + (Log (TechGradCourses ^ (Log (TechGradCourses + ((TechGradCourses ^ (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos ((((Log SWProjEstExp) / ((ProcIndExp + (Log (TechGradCourses ^ (Log (TechGradCourses + (Cos (Log (Log (TechGradCourses ^ (Cos (((((Log SWProjEstExp) / (TechGradCourses ^ (Log SWProjEstExp))) / ((ProcIndExp + (Log (Sin MgmtGradCourses))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / (TechGradCourses ^ (Log SWProjEstExp))))))) / (Sin SWPMExp))))))) / (TechGradCourses ^ (Log SWProjEstExp))) / (TechGradCourses ^ (Log SWProjEstExp))) / (TechGradCourses ^ (Log SWProjEstExp))) / (Sin SWPMExp))) http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop Too Much of a Good Thing?

26. Conclusions Viability of a human-based est. model Model assessment  Non-linear  GP Impact on Human Based Estimation 1) All Factors 2) Domain Experience  Work Experience 3) Education http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

27. Future Directions Equation Optimizer for GP Collect More Data  Further analysis without consolidation  Detailed Effect of Educational Factors Use other statistical indicators Build other models  Hybrid (Non-linear and GP)  Classifiers Impact of process on estimation http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop

28. http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop Questions?

29. http://nas.cl.uh.edu/boetticher/publications.htmlThe 2 nd International Predictor Models in Software Engineering (PROMISE) Workshop Thank You !