1. March 4, 1999Lethbridge1 Empirical Studies of Software Engineers: Some Experience and Pragmatic Considerations Dr. Timothy C. Lethbridge University of Ottawa

2. March 4, 1999Lethbridge2 Outline A Case Study: The KBRE Project Results of the KBRE Work So Far Key Difficulties and Risks University ESE Researchers Face How Can More Companies be Interested in Empirical Work Issues that Must be Agreed Between the Company and the Researchers Working With Corporate Employees and Managers

3. March 4, 1999Lethbridge3 A Case Study: The KBRE Project A collaborative project supported by: –Mitel –CSER (Consortium for Software Engineering Research) –Involving The NRC An industrially-driven project

4. March 4, 1999Lethbridge4 A Case Study: The KBRE Project Objective –To improve the productivity of software engineers who have to understand and modify large software systems Situation –Mitel has a large real-time software system undergoing continuous development

5. March 4, 1999Lethbridge5 A Case Study: The KBRE Project Problems to be tackled –New hires take a long time to learn the system –Experienced people take a long time to make changes

6. March 4, 1999Lethbridge6 A Case Study: The KBRE Project Overall research strategy: –Study work of software engineers to discover opportunities to handle problem –Analyze result, model their work patterns and develop tool requirements –Develop prototype tools –Evaluate tools in work environment

7. March 4, 1999Lethbridge7 A Case Study: The KBRE Project Types of empirical studies performed –Studied software engineers at work Interviews, work observation, synchronized shadowing –Monitored their use of all tools –Did some modeling of their mental models and knowledge –Usability studies

8. March 4, 1999Lethbridge8 Outline A Case Study: The KBRE Project Results of the KBRE Work So Far Key Difficulties and Risks University ESE Researchers Face How Can More Companies be Interested in Empirical Work Issues that Must be Agreed Between the Company and the Researchers Working With Corporate Employees and Managers

9. March 4, 1999Lethbridge9 Results of the KBRE work so far Synchronized Shadowing –Involves two ‘observers’ –Working on laptops with simple data input interfaces –Each observing a different kind of data –Clocks are synchronized –Data is merged later to obtain composite view

10. March 4, 1999Lethbridge10 Results of the KBRE work so far Using Use Case Maps to Represent SE Work Patterns

11. March 4, 1999Lethbridge11 Results of the KBRE work so far Analyzing the data to discover frequencies of patterns

12. March 4, 1999Lethbridge12 Results of the KBRE work so far Requirements for a source browsing tool –Strong support for search of all types –Make good use of screen real-estate –Make transitions from one type of information to another easy to do –Allow heterogeneous graph structures in exploration –Allow user to build their own mental model

13. March 4, 1999Lethbridge13 Results of the KBRE work so far –Allow easy context switching and return –Make it fast –Make it work with multiple languages –Make it USABLE!!!!!

14. March 4, 1999Lethbridge14 Results of the KBRE work so far A tool architecture to support program comprehension

15. March 4, 1999Lethbridge15 Results of the KBRE work so far A tool being actively used at Mitel An experimental test bed for the research of the whole research group –7 graduate students, 2 research associates Performed usability analysis –Even as a research platform, it has to be usable and used so we obtain accurate data about the validity of the ideas

16. March 4, 1999Lethbridge16 Outline A Case Study: The KBRE Project Results of the KBRE Work So Far Key Difficulties and Risks University ESE Researchers Face How Can More Companies be Interested in Empirical Work Issues that Must be Agreed Between the Company and the Researchers Working With Corporate Employees and Managers

17. March 4, 1999Lethbridge17 Key Difficulties and Risks University ESE Researchers Face Those planning empirical studies must perform risk management, taking these into account Establishing relationships with industrial partners –We had it easy –The industrial partner sought us out

18. March 4, 1999Lethbridge18 Key Difficulties and Risks Attracting graduate students –Not so easy –Students often like design and working alone –Are empirical studies a ‘marketable skill?’ Researchers tempted to take ‘short cuts’ –Just work on theory, toy problems –Just work with a few participants in the university

19. March 4, 1999Lethbridge19 Key Difficulties and Risks Learning empirical techniques –Not a traditional part of our background Having the work recognized by colleagues Training grad students Avoiding crossing the line from research to development Analyzing data from empirical studies

20. March 4, 1999Lethbridge20 Key Difficulties and Risks Research fails or goes slowly –Hypothesis proved wrong –Methodology inadequate –Co-operation does not work –Researchers lack skills –Inadequate time –Students interest drifts Company Priorities Change

21. March 4, 1999Lethbridge21 Outline A Case Study: The KBRE Project Results of the KBRE Work So Far Key Difficulties and Risks University ESE Researchers Face How Can More Companies be Interested in Empirical Work Issues that Must be Agreed Between the Company and the Researchers Working With Corporate Employees and Managers

22. March 4, 1999Lethbridge22 How Can More Companies be Interested in Empirical Work? Tangible benefits –New or improved products Semi-tangible benefits –New or improved processes Intangible benefits –Exposure of students to companies (potential employees)

23. March 4, 1999Lethbridge23 Issues that Must be Agreed Between the Company and the Researchers Company and university objectives for the research –Recognising the company imperative for the bottom line –University imperative for publication Access to facilities and employees –time commitment; participant selection process Where will the work actually take place?

24. March 4, 1999Lethbridge24 Issues that Must be Agreed Deliverables, and planning for changes in research focus. –How detailed should the project plan be? Intellectual property and publication of results –Patents, copyright, trade secrets, non-disclosure, masking of information, approval of release –Dealing with short publication lead-times. How to handle the risks described earlier

25. March 4, 1999Lethbridge25 Working with Corporate Employees and Managers There must be an informed consent process Understanding differences in attitudes of employee-participants. What participants need to know –Support of management –Nature of the research –Disposition of data (confidentiality etc.) –They should feel a net benefit of the research

26. March 4, 1999Lethbridge26 Conclusions University based software engineering researchers should do more empirical studies in industry –Should be part of regular SE research Benefits, risks, and techniques need to be better understood