State of The Art and Future Directions Dept. of Computer Science Empirical Studies of Object-Oriented Artifacts, Methods, and Processes: State of The Art and Future Directions Li Jingyue Dept. of Computer Science NTNU 18th,Oct. 2002

Agenda State of the Art Research Directions Success Factors for Empirical Studies

State of The Art Challenges Existing Empirical Studies Methodology for formal definition and empirical validation of OO measures Empirical Assessment of OO Design Principles Empirical Evaluation of Quality Models for OO Software Empirical Evaluation of Testability of OO software Common Problems and Open Issues

Challenges (1) OO technologies are claimed to improve the quality of software product deliverables, to support reuse and reduce the effort of developing and maintaining the software product But … Little evidence exists to support the claims

Challenges (1) Example: Inheritance and polymorphism are claimed to provide benefits such as greater extensibility and reusability of OO system.However, some examples showed results suggesting that,beyond a certain level, inheritance is a serious hindrance to maintainability.

Challenges (2) The goals or hypotheses underlying many of the proposed measures are often not clearly stated The measures are often not defined in a fully operational form So … It difficult to replicate the studies by other researchers

Challenges (3) Some early investigations on testing approaches for OO software have already been performed and published But … These partial solutions have not been compared, not attempt to combine them has been made

State of The Art Challenges Existing Empirical Studies Methodology for formal definition and empirical validation of OO measures Empirical Assessment of OO Design Principles Empirical Evaluation of Quality Models for OO Software Empirical Evaluation of Testability of OO software Common Problems and Open Issues

Methodology for formal definition and empirical validation of OO measures Goal of this research is to evaluate software measures and to qualify the use of certain statistical techniques

Methodology for formal definition and empirical validation of OO measures Example: (Brinand 1999) proposed methodologies for the definition and construction of OO coupling and cohesion measures.

Empirical Assessment of OO Design Principles Goal of this research is to empirically compare OO technologies and processes to traditional structured techniques.

Empirical Assessment of OO design principles (cond) Example: (Houdek 1999) conducted an experiment comparing traditional structured methods and OO methods applied to embedded software system. The results identified only minor difference in development time and quality of the developed system.

Empirical Evaluation of Quality Models for OO software Goal of this research is to relate structural attribute Encapsulation, Inheritance, Coupling, cohesion to external quality indicators Fault proneness, development effort, Test effort, rework effort, Reusability, maintainability.

Empirical Evaluation of Quality Models for OO software (cond) Example: (Binkley,1998) collected maintenance data from four development projects written in COBOL, C, C++ and Java,respectively. The result suggested that ”a significant impediment to maintenance is the level of coupling between modules”. Modules with low coupling were subjected to less maintenance effort and had fewer maintenance faults and fewer run-time failures

Empirical Assessment of Testability of OO software Goal of this research is to empirically study the feasibility and adequacy of the proposed testing techniques

Empirical Assessment of Testability of OO software (cond) Example: (kung 1995) and (Labiche 1999) propsed two different testing approaches. Both approaches are based on a class test model which captures the static dependencies between the classes and objects, but they have different limitations that have still to be empirically compared.

Common Problems and Open Issues State of The Art Challenges Existing Empirical Studies Methodology for formal definition and empirical validation of OO measures Empirical Assessment of OO Design Principles Empirical Evaluation of Quality Models for OO Software Empirical Evaluation of Testability of OO software Common Problems and Open Issues

Common Problems and Open Issues How to investigate artifacts from earlier states of the life cycle, e.g.,requirement, architectural design, in order to build early quality models How to investigate the existing product measures , following thorough, rigorous, and complete analysis procedures, within the context of well-designed empirical studies How to replicate studies across many environments to draw more general conclusions.

Identify Important Factors Research Directions Identify Important Factors Evaluation of OO Technologies Building Quality and Productivity models Meta-level issues

Identify Important Factors The proposed metrics are mainly related to static aspects of OO systems. Measurement of dynamic attributes (e.g., coupling at run-time) has not yet been considered in depth. Measurement of single class has been investigated to a great extent, measurement of class cluster(e.g., when using COTS) has not been fully investigated.

Identify Important Factors (cond) The number of measures that have been proposed for OO products is very large. We need to better understand what these measures are really capturing, whether they are really different, and whether they are useful indicator of quality or productivity properties of interest

Identify Important Factors (cond) The application domain is usually seen as a major factor determining the usefulness of measurement or the effectivness of technologies. We should characterize the type of functionality delivered, the type of development technologies used, the scale of development and any other characteristic which is typically associated with a domain

Evaluation of OO Technologies Research Directions Identify Important Factors Evaluation of OO Technologies Building Quality and Productivity models Meta-level issues

Evaluation of OO Technologies It is important to investigate the use of UML more thoroughly. The tools that support UML-based development need to be evaluated and their prerequisite for successful use must be investigated The development processes, such as Rational, should be carefully monitored since it is likely that they will require some degree of tailoring in each organization

Building Quality and Productivity models Research Directions Identify Important Factors Evaluation of OO Technologies Building Quality and Productivity models Meta-level issues

Building Quality and Productivity models Quality models need to be available earlier in the life cycle in order to drive and ensure early built-in quality. Productivity (cost to product characteristics) model should be used earlier so that accurate project planning and risk analysis can be performed

Research Directions Meta-level issues Identify Important Factors Evaluation of OO Technologies Building Quality and Productivity models Meta-level issues

Meta-level issues Results need to be combined and conclusions need to be generalized

Success Factors for Empirical Studies Nature of theData Consistent Terminology Nature of the Research

Nature of the Data Ensure the reliability, completeness and efficiency of the data collected. Ensure the data are clearly justified and its underlying assumptions are made explicit as to aid the interpretation of result. Ensure any measurement reported be defined in an operational and unambiguous manner

Consistent Terminology A common problem in SE community is the inconsistent use of terminology and notation. For example: There are many forms of coupling that can arise in systems. Establishing the different forms of coupling are still open research questions. So… Ensure the terminologies used in SE empirical studies are consistent and unambiguous

Nature of the Research Some empricial researches are understaken without establishing beforehand whether the problem is either worth investigating, is too complex at this stage of knowledge or addresses a different underlying issues So… Ensure you have clearly defined the problem and the results of which are useful to the community.

Conducting Realistic Experiments in Software Engineering Li Jingyue Dept. of Computer Science NTNU 18th,Oct. 2002

Agenda Introduction Realism in SE Experiments Conclusion Realistic Tasks Realistic Subjects Realistic Environment Conclusion

Introduction (1) The ultimate criterion for success in an applied discipline such as software engineering research is the widespread adoption of research results into everyday industrial practice

Introduction (2) Most of the studies in software engineering that have emphasized realism are case studies. However, many variables vary from one case study to another so that comparing the results to detect cause-effect relationships is difficult. Therefore, controlled experiments should be conducted to complement case studies in empirical software engineering

Introduction (3) To convince industry about the validity and applicability of the controlled experimental results, the tasks, subjects and the environments of the experiments should be as realistic as possible

Agenda Introduction Realism in SE Experiments Conclusion Realistic Tasks Realistic Subjects Realistic Environment Conclusion

Realistic tasks:Challenge The challenge is concerned with the size, complexity and duration of the involved tasks Of course, the world is diverse, so the realism of a task must be considered relative to a certain part or aspect of the world.

Realistic tasks:Solution A systematic way to define realistic tasks according to a given application area in a given context, is to collect information about the kinds and frequencies of tasks in the actural enviroment and then create "benchmark tasks", i.e. a set of tasks that is a representative sample of tasks from the population of all tasks.

Agenda Introduction Realism in SE Experiments Conclusion Realistic Tasks Realistic Subjects Realistic Environment Conclusion

Realistic Subjects:Challenge This challenge is concerned with the selection of subjects to perform the experimental tasks, that is, to what extend do the selected subjects represent the population that we wish to make claims about? A common criticism of experiments in software engineering is that most of the subjects are students, which might make it difficult to generalise the results to setting with professionals.

Realistic Subjects Students versus professionals Semi-professionals experience and skill level use of professional methods and tools team work versus individual work Semi-professionals

Realistic Subjects Variations among the professionals Age Education(number of credits in general, number of credits in computer science) General work experience Programming experience(OO, Java, C++) Knowledge of system developments methods and tools Subjective description of their own programming skills

Realistic Subjects:Solution To generalise from experiments with a given group of subjects, we should get the information about the ability and the variations among the subjects and the group of people to which the results will be generalised.

Agenda Introduction Realism in SE Experiments Conclusion Realistic Tasks Realistic Subjects Realistic Environment Conclusion

Realistic Environment:Challenge The challenge is to configure the supporting technology( process, methods, tools, etc.) that resembles an industrial development environment.

Realistic Environment:Solution Conducting an experiment on the usual work site with professional development tools implies less control of the experiment than we would have in a classroom setting with pen and paper.

Experiment Supporting Technology Administrator Researcher 1.Define experiment 5.Analyze results 2.Define users Simula Experiment Support Environment 3.Questionnaires task descriptions source code 4.Answer questions task solutions source code Subjects Professional developers, at usual work place using normal development tools

Agenda Introduction Realism in SE Experiments Conclusion Realistic Tasks Realistic Subjects Realistic Environment Conclusion

Conclusion Increasing the realism of SE experiment requires an increase in the resource needed to conduct such experiments Given the importance and challenges of software industry, the empirical SE community should apply national and multi-national research bodies to fund realistic experiments