CS527: (Advanced) Topics in Software Engineering Reading Papers Tao Xie ©D. Marinov, T. Xie

Course Communication Mailing list: cs527-fa13 AT dcs-maillist2.engr.illinois.edu –Did you get from that mailing list? Wiki –All of you should be able to access and sign on the “People” page –All of you should sign up on “People” now

General Project Topics Test thoroughly a piece of software Apply some technique(s)/tool(s) that we read about on some piece of software Improve some technique/tool that we discuss in class Evaluate some techniques/tools on a number of case studies Develop a new technique/tool … Some specific proposals will be on Wiki

How to Choose a Project? Talk –Discuss with other people Group projects: talk with students TA and/or me Read –Read papers and see what the limitations are –Reuse problem or solution Program –Problems from you own development experience Dream :)

Today’s Lecture/Meeting Guideline for reading papers Paper on writing papers

Guideline for Reading Papers How to Read an Engineering Research Paper by William G. GriswoldHow to Read an Engineering Research Paper Did you get to read the guideline? –How long is the report form? Our form for reports will be shorter –Later on, papers should be read (and reports written) in advance, before the meeting, so that we can have a good discussion Any questions about the guideline?

(Meta-)Paper on Writing Papers Writing Good Software Engineering Research Papers by Mary Shaw (ICSE 2003)Writing Good Software Engineering Research Papers Could we analyze this paper itself using the guideline for reading papers?

Writing Good SE Papers Motivation –Guidelines for writing papers for ICSE Approach –Analysis of papers submitted to ICSE 2002 –Distribution across three dimensions Question Result Validation Results –Writing matters, know your conferences!

Types of Question Method or means of development Method for analysis or evaluation Design, evaluation, or analysis of a particular instance Generalization or characterization Feasibility study or exploration

Types of Result Procedure or technique Qualitative or descriptive model Empirical model Analytic model Tool or notation Specific solution, prototype, answer, or judgment Report

Types of Validation Analysis Evaluation Experience Example Persuasion Blatant assertion

Some Questions for Discussion Why do analysis papers have an edge? What about other (software engineering) conferences? Does the mentality of PC differ from one conference to another? Development vs. evaluation in software engineering?

Other Papers/Projects? Some papers that you read or will read –Do they follow these guidelines? Your reports on papers should also consider question/result/validation Your own project –Question? –Result? –Validation?

Traceability Links Contribution/claim 1 Contribution/claim 2 … Introduction/main contribution list Research question 1 Research question 2 … See GQM by Weiss/Basili Metric 1 Metric 2 … Make sure each contribution/claim is translated to (appropriate) research question(s)  no unsubstantiated claims Make sure each question is answered with help of (appropriate) metric(s) Evaluation

Factors Affecting Choosing a Problem/Project What factors affect you (not) to choose a problem/project? –Besides your supervisor/mentor asks you (not) to choose it

Factors Affecting Choosing a Problem/Project Impact/significant: Is the problem/solution important? Are there any significant challenges? Industrial impact, research impact, … DON’T work on a problem imagined by you but not being a real problem E.g., determined based on your own experience, observation of practice, feedback from others (e.g., colleagues, industrial collaborators) Novelty: is the problem novel? is the solution novel? –If a well explored or crowded space, watch out (how much space/depth? how many people in that space?)

Factors Affecting Choosing a Problem/Project II Risk: how likely the research could fail? –reduced with significant feasibility studies and risk management in the research development process –E.g., manual “mining” of bugs Cost: how high effort investment would be needed? –Sometimes being able to be reduced with using tools and infrastructures available to us –Need to consider evaluation cost (solutions to some problem may be difficult to evaluate) –But don’t shut down a direction simply due to cost

Factors Affecting Choosing a Problem/Project III Better than existing approaches (in important ways) besides new: engineering vs. science Competitive advantage –“secret weapon” –Why you/your group is the best one to pursue it? –Ex. a specific tool/infrastructure, access to specific data, collaborators, an insight,… –Need to know your own strengths/weaknesses Underlying assumptions and principles - how do you (systematically) choose what to pursue? –core values that drive your research agenda in some broad way This slide was made based on discussion with David Notkin