ISTE Workshop Research Methods in Educational Technology IIT Bombay February 2-9, 2013

Analysis of strong and weak research papers in ET Session 2b: Analysis of strong and weak research papers in ET

Activity – Compare papers Paper-A: Effectiveness of learning simulations for electronic labs Paper-B: Innovative lab experiences for intro. EE students What was the learning objective of the above activity? Be able to: 1) Critically read a paper 2) Analyze a paper 3) Compare papers By looking for the features (Q1), and asking Qs 2, 3, 4 ?

Q1– is the item present in each paper? Check your answer. N Y (introduction) The importance of the problem (Sec IV) Specific research questions or objective Partially. Paper suggests that it addresses issues related to achieving ABET outcomes (1st sentence abstract) The problem being addressed by the paper Is the item present in Paper-B? Y/N? Is the item present in Paper-A? Y/N? Item

A strong paper has Problem being addressed Paper-A, 1st sentence, Abstract: This work investigates the efficacy of software simulations of electronic circuits laboratories to support beginning electrical engineering students. Paper-A Sec II The studies reported here investigate the extent to which laboratory simulations of electronics circuits that add realistic graphic representations of equipment may replace some physical electronics laboratories. *

A strong paper has Importance of problem explained Paper A : Sec I - Introduction Electronic simulations may increase student access to a laboratory experience, since they are not constrained to one specific time and place. Quality simulations might simplify scheduling and also reduce cost by minimizing the use of expensive equipment. *

A strong paper has Specific research questions Paper A: Sec IV This study sought to answer the questions, To what extent will subjects who use the ELS simulated labs, together with physical labs and classes, improve their performance on written theory and lab tests? What difficulties will they encounter that suggest ways that ELS can be improved? *

A weak paper Mentions problem in a broad, fuzzy manner even if authors have an intended instructional goal in mind; no specific Mentions what was done , no mention of importance Paper-B: Sec - Background This paper describes the new experiments and open-ended design projects. Furthermore, the paper describes enhancing the students’ ABET outcomes C and I, design for realistic constraints and life-long learning. *

Q1– is the item present in each paper? Check your answer. N Y Gaps in the prior work Not reported (Sec II- Related work) Prior work that has been done to solve the problem Is the item present in Paper-B? Y/N? Is the item present in Paper-A? Y/N? Item

A strong paper has Related work stated, analyzed Sec II A. Related work. Other papers that have a solution approach similar to the authors’. Sec IIB. Learning Issues. Authors’ solution based on the application of appropriate educational theories. *

A strong paper has Gaps in prior work identified Sec II A. Both Circuit Tutor and Electronic Workbench may reduce the cost and time of laboratory experiences, but the efficacy of these simulations compared with that of physical equipment labs is not well understood. *

Q1– is the item present in each paper? Check your answer. Y – all details are related to development of instructional material Y (Sec IV) Outline of how the authors solved the problem Details of implementation of procedure Is the item present in Paper-B? Y/N? Is the item present in Paper-A? Y/N? Item

A strong paper has Broad, overall solution explained first, then details Sec IV – 1st sentence Two experiments were performed: Experiment 1 compared the performance on written laboratory tests by learners at …. Experiment 2 was an investigation that compared learning performance from physical-equipment-only labs with the performance in combined physical and simulated labs. Then 1-paragraph details for each: Method: Subjects: Procedure: *

A weak paper has Missing - The thought process of researchers, why and how the solution should work *

Q1– is the item present in each paper? Check your answer. N Y (Sec IIB – Learning issues) Educational theories on which solution is based Is the item present in Paper-B? Y/N? Is the item present in Paper-A? Y/N? Item

A strong paper is Based on sound theory Sec IIB. Learning Issues. Authors’ solution based on the application of appropriate educational theories. *

Q1– is the item present in each paper? Check your answer. Briefly (last paragraph of paper) Y (Sec IV A,B) Evaluation of the authors’ solution of the problem Is the item present in Paper-B? Y/N? Is the item present in Paper-A? Y/N? Item

A strong paper has Solution defended Results clearly shown Results connected to problem being solved *

A strong paper has Results that are easily understandable   A strong paper has Results that are easily understandable Show tables, graphs + explanations Results of the written post-test indicate a significant difference in the scores of the two groups after the treatment, in favor of the combined group that used primarily simulations (Table 3). *

A Weak Paper May have Attempts to evaluate solution, but evaluation not systematic or thorough. Paper-B Outcome I was easier to measure because it was less subjective. Students receiving the highest scores referenced journal articles, conference papers, and books that they found at the library. Lower scores were achieved by students who merely surfed the web, obtained information from their classmates, or didn't document their sources well. *

Q1– is the item present in each paper? Check your answer. Attempt Y Connection of the results to the research questions Is the item present in Paper-B? Y/N? Is the item present in Paper-A? Y/N? Item

A strong paper has Connection of results to research questions/ objective Objective Our goal was to establish that electronic lab learning simulation with limited physical lab practice could produce results comparable to physical labs. Connecting results and objective Sec. 4A. Experiment 1 – 4.Discussion The improvement cannot be attributed to the lab simulation, since simulation was used in conjunction with other learning tools. Additionally, the improvement was not consistent between labs. Early labs showed improvement, and later labs did not. *

Q1– is the item present in each paper? Check your answer. No, from an ET researcher’s perspective Y Key contribution of the paper claimed by the authors Is the item present in Paper-B? Y/N? Is the item present in Paper-A? Y/N? Item

A strong paper has Key contribution of paper clearly stated (don’t make the referee search for what you have done) Last paragraph in paper These studies provide an indication that there exists an electronics laboratory simulation that yields learning that is at least equivalent to that gained from physical labs. *

A weak paper has No clearly stated research contribution. Reader has to search. In Paper-B, the experiments proposed may be effective in terms of their content from knowledge domain perspective but their contribution in the teaching learning process is not clear due to weak research methods. *

Comparing Papers A & B Paper-A : ET Researcher Paper-B : ET Practitioner only *

A well-written paper Consistent flow Introduction Research Question Methodology Data Analysis Results Discussion Future Work *

A strong paper - features This is important enough that it is worth repeating Identify focus, key idea Key contribution(s) of paper clearly stated Problem clearly stated and importance explained Related work stated, analyzed Solution explained: overall idea first details later Solution defended Results easily understandable Consistent flow

Q2– compare A & B Which paper does a better job of analyzing prior work? Which paper is better in terms of describing procedure? Which paper achieves its stated objective convincingly? You and your partner have already discussed above questions.

HW Assignment – Evaluate your own idea You now know the features that be present in a strong paper. Goal of activity: Self-assess which of these features is present in your idea that you submitted in the pre-workshop assignment? Individual Activity. NOTE: You may change your idea at this point if you have thought of a better one. Write your new idea in your notebook, and apply the checklist to the new idea.

HW – Evaluate your own idea Goal: Self-assess which of these features is present in your idea that you submitted in the pre-workshop assignment? Individual Activity - notebook At this stage your idea might not contain all the required features. This is okay. You will work on incorporating them in the next Assignment. Description of the specific problem you are addressing Details of how you will evaluate your solution Sound procedure to implement solution Educational theory on which solution is based Analysis of related work Innovative strategy to solve problem Importance of the specific problem Is the feature present in your idea? Y / N? Feature present in a research paper

Recall Q3– If you are the referee … Will you agree that the paper is: Innovative Sound Valid NEXT SESSION: What does a referee look for?

TEABack at 3.55pm