1. ISTE Workshop Research Methods in Educational Technology
IIT Bombay
February 2-9, 2013

2. Analysis of strong and weak research papers in ET
Session 2b:
Analysis of strong and weak research papers in ET

3. Activity – Compare papers
Paper-A: Effectiveness of learning simulations for electronic labs
Paper-B: Innovative lab experiences for intro. EE students
What was the learning objective of the above activity?
Be able to:
1) Critically read a paper
2) Analyze a paper
3) Compare papers
By looking for the features (Q1), and asking Qs 2, 3, 4 ?

4. Q1– is the item present in each paper?
Check your answer. N Y
(introduction)
The importance of the problem
(Sec IV)
Specific research questions or objective
Partially.
Paper suggests that it addresses issues related to achieving ABET outcomes
(1st sentence abstract)
The problem being addressed by the paper
Is the item present in Paper-B? Y/N?
Is the item present in Paper-A? Y/N?
Item

5. A strong paper has Problem being addressed
Paper-A, 1st sentence, Abstract:
This work investigates the efficacy of software simulations of electronic circuits laboratories to support beginning electrical engineering students.
Paper-A Sec II
The studies reported here investigate the extent to which laboratory simulations of electronics circuits that add realistic graphic representations of equipment may replace some physical electronics laboratories. *

6. A strong paper has Importance of problem explained
Paper A : Sec I - Introduction
Electronic simulations may increase student access to a laboratory experience, since they are not constrained to one specific time and place. Quality simulations might simplify scheduling and also reduce cost by minimizing the use of expensive equipment. *

7. A strong paper has Specific research questions Paper A: Sec IV
This study sought to answer the questions, To what extent will subjects who use the ELS simulated labs, together with physical labs and classes, improve their performance on written theory and lab tests? What difficulties will they encounter that suggest ways that ELS can be improved? *

8. A weak paper
Mentions problem in a broad, fuzzy manner even if authors have an intended instructional goal in mind; no specific
Mentions what was done , no mention of importance
Paper-B: Sec - Background
This paper describes the new experiments and open-ended design projects. Furthermore, the paper describes enhancing the students’ ABET outcomes C and I, design for realistic constraints and life-long learning. *

9. Q1– is the item present in each paper?
Check your answer. N Y
Gaps in the prior work
Not reported
(Sec II- Related work)
Prior work that has been done to solve the problem
Is the item present in Paper-B? Y/N?
Is the item present in Paper-A? Y/N?
Item

10. A strong paper has Related work stated, analyzed
Sec II A. Related work.
Other papers that have a solution approach similar to the authors’.
Sec IIB. Learning Issues.
Authors’ solution based on the application of appropriate educational theories. *

11. A strong paper has Gaps in prior work identified Sec II A.
Both Circuit Tutor and Electronic Workbench may reduce the cost and time of laboratory experiences, but the efficacy of these simulations compared with that of physical equipment labs is not well understood. *

12. Q1– is the item present in each paper?
Check your answer.
Y – all details are related to development of instructional material
Y (Sec IV)
Outline of how the authors solved the problem
Details of implementation of procedure
Is the item present in Paper-B? Y/N?
Is the item present in Paper-A? Y/N?
Item

13. A strong paper has
Broad, overall solution explained first, then details
Sec IV – 1st sentence
Two experiments were performed: Experiment 1 compared the performance on written laboratory tests by learners at ….
Experiment 2 was an investigation that compared learning performance from physical-equipment-only labs with the performance in combined physical and simulated labs.
Then 1-paragraph details for each:
Method:
Subjects:
Procedure: *

14. A weak paper has
Missing - The thought process of researchers, why and how the solution should work *

15. Q1– is the item present in each paper?
Check your answer. N
Y (Sec IIB – Learning issues)
Educational theories on which solution is based
Is the item present in Paper-B? Y/N?
Is the item present in Paper-A? Y/N?
Item

16. A strong paper is Based on sound theory Sec IIB. Learning Issues.
Authors’ solution based on the application of appropriate educational theories. *

17. Q1– is the item present in each paper?
Check your answer.
Briefly (last paragraph of paper)
Y (Sec IV A,B)
Evaluation of the authors’ solution of the problem
Is the item present in Paper-B? Y/N?
Is the item present in Paper-A? Y/N?
Item

18. A strong paper has Solution defended Results clearly shown
Results connected to problem being solved *

19. A strong paper has Results that are easily understandable
A strong paper has
Results that are easily understandable
Show tables, graphs + explanations
Results of the written post-test indicate a significant difference in the scores of the two groups after the treatment, in favor of the combined group that used primarily simulations (Table 3). *

20. A Weak Paper
May have Attempts to evaluate solution, but evaluation not systematic or thorough.
Paper-B
Outcome I was easier to measure because it was less subjective. Students receiving the highest scores referenced journal articles, conference papers, and books that they found at the library. Lower scores were achieved by students who merely surfed the web, obtained information from their classmates, or didn't document their sources well. *

21. Q1– is the item present in each paper?
Check your answer.
Attempt Y
Connection of the results to the research questions
Is the item present in Paper-B? Y/N?
Is the item present in Paper-A? Y/N?
Item

22. A strong paper has
Connection of results to research questions/ objective
Objective
Our goal was to establish that electronic lab learning simulation with limited physical lab practice could produce results comparable to physical labs.
Connecting results and objective
Sec. 4A. Experiment 1 – 4.Discussion
The improvement cannot be attributed to the lab simulation, since simulation was used in conjunction with other learning tools. Additionally, the improvement was not consistent between labs. Early labs showed improvement, and later labs did not. *

23. Q1– is the item present in each paper?
Check your answer.
No, from an ET researcher’s perspective Y
Key contribution of the paper claimed by the authors
Is the item present in Paper-B? Y/N?
Is the item present in Paper-A? Y/N?
Item

24. A strong paper has Key contribution of paper clearly stated
(don’t make the referee search for what you have done)
Last paragraph in paper
These studies provide an indication that there exists an electronics laboratory simulation that yields learning that is at least equivalent to that gained from physical labs. *

25. A weak paper has
No clearly stated research contribution. Reader has to search.
In Paper-B, the experiments proposed may be effective in terms of their content from knowledge domain perspective but their contribution in the teaching learning process is not clear due to weak research methods. *

26. Comparing Papers A & B Paper-A : ET Researcher
Paper-B : ET Practitioner only *

27. A well-written paper Consistent flow Introduction Research Question
Methodology
Data Analysis
Results
Discussion
Future Work *

28. A strong paper - features
This is important enough that it is worth repeating
Identify focus, key idea
Key contribution(s) of paper clearly stated
Problem clearly stated and importance explained
Related work stated, analyzed
Solution explained: overall idea first details later
Solution defended
Results easily understandable
Consistent flow

29. Q2– compare A & B
Which paper does a better job of analyzing prior work?
Which paper is better in terms of describing procedure?
Which paper achieves its stated objective convincingly?
You and your partner have already discussed above questions.

30. HW Assignment – Evaluate your own idea
You now know the features that be present in a strong paper.
Goal of activity: Self-assess which of these features is present
in your idea that you submitted in the pre-workshop
assignment? Individual Activity.
NOTE:
You may change your idea at this point if you have thought of a better one. Write your new idea in your notebook, and apply the checklist to the new idea.

31. HW – Evaluate your own idea
Goal: Self-assess which of these features is present in your idea that you submitted in the pre-workshop assignment? Individual Activity - notebook
At this stage your idea might not contain all the required features. This is okay. You will work on incorporating them in the next Assignment.
Description of the specific problem you are addressing
Details of how you will evaluate your solution
Sound procedure to implement solution
Educational theory on which solution is based
Analysis of related work
Innovative strategy to solve problem
Importance of the specific problem
Is the feature present in your idea? Y / N?
Feature present in a research paper

32. Recall Q3– If you are the referee …
Will you agree that the paper is:
Innovative
Sound
Valid
NEXT SESSION: What does a referee look for?

33. TEABack at 3.55pm