1. ISTE Workshop Research Methods in Educational Technology
IIT Bombay
February 2-9, 2013 *

2. What is the referee looking for in your paper?
Session 3:
What is the referee looking for in your paper? *

3. What do referees look for
Your paper must have
Novelty
Analysis of prior work to show that your idea is unique
Positioning
Analysis to show that your work is required, how your work advances the state of the art
Soundness of procedure
Steps to show that you have implemented solution carefully
Evidence to support claim
Data to show that your solution works as claimed
Overall coherence
Consistency between parts of your paper – treatment should address problem, results should give answer to problem *

4. What do referees look for
Your paper must have
We discussed this in
Novelty
Analysis of prior work to show that your idea is unique
“Is this an acceptable paper”, features of published papers
Positioning
Analysis to show that your work is required, how your work advances the state of the art
this session
Soundness of procedure
Steps to show that you have implemented solution carefully
Analysis of strong paper
Evidence to support claim
Data to show that your solution works as claimed
Features of published papers, analysis of strong/ weak paper activity
Overall coherence
Consistency between parts of your paper – treatment should address problem, results should give answer to problem
Analysis of strong / weak paper activity *

5. Is the paper positioned well?
1) Has your paper analyzed related prior work?
Have you referred to other papers that have addressed a problem similar to yours?
Have you referred to other papers that have a solution approach similar to yours?
Have you clearly brought out the gaps?
This will bring out the need for your research study.
2) Does your solution address any of the gaps above?
This will bring out the novelty of your solution.
3) Is your solution based on the application of appropriate educational theory?
This will bring out the soundness of your solution. *

6. Have you referred to other papers that have addressed a problem similar to yours
Example from PaperA - The Effectiveness of Learning Simulations for Electronic Laboratories (Section II A)
“Several simulations have been formulated for electronics labs. Circuit Tutor, an electronic simulation for building circuits at the University of Illinois, is embedded in a virtual classroom that includes the simulation along with online discussions.
“The Electronic Workbench [2] is a popular circuit capture and simulation system that is frequently used for education and training. A simulation with a long history and positive results, it can be used with circuit design software to create and try out various circuits.”
Brief Description of what they do
Brief Description of their results *

7. Identification of their drawbacks
Have you referred to other papers that have a solution approach similar to yours
Example from PaperA - The Effectiveness of Learning Simulations for Electronic Laboratories (Section II A)
“The system (Circuit Tutor) continues to use lectures to teach theory.”
“The Electronic Workbench package includes tutorials and reference information, but without the detailed coaching of the ELS.”
“Both Circuit Tutor and Electronic Workbench may reduce the cost and time of laboratory experiences, but the efficacy of these simulations compared with that of physical equipment labs is not well understood. The studies reported here (ELS) investigate the extent to which laboratory simulations of electronics circuits that add realistic graphic representations of equipment, with immediate computer coaching, may replace some physical electronics laboratories.”
Identification of their drawbacks *

8. Have you clearly brought out the gaps in related work?
Example from PaperA - The Effectiveness of Learning Simulations for Electronic Laboratories(Section II A)
Both Circuit Tutor and Electronic Workbench may reduce the cost and time of laboratory experiences, but the efficacy of these simulations compared with that of physical equipment labs is not well understood.
Gap *

9. Does your solution address the gaps?
Example from PaperA - The Effectiveness of Learning Simulations for Electronic Laboratories (Section II A)
Both Circuit Tutor and Electronic Workbench may reduce the cost and time of laboratory experiences, but the efficacy of these simulations compared with that of physical equipment labs is not well understood. The studies reported here investigate the extent to which laboratory simulations of electronics circuits that add realistic graphic representations of equipment may replace some physical electronics laboratories.
Gap
How authors’ solution addresses gap *

10. Why this theory is appropriate to apply in this case
Is your solution based on application of appropriate educational theory?
Example from PaperA - The Effectiveness of Learning Simulations for Electronic Laboratories (Section II B)
Active Learner Construction and Use of Knowledge and Skills
Learning simulations typically require job-like performance by learners; thus, active learning is inherent in the methodology ([6], [7]). Campbell and Gibbons [8] also describe progressively challenging evaluative simulations that integrate assessment, learning and performance support.
Why this theory is appropriate to apply in this case
Theory *

11. Recap: Positioning your paper
First, have good research to write about
Claims must match your results
Results must be a significant improvement over known results
Results are more convincing if they are confirmed in different ways
Focus on presenting one key idea, for a conference paper
What precisely was your research question?
What exactly is your new result?
Use verbs that show results and achievement, not just effort and activity
Explain the relation to other work clearly (See next slide) *

12. Explain the relation to other work clearly
Awful
The galumphing problem has attracted much attention [3,8,10,18,26,32,37]
Bad
Smith [36] and Jones [27] worked on galumphing.
Poor
Smith [36] addressed galumphing byblitzing, whereas Jones [27] took a flitzing approach
Good
Smith's blitzing approach to galumphing[36] achieved 60% coverage [39]. Jones [27] achieved 80% by flitzing,
but only for pointer-free cases [16].
Better
(Good Above) + We modified the blitzing approach to use the kernel representation of flitzing and achieved 90% coverage while relaxing the restriction so that only cyclic data structures are prohibited.
Source: Mary Shaw, Writing good Software Engineering Research Papers, ICSE 2003 *

13. Positioning your paper
Idea Proposal Form – Next Assignment
Let us walk through the Assignment *

14. Home-works Go through all the slides posted for today’s session
Watch the videos posted for
How to read a paper
How to do literature survey
Find three papers related to your idea
Fill out the Idea Proposal Form
Posted on Moodle *

15. Concluding remarks for the day
What is next.
Next hour – Reflection, Q & A
Next week Feb 3-8 – Complete Idea Proposal Assignment, think how to apply what you learned today to go from ET Practitioner to ET researcher
Next Saturday – How to set up ET research study, methods to evaluate your solution, more activities!
Week after next Saturday Feb – Final Assignment
And after that … *