1. ISTE Workshop Research Methods in Educational Technology
IIT Bombay
February 2-9, 2013

2. Session 5:
Delving into RM-ET

3. Our Goals as ET Researchers
Our Goals as ET Researchers
conduct systematic studies to get data about whether our ideas are working.
provide evidence to support our conclusions.
This will establish soundness of your study.
We saw one example of how to do so.

4. Our Goals as ET Researchers
Our Goals as ET Researchers
conduct systematic studies to get data about whether our ideas are working.
provide evidence to support our conclusions.
This will establish soundness of your study.
We saw one example of how to do so.
Let us now examine the systematic methods to set up a study.

5. Recall - We saw ‘single group post only’ research design was not sound
Recall - We saw ‘single group post only’ research design was not sound
Ques: How do we know that it was the treatment (my strategy) that produced the outcome, and nothing else ?
Problems:
There is no comparison to a group that did not receive the treatment.
The outcome could have existed even before the treatment occurred.

6. How to design a ET research study
Session 5c:
How to design a ET research study

7. Problems with single group post only research design, and potential solutions
Problem
Potential solution
There is no comparison
to a group that did not
receive the treatment.
Compare a group that got ‘my strategy’ with a group that did not
(Two group post test design)

8. Problems with single group post only research design, and potential solutions
Problem
Potential solution
There is no comparison
to a group that did not
receive the treatment.
Compare a group that got ‘my strategy’ with a group that did not
(Two group post test design)
The outcome could
have existed even
before the treatment
occurred.
Compare performance of group before and after the treatment: check how much result changed after the treatment
(Single group pre-post design)

9. Research Designs Single group post test only
Research Designs
Single group post test only
Two group post test
Example1
Single group pre - post test
Example2
Two group pre- post test
Example3

10. Research Designs Single group post test only
Research Designs
Single group post test only
Two group post test
Example1
Single group pre - post test
Example2
Two group pre- post test
Example3

11. What do we mean by Research Design
Need to make decisions regarding:
How subjects are selected (sample)
How many groups
What strategy for each group
What data to collect
How data are analyzed …

12. What do we mean by Research Design
Need to make decisions regarding:
How subjects are selected
How many groups
What strategy for each group
What measures
How data are analyzed
Research design is the overall scheme of the study
Gives the structure of the study – how different elements of the overall scheme relate to each other to address the research questions

13. Research Design 1: Two group post-test
Overview
Assign students into two groups
Implement your strategy with one group (Experimental), do not implement your strategy with the other group (control)
Implement Strategy1 with one group (Experimental1), implement Strategy2 with the other group (Experimental2)
Measure the same variable afterwards
Check if outcome is different for different groups

14. Research Design 1: Two group post-test
When is this a suitable research design?
Useful to show that the outcome happened due to the treatment. For example:
“It was due to the strategy of peer-discussion that students’ conceptual understanding improved”

15. Research Design 1: Two group post-test
Things we should worry about (else referee would complain that our design is not valid)
Are the two groups equivalent?
Are all conditions for the two groups same except “my strategy” ?
Are the numbers large enough ?
Is the strategy implemented for each group valid (consistent with what was intended) ?
Is the post-test valid – measuring what it should? (more on this in next session)

16. Research Design 1: Two group post-test
Things we should worry about (else referee would complain that our design is not valid)
Are the two groups equivalent? How to ensure?
Randomly assigning students (randomization)
Matching
Checking equivalence

17. Research Design 1: Two group post-test
Random Assignment : why is this desirable?
Random assignment or randomization in assigning:
i) students to groups, and ii) treatments to groups
Recall - Are all conditions for the two groups same except “my strategy” ?
Randomization + large numbers =>
other conditions are “controlled”

18. ET Jargon Research Design Variable Sample
Control group and Experimental group
Treatment
Randomization / random assignment
Equivalence
Validity
(Readings will be posted on Moodle)

19. Take-away For a strong research study, have: Two-groups
Randomized assignment
Equivalent groups
Valid strategy

20. Research Designs Single group post test only
Research Designs
Single group post test only
Two group post test
Example1
Single group pre - post test
Example2
Two group pre- post test
Example3

21. Research Design 2: Single group pre-post test
Overview
Compare performance of a group to itself before and after the treatment
Measure variable of interest before implementing the strategy via pre-test
Administer treatment - Implement strategy
Measure variable of interest after implementing the strategy via post-test
Check if there is a change between post and pre-test. If there is a difference, attribute it to the treatment.

22. Research Design 2: Single group pre-post test
When is pre-test needed?
Pre-test is useful when the students’ prior knowledge varies, and could affect the outcome

23. Research Design 2: Single group pre-post test
Things we should worry about (else referee would complain that our design is not valid)
Has nothing happened between pre-test and post- test except treatment?
Are the two tests equivalent (same difficulty level, same topics)?
Is the strategy implemented valid (consistent with what was intended) ?

24. Take-away
Use a pre-test when there is a variation in prior knowledge of students
Ensure equivalence between pre- & post-tests
Do not have too much time between pre- & post-tests

25. Another example Virtual Labs to Teach Programming Anita Diwakar
Faculty member, V.P.M’s Polytechnic Mumbai
Ph.D. student, Educational Technology, IIT Bombay