1. ISTE Workshop Research Methods in Educational Technology
IIT Bombay
February 2-9, 2013

2. Session 5:
Delving into RM-ET

3. Our Goals as ET Researchers
Our Goals as ET Researchers
conduct systematic studies to get data about whether our ideas are working.
provide evidence to support our conclusions.
We saw how to set up a research study.
Next -
What data to collect? How to collect it?

4. Session 5e:
What data to collect?
How to collect it?

5. What is the goal of your study?
Learning Effectiveness – student performance
Engagement – student interest, satisfaction

6. How to measure learning effectiveness?
What to measure?
performance on test related to concept in study
How to measure? Need instrument / tool.
Use standardized test:
concept-inventory, test for specific ability, rubrics
Create instrument:
conceptual questions, problems specific to research objective (for ex., write a program)
Measure what is relevant:
For ex., analyze number of errors in the program
marks in regular quiz or final exam (weak)

7. How to measure student engagement?
What to measure?
– student perception of their own learning
satisfaction
interest in course topics / course format
attendance
How to measure ?
Questionnaire to measure perception of learning / satisfaction / interest
Carefully structured interviews (not simply a conversation)

8. How to measure student engagement?
Creation of instrument : Some Do’s & Don’ts
Don’t
Instead Do
Simply ask –
Did you like / dislike it?
Ask questions related to what you want to measure – for ex. perception of learning or satisfaction
Single leading question
Is the method interesting?
Ask many specific questions related to what you want to measure
Open descriptive question (analysis is hard)
Preferably use a scale / rating / ranking

9. Take-away
Follow guidelines to systematically create an instrument that addresses your research objective
Links to readings of systematic instrument creation will be posted on Moodle

10. Instrument validity and reliability
Is your test / questionnaire accurate ? Does it measure what it is supposed to measure?
If yes, instrument is valid.
Is your test / questionnaire precise? Does it give similar results under similar conditions?
If yes, instrument is reliable.
Robust instrument = valid and reliable

11. Notion of validity and reliability
Neither valid nor reliable

12. Notion of validity and reliability
Neither valid nor reliable
Valid but not reliable

13. Notion of validity and reliability
Neither valid nor reliable
Reliable but not valid
Valid but not reliable

14. Notion of validity and reliability
Neither valid nor reliable
Reliable but not valid
Valid but not reliable
Both valid and reliable

15. You need to show that your instrument is both valid and reliable
Take-away
You need to show that your instrument is both valid and reliable
Links to readings for methods to show instrument validity and reliability will be posted on Moodle

16. How to strengthen study claims: Notion of triangulation
What is triangulation?
Use of two or more data collection methods
Why triangulation?
Study claims are
stronger since
multiple methods
used to establish
same conclusion
Measurement Method 1
Measurement Method 2
Conclusion

17. Notion of Triangulation
How to triangulate?
Multiple data sources such as:
Test of conceptual understanding + survey of student perception of learning (to establish claim that “my strategy works”)
Quantitative questionnaire (to understand what ) + qualitative interview (to understand why)

18. Take-away
Use two (or more) data collection methods to establish evidence for your claim.

19. More ET Jargon Research Design Variable Sample
Control and experimental group
Treatment
Random assignment
Equivalence
Validity
(Readings will be posted on Moodle)
Validity
Reliability

20. Final Example Interactive Visualizations in Signals & Systems
Mrinal Patwardhan
D. J. Sanghvi College of Engineering, Mumbai
Ph.D. student, Educational Technology, IIT Bombay