1. Unit 4. Data, data collection and the research process
In this unit we will focus on the following important concepts:
Reliability (consistency) of the way data are collected.
The research process.
© 2015 Dávid & Ryan

2. Reliability – essentially consistency
Questions to consider
What would you do to increase the consistency of (lesson) observations (if it is your chosen instrument)?
What would you do to increase the consistency of your questionnaire/ your test?
Class discussion to clarify answers, alternatives.
Look at the list of techniques again (next slide) and choose one or two other instruments.
How do you think you could increase consistency in these cases?
Discuss the two questions set.
Doing obs. more often (occasions) or doing it with a number of observers.
Make the qnnaire longer or use more scales in it.
© 2015 Dávid & Ryan

3. Instruments and data collection techniques
Observation
Done in Focus on the Learner.
Interviews
Questionnaires, feedback and tests
Some of this done in Focus on the Learner.
Introspection
Concurrent/retrospective?
Verbal protocols
Repertory grids
Done in Focus on the Learner.
Text (discourse) analysis
Conversation Analysis
Diary studies
Case studies
Choosing the appropriate instruments is part of proper operationalisation.
© 2015 Dávid & Ryan

4. Reliability – consistency (contd.)
Consider the cases to the right. A good measure of reliability can only be achieved in one.
Which one?
How is the validity of conclusions threatened in all the other cases? Validity or reliability itself is the problem?
A long questionnaire. Some questions are leading, misleading or miss the point.
Brilliant interview questions capture the problem very well. A considerable number of interviews made.
Excellent ideas and questions. The number of questions is 3, and the qnnaire was done by 7 out of 15 students.
Which one? Choice 2.
No. 1 would probably be reliable, but the questions are problematic (operationalisation) so validity in question.
No. 2 has potentially high validity.
No. 3 suffers from low reliability (representativenesss) so the conclusions cannot possibly be valid either, no matter how brilliant everything else might be.
© 2015 Dávid & Ryan

5. Two important terms
Remember these terms from your studies earlier (Focus on the Learner)?
T_ _ _ _ _ _ _ _ _ _ _ _
S _ _ _ _ _ _ _ _ _
Can you recap what they mean?
Can you guess or shall we play „Hangman”?
This is the tradition from action research.
How are these two terms related to reliability?
What important aspect of research do they miss?
Triangulation and saturation (of data)
Essentially, they are based on the notion of reliability, but they do not speak of validity .
© 2015 Dávid & Ryan

6. Background
In surveying, triangulation is the process of determining the location of a point by measuring only angles to it from known points at either end of a fixed baseline, rather than measuring distances to the point directly as in trilateration. The point can then be fixed as the third point of a triangle with one known side and two known angles.
The geodesic background to the concept (Wikipedia)
© 2015 Dávid & Ryan

7. The research process 1 Class question: Assuming you have
selected the research instruments appropriately,
you have done the operationalisation successfully,
what other principle is important?
The other principle is keeping to the stages of the research process.
The stages are set out in the slide two down below.
© 2015 Dávid & Ryan

8. The research process 2
The importance of keeping to the stages of the research process
What do you think the stages are?
Discussion in pairs or groups
What sort of validity are you likely to get?
Process validity= validity from the (good) quality of the process.
© 2015 Dávid & Ryan

9. Stages of the research process
Formulating the question(s) and reading
Operationalisation: Design of instruments
Moderating and pretesting (piloting) the instruments
Data collection
Analysis
Reporting research (write- up), drawing conclusions.
Questions:
Is the ordering important?
What important quality aspect will probably be affected if the steps are ignored or some get skipped?
What sort of validity would be harmed? How would you describe it?
Left side column: stages of the process. Right side column: Students answer questions in pairs/small groups.
Validity of conclusions will be affected if steps are ignored/skipped.
Procedural (process) validity (Somlai 1997).
© 2015 Dávid & Ryan

10. A summary What is reliability a property of?
What is your observation about the relationship between reliability and validity?
The relationship between reliability and validity is asymmetrical (Bachman 1990, Messick 1989).
The possibility of being „consistently wrong” about otherwise right goals, things to do, etc.
What follows from this?
Popular beliefs: The more you increase reliability, the weaker validity becomes (Gardner et al. 2010)
Comment.
Reliability is the property of the data, e.g. scores, responses, and the particular version of the research instrument we use.
Reliability is a precondition to validity. Asymmetrical relationship. Poor reliability can undermine the validity of conclusions.
There is no necessary opposition/contradiction between rel. and val.
Ref. Gardner et al. 2010, p. 36.
© 2015 Dávid & Ryan