1. Unit 7. Questionnaires and tests
Reason for dealing with them together:
their similarity, sharing
many of the techniques (item types)
E. g. hard to distinguish between, e.g. multiple-choice as a questionnaire or test
preparation techniques and relevant useful advice
stages of the development process
widely used in research
Best for studies in which a cross-section of answers, opinion, etc. is sought for, at a particular time.
Cohen, L. Manion, L. & Morrison, K. (2007). Research methods in education. London: Routledge. pp
Dörnyei, Z. with Tatsuya Taguchi (2010). Questionnaires in Second Language Research. Construction, administration, and processing. Second edition. Abingdon, UK/New York,NY: Routledge.
© 2015 Dávid & Ryan

2. Types of questionnaire items
Can different types of questions mix in a questionnaire?
Dichotomous questions
MC questions
Rank order questions
Rating scales
Likert
Semantic differential
Constant sum questions
Ratio data questions
Open ended questions.
Yes, but in the process of analysis/calculation they cannot be treated together, as if they were of the same type.
The answer leads us to four different types of scales (data), the discussion of which will follow later.
© 2015 Dávid & Ryan

3. Must focus on the respondent
Special focus on the respondent. Why?
Underlying all these shared features is the situation the respondent is in:
To what extent is the respondent in more of a position to negotiate the meaning, intent or wording of the question
when responding to closed items?
when responding to open ended items?
What implications does this important question have for the process of research?
The construction, the moderation (pre-pilot) and pretesting, and anaylsis of pretest are stages that are of the utmost importance, esp. in the case of closed types of questions.
Implication: Respondent is „at the mercy” of the qnnaire/test constructor.
© 2015 Dávid & Ryan

4. What do researchers expect of items?
All must contribute usefully, with information, to the “whole picture”, i.e. to answering the research question.
If it is scored, those who score(d) high on the whole of the qnnaire, should also score high on each individual item.
A kind of consistency
The typical problem: Overall high respondents lose points, whereas overall low respondents score high.
The construction, the moderation (pre-pilot) and pretesting, and analysis of pretest are stages that are of the utmost importance, esp. in the case of closed types of questions.
Implication: Respondent is „at the mercy” of the qnnaire/test constructor.
© 2015 Dávid & Ryan

5. Piloting
While pretesting is large scale (as some say) , it is possible to construe piloting as
Very small scale
Informal
Early in the research
Followed by proper pretesting on larger sample
If piloting is conceived in this way, what is it good for and what would you do?
To formulate your answers, work from this list:
Response choices in MC items
Choice of question types
Items to further explore a question.
Timing
 2015 Dávid & Ryan

6. The stages –with a task Formulating the question(s) and reading
Operationalisation of question (instruments)
Moderating and pretesting (piloting) the instruments
Data collection
Analysis
Reporting research (write- up), drawing conclusions.
Questions:
Which stages are most important from the respondents’ point of view?
What is the importance of stages 1 and 2?
What more should not be forgotten about stage 3?
Stages 1,2,3.
The „translation procedure” should faithfully reflect research questions.
Moderation, pretesting crucial for closed types of question, but also important for open types.
© 2015 Dávid & Ryan

7. What use is piloting?
Most of the uses of piloting (pretesting) are also common sense. Can you work out what exactly to do, with the help of these prompts?
Questions/ items
Irrelevance
Redundancy
Chances of misunderstanding
Sensitive topics
Extreme easiness/difficulty
Double-barreled formulations
Extreme complexity
Offensiveness, intrusiveness
Approriacy of question stems
Leading the respondent.
The qnnaire as a whole
Readability
Ambiguities and poor wording
Layout
Sections
Instructions
Appearance
Reliability
Based on/inspired by Cohen, Manion and Morrison (2007)
 2015 Dávid & Ryan

8. The importance of piloting
Piloting/pretesting is to find out about problems in advance of using the instruments.
A highly important stage of the research process is piloting/pretesting
Especially important in the preparation of questionnaires, or other instruments, where respondents (by and large) cannot negotiate the questions.
Some difference is made between the terms:
Piloting supposedly more informal, pretesting is more formal and typically larger scale.
 2015 Dávid & Ryan

9. More or less of structure
Continuum of structuredness
Between completely stuctured and unstructured
The location of qnnaire on continuum depends on a number of features.
Fit 6 ends to complete question and answer:
Towards which end of the continuum would these features push your qnnaire if ….
the size of sample of respondents is small?
open ended questions seem suitable?
the sample is large?
numerical answers are needed?
closed questions may have to be used?
non-numeric answers are needed?
Less structured: 1, 2, 6. More structured: 3, 4, 5.
© 2015 Dávid & Ryan

10. The quality of operationalisation
Remember the term? Or the synonym?
Does the question reflect the problem well?
Wording OK?
Do the questions in the qnnaire match the research questions?
How many match each? All relevant?
What different types can you identify? Is the choice of type appropriate to the kind of info sought?
For visual types, especially: drawing a flowchart
One or more subsets of questions?
© 2015 Dávid & Ryan

11. Critique a questionnaire
Student research into the BA group oral at ELTE
Based on qnnaire in combination with observation
A note about cross cultural validity in Sacks, Schegloff and Jefferson (1974) p. 700
Potentially interesting research – the questions were:
How does the cultural background of the candidates have an effect on language use in a conversation?
Can different cultural backgrounds weaken the validity of test results during oral language testing?
If so, what are the problems and what would be a possible solution?
Sacks et al. were aware that their observations/conclusions regarding the norms of conversation may be biased in favour of Western (North-American or European) speakers. Sacks, H, E. A. Schegloff and G. Jefferson (1974). A Simplest Systematics for the Organisation of Turn-taking for Conversation. Language. 50,
The tutor may of course use any suitable other qnnaire with which the problems can be illustrated.
© 2015 Dávid & Ryan