Survey Design Workshop: Uses, Payoffs, and Pitfalls

Workshop Agenda: Questionnaires revisited... Uses, benefits, and limitations Question Wording and Levels of Data Activity: Critique of the exemplar by Li, D. (1998). 2. Interviews Things to consider Creation of questions for the exemplar by Li, D. (1998).

Questionnaires: Uses To get a ‘snapshot’ of what is happening now: factual/demographical information information about attitudes/preferences information about behaviour can obtain correlational data

Benefits of Questionnaires: we gain insight into what is happening now data collection is efficient, easy questionnaires are inexpensive to administer sites like Survey Monkey ease the process

Limitations of Questionnaires: can not infer a causal relationship*** →the independent variable is not controlled like in an experimental method the level of questioning may limit the type of analysis you can do (more on this later) generalizability may be difficult with a small sample size consider that your findings may be contextual (N=?; random ?) getting the questions ‘right’ is crucial and requires skill

Things to consider in writing questions In addition to the criteria we studied, it is important to: ask questions sensitively: some topics worth studying are sensitive in nature ie. morals, religion, attitudes and experiences with racism

Quick Review of Question Types Closed-ended questions: limited options in the answer Types: Nominal, Ordinal, Interval, Ratio Open-ended questions: a wide variety of answers Types: fill in the blanks, short answer, partial dialogue

Planning your questionnaire Theory Research questions Questionnaire questions (content) Questionnaire questions (format) Data Analysis Findings and Conclusions

Considerations for close-ended question format and levels of data: Because respondents are ‘forced’ to answer from the options provided, you cannot get into the depth of their answers (hence, a ‘snapshot’) You may find that afterwards, you lack a deeper understanding of the issue than you intended You will have to pilot your questions beforehand You have to apply the right level of questioning in response to your research question and intended data analysis

Nominal Questions What are they?  attributes are named; they are categorical only ie. Demographics, favorite television show What kind of stats can you obtained ?  frequency of each category and mode  percents and frequencies can be used to test your hypothesis What stats can’t be obtained? A median or mean.  You can’t talk about average scores since averages are numerical, not nominal.

Ordinal Questions What are they?  ranking of items, they measure direction of answer choices ie. beginning, intermediate, and high What kind of stats can you describe?  frequency of each category and mode  percents and frequencies can be used to test your hypothesis  median, the middle value from the respondents’ answers What stats can’t be obtained? mean.  averages are numerical, not nominal.

Interval Scale Questions What are they?  the distance between intervals is numerical, equal ie. Likert scale My teacher is friendly: 1 2 3 4 5 ie. Semantic differential scale My teacher is unfriendly _ _ _ _ _ friendly.

Interval Scale Questions What kind of stats can you describe? → you can get all measures of central tendency INCL. mean Why? Interval scales are numerical. The benefit: With a mean AND a large sample size, you can begin to generalize. (T test) ?

Activity: Critique of the exemplar Li, D. (1998). “It is always more difficult than you plan and imagine”: Teachers’ perceived difficulties in introducing the communicative approach in South Korea. TESOL Quarterly 32(4) 677-703 Jeff, how is our APA?

Let’s begin with Li’s research question p. 682

Interviews The 3 major interview types can be classified as unstructured, semi structured and structured. Patton (1990)groups them into: informal conversational interviews, general interview guide approach & standardized open ended interviews

Uses To learn about teachers and students background To learn more about teachers and students “Reported Behaviour” To learn more about teachers and students “opinions and attitudes”

Informal interviews BENEFITS LIMITATIONS Informal/ conversational interviews Good for Intensive Case Studies Open ended length Flexibility in asking interviewee different questions Unstructured Time consuming for gathering systematic information Not advisable for large populations Difficult to find patterns in data collected

General interview guide BENEFITS LIMITATIONS General interview guide approach More structured in terms of question design Allows for easier data compilation Interviewer refrains from pursuing open ended Questions Less difficult to establish patterns in data collection

Standardized open- ended BENEFITS LIMITATIONS Standardized open ended interview Highly structured using exact wording Data Collected is easy to compile and Analyse No Flexibility in Format

Focus Group Interviews BENEFITS LIMITATIONS Focus Groups More People (6-8) can be interviewed at a time Less Time consuming for Data Collection No Flexibility. Power Relationships/ Asymetrical Relationships Requires Facilitation Not very useful for compiling statistical data

Things to consider Question wording Power Relationships Recording the interview: Note taking/ tape recording) Interview Data Analysis (case analysis/ cross case analysis) Content Analysis: topic labelling + coding General Limitations: Not everything an interviewee says reflects reality.

Characteristics of successful interview designs Good question wording such that yes /no questions are avoided Minimize biases & avoid “leading questions Brief and debrief: Inform interviewees about the study and provide some assurance to students ( anonymity or confidentiality if necessary) Ethics Be sensitive to students/ respondents’ responses

Activity Write 5-10 Questions you would use in an interview

References Gravetter, F.J. & Wallau, L.B. (2016). Statistics for the behavioral sciences.(10th ed.) Wadsworth, CENGAGE Learning. Li, D. (1998). “It is always more difficult than you plan and imagine”: Teachers’ perceived difficulties in introducing the communicative approach in South Korea. TESOL Quarterly 32(4) 677-703 McKay, S. L. (2006). Researching second language classrooms. Mahwah, NJ: Lawrence Erlbaum Associates. Nunan. D. & Bailey, K. M. (2008). Exploring second language classroom research: A comprehensive guide. Boston: Heinle ELT