1. Coordinator, UWS SoTL Leadership Site
Gathering SoTL Evidence: Methods for Systematic Inquiry into Student Learning
Renee A. Meyers
Coordinator, UWS SoTL Leadership Site
Faculty College, May 2009

2. Using Questions to Gather SoTL Evidence
Types of Questions
Question Design
Question Issues and Problems
Types of Response Scales
Design and Preparation of Survey

3. Asking Questions To Gather Evidence

4. Methods for Gathering Evidence
Evidence can include:
Course-Intrinsic Evidence
Course-Extrinsic Evidence
Using your research question, what is one type of course-intrinsic and one type of course-extrinsic evidence you might collect?

5. Asking Questions
All SoTL evidence gathering is focused on asking questions of students.
Research on writing survey questions provides comprehensive information on how to do this best

6. Survey Evaluation Look at survey in packet. What works?
What doesn’t work?
What is missing?

7. What Survey Research Tells Us About Asking Good Questions

8. The Impact of Questions on the Research Process
Poor questions lead to lousy results.
How might answers to these questions differ?
Do you attend class?
Do you regularly attend class?
How many times per week do you attend this class?
Asking survey questions is part science and part art.

9. Types of Survey Questions
Open Ended and Closed Ended Question Formats
Attitude, Knowledge, and Behavior Question Formats

10. Open-Ended Questions
Open-ended questions ask respondents to state in their own words their response to the question.
Closed-ended (or fixed alternative) questions ask respondents to respond by picking one or more alternatives from a list of available options.
Can you find any examples of open-ended questions on the survey? Are these good open-ended questions?

11. Open Ended Questions
What are advantages and disadvantages to open- and close-ended questions?

12. Advantages and Disadvantages
Are often less time consuming
May provide more complete answers
May provide information researchers did not anticipate
Responses are often easier to analyze
May produce better response rate

13. Surveys: Asking Questions
Surveys are used most often in SoTL research when we are interested in asking questions about:
how students FEEL or THINK about certain issues, topics, activities, related to their learning,
what students KNOW about various topics, etc.
how often students engage in certain learning BEHAVIORS

14. Attitude/Feelings Questions
Attitude/feelings questions ask respondents for their views on certain topics or issues
There are no right or wrong answers to attitude/feeling questions.
Attitude/feeling questions should be “neutrally” written so that the respondent is not led to believe that a certain response is desired.

15. Knowledge Questions
Knowledge questions ask about what a respondent knows about a particular topic
Knowledge questions can be asked as a screening question to determine if respondents are qualified to answer attitude questions.
“What do you know about evolution?
“How do you feel about evolution as a theory of creation?”

16. Behavior Questions
Behavior questions ask respondents to report on their past or future behavior
Responses to behavior questions may be biased because of faulty or biased recall by respondents

17. Attitude, Knowledge, and Behavior Questions
Can you find an attitude/feelings, knowledge, and behavior question on the survey?
Write one attitude/feelings, knowledge, or behavior question for your own SoTL project.
Pre-assessment measure
Question for an assignment you will collect
Post-assessment

18. Question Issues and Problems

19. Recall Questions
Is there a question on survey that asks students to “recall” some behavior?
How would you evaluate these questions?
Would you know how to answer them?
Is there a better way to ask these questions?

20. Using Diaries for Recall Questions
To minimize recall errors, researchers can ask respondents to keep personal diaries to record their actual teaching or learning activities over a period of time
Caution: Respondents may forget to fill out their diaries on a timely basis and instead “catch up” their diaries just before they are due to be collected

21. Unaided vs. Aided Recall Questions
Because respondents often forget, or fail to recall accurately, their activities and behavior, you can use aided recall questions to “jog” their memories.
Unaided recall: “Please list all of the Communication classes in which you participated in learning groups last semester.” (open-ended question)
Aided recall: “Please place a check by any of the Communication courses in which you participated in a learning group last semester.” (closed-ended question)
Public Speaking
Introduction to Interpersonal Communication
Small Group Communication

22. Recall Questions and Time Frame
We often want respondents to focus on a specific time frame when answering a question.
Avoid ambiguous terms like “past year” or “next year” because respondents may not know if this is a calendar year (i.e., 2006, 2007, 2008) or a 12-month period.
Avoid time frames that are too large (for the recall question) or too far in the past.
Write one recall question for gathering evidence for your SoTL research question.

23. Intimidating or Inappropriate Questions
Respondents will consider a question to be intimidating or inappropriate if:
It addresses highly personal and private behavior (drinking behavior, sexual activity, illegal drug use, etc.)
It addresses subjects for which the respondent believes there are socially desirable norms (communication with instructors, behavior in class, spiritual behavior, etc.)
If respondents perceive a question to be intimidating, they are more likely to refuse to answer or to give an inaccurate answer.

24. Intimidating Questions
Would you use an open- or closed-ended question to ask a potentially intimidating question to gather evidence from your students?
Examples?

25. Forms for Asking Potentially Intimidating Questions About Behavior
If you ask a closed-ended question with a set of response options, people typically do not choose the extreme options.
Therefore, it is often better to use an open-ended question and allow the respondents to indicate how often they engage in certain behaviors, or define the terms as they perceive them.

26. Forms for Asking Potentially Intimidating Questions About Behavior
Third person questions ask respondents to report on the behavior and activities of other people.
Example: Instead of asking college students directly about their alcohol consumption behavior and its impact on their learning, you could ask students to report how much alcohol their roommates consume and how that affects their learning behavior.

27. Double-Barreled Questions
Double-barreled questions ask respondents for their opinions about two distinct issues in the same question.
“Do you prefer working in groups and learning from your fellow students, or do you prefer lectures and learning from the instructor?
Any other examples?

28. Leading and Loaded Questions
Some “surveys” seek to generate results that support a set of predetermined goals and objectives.
How might we re-write a question on the survey to make it a leading question?

29. Survey Question Design

30. Ordering of Questions
Start with easy, salient, and non-threatening questions first.
Complete questions on a single topic before moving on to a new topic.
When switching topics, use transitional phrases to make it easier for respondents to switch their train of thought.
Put demographic questions near the end.
Do a quick evaluation of survey to see how it fits criteria.

31. Funnel Questions
Questions regarding a specific topic should usually start with the most general questions and then follow with increasingly more restrictive questions.
If you were required to work in groups in your class, what types of students would you want in your group?
If you had to select just three key characteristics that you would use to choose among these different students, what would these 3 factors be?
Would “student’s GPA” be an important factor in your choice of students for your work group?
If your instructor indicated that she was going to put students in groups according to similar GPAs, would you be interested in working in such a group?

32. Skipping and Branching
Surveys can instruct respondents to skip over certain questions if these questions do not pertain to them.
Question 7: “Have you worked in a classroom learning group in the past semester?”
____ Yes (please proceed to question 8)
____ No (please skip to question 12)
Any examples on survey?

33. Demographic Questions
In most surveys, questions regarding demographic characteristics of the respondents are included at end.
Typical demographic questions:
Sex
Ethnicity/Racial background
Age
Year in School
Major
Are there better ways to write these demographic questions on the survey?

34. Wording of Survey Questions

35. Avoid Ambiguity in Wording
Some common words, such as “regularly” or “often”, may be interpreted very differently by respondents.
Do you go to talk to your instructor regularly (outside of class)?
How often do you go talk to your instructor (outside of class):
1-2 times a week
3-4 times a week
5-7 times a week
More than 7 times a week
What might be the issue with these questions?

36. Response Scales Used To Collect Survey Data

37. Features of Response Categories
Any response category must be mutually exclusive.
There should not be any overlap in the categories.
Any response category system should be collectively exhaustive.
All possible responses should be included in the category system.
Usually it is advisable to include an “other” category to allow respondents to add a response that is not included in the category system.
On survey, evaluate questions for exclusivity and collective exhaustion. Any issues?
Could respond more than once a day; need an ‘other’ category

38. Semantic Differential Scales
A scale, typically used to measure attitudes or perceptions, that includes end points that are anchored by bipolar adjectives.
Outstanding _ _ _ _ _ _ _ Unacceptable
Strong _ _ _ _ _ _ _ Weak
Moral _ _ _ _ _ _ _ Immoral
Any examples of a semantic differential scale on survey?
Question 4, 7

39. Likert Scales
A scale, typically used to measure attitudes, that usually includes end points of “strongly agree” and “strongly disagree.”
Abiding by page limits is important for writing a good paper:
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
Any examples of a Likert scale on the survey?
Aren’t any All are numbered

40. Numerical Scales
A scale that uses multiple numbers, usually 5 to 10, with anchored end points to represent the strength or position of respondents’ attitudes.
1 strongly disagree
2 disagree
3 neutral
4 agree
5 strongly agree
Any examples of a numerical scale on survey?

41. The Use of “Don’t Know” and “Not Sure” and “Not Applicable”
Rather than have respondents feel forced to give an invalid response, researchers often add a category to scales that is labeled “don’t know” or “not sure” or “not applicable”.
Write a question for your research study that includes a response scale. Then explain why you wrote the question using that response scale.

43. Survey Design and Preparation

44. Length of Surveys
How long respondents will spend answering a survey is directly related to the saliency of the issues for them.
For salient topics, questionnaires of pages are possible.
For non-salient topics, questionnaires are usually limited to 2 to 4 pages.
How does this survey rate on these criteria?

45. Introducing the Survey
In a cover letter or introductory comments, the following points should be addressed:
What the study is about and its relevance to the participant;
Why the respondent’s cooperation is important;
Promise of confidentiality and security of information (IRB rules and regulations must be followed);
Reward for participation (if relevant);
Deadline for completion (if relevant);
Who to contact regarding any questions;
Thank you
Evaluate survey in terms of these criteria

46. Pretest, pretest, pretest
You should always pretest draft versions of your questions with representatives from the target population.
But never pretest your questionnaire with people who may be in your final sample of respondents.
Who might I ask to pretest this survey?

47. Online survey help sites
Advanced Survey
Zoomerang
Web Online Surveys
SurveyMonkey

48. Final Comments on Survey Design
Never underestimate the time to takes to create a high quality survey instrument.
No survey is ever perfect.
There is no substitute for pre-testing and soliciting feedback from other research experts.
Shorter is always better.
Only ask questions on surveys that are directly related to specific and important research issues.

49. Reflection
How might I use questions to collect evidence for my research project?
What will be most important for me to remember about the questions I want to ask?