1. Surveys as Student Learning Measures CLAS Unit Head Meeting September 27, 2006 Maria Cimitile, CLAS Associate Dean Priscilla Carreras, CLAS Graduate Assistant Julie Guevara, Accreditation and Assessment Officer

2. What are surveys? Systematic efforts to collect information about people by asking them to respond to specific questions. Systematic efforts to collect information about people by asking them to respond to specific questions. Helpful in gaining backgrounds, experiences, plans for the future, opinions, and attitudes. Helpful in gaining backgrounds, experiences, plans for the future, opinions, and attitudes. Can use paper, phone, or internet format. Can use paper, phone, or internet format. Provide insight into student attitudes about programs, assignments, courses, exams etc. Provide insight into student attitudes about programs, assignments, courses, exams etc.

3. Survey Planning and Conceptualization Should you use a survey? Should you use a survey? What do you want to learn? What do you want to learn? Who is the sample or audience? Who is the sample or audience? How will you distribute the survey? How will you distribute the survey? Sampling techniques – interviews vs. individual survey. Sampling techniques – interviews vs. individual survey.

4. Survey Planning and Conceptualization Obtain representative sample. Obtain representative sample. Determine time and people power. Determine time and people power. Evaluate survey process when done. Evaluate survey process when done. Timeline – when it’ll be created, returned. Timeline – when it’ll be created, returned. IRB Institutional Review Board. IRB Institutional Review Board. Divide items into two categories: “have to know” and “nice to know”, then omit “nice to know”. Divide items into two categories: “have to know” and “nice to know”, then omit “nice to know”.

5. Designing the Survey Questions Close-ended Questions Closed ended questions are those questions, which can be answered finitely. Quick and require little time investment. Can also provide incomplete responses. By definition, these questions can be restrictive. Closed ended questions are those questions, which can be answered finitely. Quick and require little time investment. Can also provide incomplete responses. By definition, these questions can be restrictive. Examples:Multiple choice Examples:Multiple choice Rankings Rankings Rating Scale Rating Scale Ecosystem Rating Scale Likert Scale Likert Scale

6. Multiple Choice Quickly answered and tallied. No room for unexpected answers. Quickly answered and tallied. No room for unexpected answers. Chemistry Example: - Goal: Graduate students who are well equipped to enter the job market. - Objective: Technical emphasis majors will enter the workforce and apply their skills and knowledge. - Measure: develop and send employer surveys to companies that hired our graduates 6-12 months after they’ve been hired.

7. Rankings Tedious. Tedious. Assume respondents feel differently about each response. Assume respondents feel differently about each response. Do not provide size of feeling between ranked items. Do not provide size of feeling between ranked items. AVOID THEM!!! AVOID THEM!!! Example: on a scale where “10” means you have a great interest in a subject and “1” means you have none at all, how would you rank your interest in each of the following subjects?

8. Rating Scale Another type of multiple choice question. Another type of multiple choice question. Example: States conclusions based on data collection.  Excellent  Good  Fair  Poor

9. Eco-system Rating Scale Ask for two ratings. The second gives information on the environment in which the first was made Ask for two ratings. The second gives information on the environment in which the first was made Observes difference between growth obtained and growth desired. Observes difference between growth obtained and growth desired. Measures attainment of goals and objectives. Measures attainment of goals and objectives. Example: Ability to conduct a psychosocial assessment Importance: Not Critical __1 __2 __3 __4 __ 5 Vital Preparation: Poor __1 __2 __3 __4 __ 5 Excellent

10. Likert Scale (Agreement Scale) Efficient Efficient Permits comparison Permits comparison Can be used in place of ranking Can be used in place of rankingExample: Employees exhibit requisite safety skills in the lab. Strongly Agree Agree Disagree Strongly Disagree NA

11. Numeric Open-ended Questions Example: In the past two years (1/04 – 1/06) how many of our graduates have you hired?

12. Text Open-ended Questions Participants compose their own answers. Participants compose their own answers. Provide infinite response. Provide infinite response. Develop trust, are perceived as less threatening, allow an unrestrained or free response, and may be more useful with articulate users. Develop trust, are perceived as less threatening, allow an unrestrained or free response, and may be more useful with articulate users. Not popular in surveys because: Not popular in surveys because: Lengthen time required to complete survey. Lengthen time required to complete survey. Require more mental energy. Require more mental energy. Examples: How can we achieve the objectives of this course?

13. Writing Good Survey Questions Keep It Short and Simple (KISS) Keep questions short. Keep questions short. Each item should ask only one question. Each item should ask only one question. Use simple vocabulary – NO JARGON! Use simple vocabulary – NO JARGON! Make definitions, assumptions, and qualifiers clear. Make definitions, assumptions, and qualifiers clear. Do not make significant memory demands. Do not make significant memory demands.

14. Writing Good Survey Questions Do not ask for broad generalizations of attitudes “Do you consider yourself a good student?” Do not ask for broad generalizations of attitudes “Do you consider yourself a good student?” Do not ask questions which are uncomfortable to answer honestly. Do not ask questions which are uncomfortable to answer honestly. When possible, give the option of not knowing; Not applicable. When possible, give the option of not knowing; Not applicable. Avoid asking leading questions Avoid asking leading questions Ex. Why are GVSU students your best employees? Ex. Why are GVSU students your best employees? Define terms, qualifiers, assumptions, etc. Define terms, qualifiers, assumptions, etc.

15. Maximizing Participation Rate Professional appearance of survey is important. Professional appearance of survey is important. People are more likely to complete and return a survey if it looks like someone cares about it. People are more likely to complete and return a survey if it looks like someone cares about it. Be considerate of your participants. Be considerate of your participants. Include self-addressed & stamped envelope. Include self-addressed & stamped envelope. Give adequate time for response. Give adequate time for response. Remember that they are doing you a favor. Remember that they are doing you a favor.

16. Telephone Survey Survey should be no more than 10 minutes. Survey should be no more than 10 minutes. Use simple format, with easy to follow directions. Use simple format, with easy to follow directions. Avoid open-ended questions. Avoid open-ended questions. Pre-test survey by reading it to people without looking at them. Pre-test survey by reading it to people without looking at them. Document attempts to reach participants. Document attempts to reach participants.

17. Employer Surveys Before completing employer survey, inform alumni. Before completing employer survey, inform alumni. Skills Skills Importance of skills in particular field/work place. Importance of skills in particular field/work place. Skills employee attained at university or in specific program. Skills employee attained at university or in specific program.

18. References Creative Research Systems (2006). The Survey System. http://surveysystem.com/sdesign.htm Creative Research Systems (2006). The Survey System. http://surveysystem.com/sdesign.htm http://surveysystem.com/sdesign.htm Introduction to Developing Surveys. Georgia Tech Office for Assessment. http://www.assessment.gatech.edu/eReports/slide_shows/Short_Gu ide_to_Survey_Research.pdf#search=%22developing%20surveys% 22 Introduction to Developing Surveys. Georgia Tech Office for Assessment. http://www.assessment.gatech.edu/eReports/slide_shows/Short_Gu ide_to_Survey_Research.pdf#search=%22developing%20surveys% 22 http://www.assessment.gatech.edu/eReports/slide_shows/Short_Gu ide_to_Survey_Research.pdf#search=%22developing%20surveys% 22 http://www.assessment.gatech.edu/eReports/slide_shows/Short_Gu ide_to_Survey_Research.pdf#search=%22developing%20surveys% 22 Palomba, C.A. and Banta, T.W. (1999). Assessment essentials. San Francisco, CA : Jossey-Bass. Palomba, C.A. and Banta, T.W. (1999). Assessment essentials. San Francisco, CA : Jossey-Bass. Richardson, Dr. John V. (2002). Open versus closed ended questions in the reference environment. http://polaris.gseis.ucla.edu/jrichardson/dis220/openclosed.htm Richardson, Dr. John V. (2002). Open versus closed ended questions in the reference environment. http://polaris.gseis.ucla.edu/jrichardson/dis220/openclosed.htm http://polaris.gseis.ucla.edu/jrichardson/dis220/openclosed.htm Survey How To (2006). University of Texas at Austin Division of Instructional Innovation and Assessment. http://www.utexas.edu/academic/diia/assessment/iar/how_to/meth ods/survey.php Survey How To (2006). University of Texas at Austin Division of Instructional Innovation and Assessment. http://www.utexas.edu/academic/diia/assessment/iar/how_to/meth ods/survey.php http://www.utexas.edu/academic/diia/assessment/iar/how_to/meth ods/survey.php http://www.utexas.edu/academic/diia/assessment/iar/how_to/meth ods/survey.php Suskie, L. (2004). Assessing student learning: A common sense guide. Bolton, MA : Anker Publishing. Suskie, L. (2004). Assessing student learning: A common sense guide. Bolton, MA : Anker Publishing.

19.  USE SPARINGLY When there are more than 6 possible answers. When there are more than 6 possible answers. When asking for occupation. When asking for occupation. When structured item may bias responses by steering participants in one direction. When structured item may bias responses by steering participants in one direction.  When asking a high interest question, such as “How would you change/improve/etc.”

20. Student Surveys Surveys of graduating seniors. Surveys of graduating seniors. Longitudinal Surveys of students attending university, enrolled in specific program. Longitudinal Surveys of students attending university, enrolled in specific program.

21. Alumni Surveys Employment related surveys. Employment related surveys. Programmatic. Programmatic. Departmental mission Departmental mission Skill attainment Skill attainment

22. Who to Survey? Students Students Alumni Alumni Employers Employers