Introduction to Indirect Student-Learning Assessment (Part V) Summer DeProw Topeka Small Wayne Wilkinson November 9, 2016 ITTC Faculty Center Arkansas State University

Indirect Assessments Require the inference of student learning: No direct evidence or demonstration Common topics of indirect assessments: Perceptions of successfully meeting program outcomes Satisfaction/attitudes toward program Utility of program

Common Forms of Indirect Assessment Interviews Focus groups Classroom assessment techniques Curriculum and syllabus analysis Surveys

Analyzing Interviews and Focus Groups Dr. Topeka Small

Coding, Simply The first step to analyzing interview or focus group information is coding the data. Coding, loosely defined, is the process by which information is categorized to facilitate analysis. Codes are assigned to information that is similar in nature or in the same category. For example, comments from a survey may fall under two different categories: positive and negative. Perhaps you could outline all positive information in green and negative information in red. In this case, color “coding” is used. However, other codes may be used. Plus and minus signs could be used as codes. From here, information can be categorized into more narrow categories. For example, positive comments in green could be further coded as positive comments about the faculty, positive comments about the facilities, positive comments about the curriculum, etc. These comments could be delineated using another code like happy faces and sad faces. Keep in mind the “code” you use is not nearly as important as what the codes mean. Create a legend if your coding system is very intricate. After coding the information as much as necessary, you can begin to identify the themes that emerge from the data. If you used color “codes”, a simple glance reveals if there are more positive comments than negative. Then, the next level of color “coding” could easily reveal if most of the positive comments are program-related, faculty-related, curriculum-related, etc.

Analyzing Curriculum Maps and Syllabi Dr. Summer DeProw

Curriculum Maps: Horizontal Analysis

Curriculum Maps: Horizontal Analysis Questions to ask if you find that a course/s does not contribute to any of the program- level outcomes: Does this class need to be a part of the curriculum? If the class is important to the program, should you reconsider your outcomes? If the class is important to the program and the outcomes are accurate, does the class need to be updated or rethought?

Curriculum Maps: Vertical Analysis

Curriculum Maps: Vertical Analysis Questions to ask if you find that a program-level outcome is not supported by any courses: Is the outcome/s important enough to be a program-level outcome? If the outcome/s is important, should the curriculum be restructured to support the outcome/s?

Syllabi Analysis Course Descriptions Program-Level Outcomes Coordination with Curriculum Maps

Analyzing Survey Data Dr. Wayne Wilkinson

Prerequisites for Data Analysis Important qualities of survey questions: Reliability: responses are due to attribute rather than error Can be estimated and quantified Validity: The questions measure what you think they are measuring Associations with other measures (e.g., direct assessments)

Types of Data The appropriate analysis depends on scale of measurement: Nominal (numbers as categories) Ordinal (rough categorical estimates) Interval (continuous variables) Ratio (continuous with absolute zero) 1 2 3 4 Freshman Sophomore Junior Senior

Types of Data Analysis Data analysis can have multiple purposes: Descriptive: Describing the phenomenon Relational: Looking for relationships among variables Inferential: Looking for differences between groups

Bringing It All Together Scale of Measurement Analysis Type Common Procedures Nominal/Ordinal Descriptive Counts/Frequencies Relational Phi (nominal) or Tau (ordinal) correlation coefficients Inferential Chi-square Tests Interval/Ratio Central tendency (mean, median, & mode) Pearson correlation coefficient t-tests (2 groups) and ANOVA (3+ groups) Note: Other correlation coefficients exist for when the two variables being compared have different scales of measurement (e.g., point-biserial for nominal/interval data)