Introduction to Indirect Student-Learning Assessment (Part II) Dr. Wayne W. Wilkinson September 28, 2016 ITTC Faculty Center Arkansas State University

Refresher: Indirect Assessments Require the inference of student learning: No direct evidence or demonstration Common topics of indirect assessments: Perceptions of successfully meeting program outcomes Satisfaction/attitudes/feelings toward program Utility of program

Criteria: The Survey Must Have a Purpose Standards used to help make evaluative judgments Program outcomes, program quality, program utility . . . Common issues Choices of proper criteria to use (e.g., program outcomes) Disagreements over definition (what does “communicate effectively” mean?)

Conceptual vs. Actual Criteria Theoretical construct: Ideal set of quality factors Measures of conceptual criteria: Operational definitions Conceptual Criteria (English B.A. Program Outcomes) Actual Indirect Criteria Students will be able to discuss the conventions of literary genres (poetry, fiction, non‐fiction, and drama) in close reading and analysis of diverse texts. ? Students will demonstrate breadth of knowledge regarding literary traditions of British, American, and world texts and also regarding different theoretical approaches to the study of literature, language, and writing. Students will explain how texts are written and received within diverse cultural and sociohistorical contexts, including global and multicultural perspectives. Students will develop effective oral and written communication skills, including using primary and secondary sources.

Conceptual and Actual Criterion Relations Conceptual criteria are theoretical . . . actual and conceptual should overlap Criterion deficiency Criterion contamination Criterion relevance

Content and Face Validity Content Validity: The degree that an assessment includes relevant aspects of the criteria Determined by Subject Matter Experts Face Validity: do the items seem legitimate for a measure of the criteria? Reactions and attitudes of test takers

Types of Items Open-ended Restricted (closed-ended) Partially open-ended Rating scales

Rating Scales Likert-type scale: Sematic differential: Aggregated ratings of favorable or unfavorable statements Sematic differential: Bipolar adjectives Measure several variables on common scale

Rating Scale Issues Number of points on scale: Labeling of scale: Odd or even (neutral point) Labeling of scale: Number of anchors Numerical values Item-rating scale congruence

Writing Good Items Avoid very long items Avoid negative statements (no, none, not, etc.) 5th – 6th grade reading level Avoid “check all that apply” items

Writing Good Items Avoid items that express more than one thought (double barreled items) Avoid evaluative assumptions The program aided my understanding of selecting and conducting the proper analysis for specific research questions. The program provided an excellent preparation for my career.

Assembling Your Survey Survey space and justification Question organization: Keep related items together Question order can have unintended effects Sensitive topic items after less sensitive Graphic navigation paths

Survey Evaluation Session Survey goal: English BA Program Indirect Assessment Justification for each item Criteria Content Validity Contamination or deficiency Item issues Conceptual Criteria (English B.A. Program Outcomes) Students will be able to discuss the conventions of literary genres (poetry, fiction, non‐fiction, and drama) in close reading and analysis of diverse texts. Students will demonstrate breadth of knowledge regarding literary traditions of British, American, and world texts and also regarding different theoretical approaches to the study of literature, language, and writing. Students will explain how texts are written and received within diverse cultural and sociohistorical contexts, including global and multicultural perspectives. Students will develop effective oral and written communication skills, including using primary and secondary sources. Common indirect assessment topics: Perceptions of successfully meeting program outcomes Satisfaction/attitudes/feelings toward program Utility of program