1. Developing and Applying Tools: Rubrics and Exam Questions Linda Dickens, Ph.D Dawn Zimmaro, Ph.D. John Kucsera, M.A.

2. Agenda 1.Overview of SACS 2.Mapping Assessments to Program Outcomes 3.Writing Critical Thinking Exam Questions 4.Creating Rubrics

3. Academic units Administrative units

4. Agenda 1.Overview of SACS 2.Mapping Assessments to Program Outcomes 3.Writing Critical Thinking Exam Questions 4.Creating Rubrics

5. Mapping Assessments to Program Outcomes At what level should you collect data to measure your outcomes? Classroom Program

6. Issues to consider What existing course assignments, exams, projects, etc. can you use to measure your learning outcomes? What program level assessments can you identify or develop? What is unique about your program that is measurable?

7. Classroom level Exam Assignment Presentation Paper Project

8. Program level Exit exam Thesis/dissertation Capstone course paper or project Portfolio Licensure/professional exam Exit survey

9. English Department B.A. Example learning outcome: Students will be able to construct arguments and communicate them effectively in writing.

10. Classroom level assessment Read description of E 370W – Major Authors (Emerson)

11. Assignment Writing assignment that asks students to develop an argument about an issue –Emerson’s efforts to examine the authority and effects of religious and educational institutions help frame discussions about literature and education for subsequent generations. –Given the above statement develop a central thesis with supporting arguments about how Emerson’s works have helped to frame discussions about literature and education.

12. Program level assessment A collection of individual student’s writing assignments on constructing arguments taken from several English courses –All writing assignments are evaluated as a body of work –Over the course of the whole degree program

13. Second learning outcome Example learning outcome: Students will be able to demonstrate knowledge of the history and the structure of the English language.

14. Classroom level assessment Classroom exam –Topics: History of English: Old to Early Modern English History of English: Modern and Future English Topics should all relate to the learning objective

15. Program level assessment Exit exam –Questions taken from all the courses that relate to the learning outcome

16. Sample assessment map See handout

17. Discussion Have you tried this approach? What worked? What didn’t? If you haven’t tried this, why not?

18. Agenda 1.Overview of SACS 2.Mapping Assessments to Program Outcomes 3.Writing Critical Thinking Exam Questions 4.Creating Rubrics

19. Writing Critical Thinking Exam Questions Purpose of session Learning Outcomes Critical thinking exam questions Objective and Subjective exam questions Application activity Final thoughts

20. Purpose Objective: –To compose critical thinking questions for program or classroom level exams WYMIWYG Aligning outcomes with methods

21. Learning Outcomes Program to classroom level (alignment) General statements to specific outcomes –Person centered –Clear, specific, and measurable Classroom learning outcomes should contain –Procedure/method (assignments that will be provided) –Behavior (what the students will do; usually the action verb) –Criteria (how well a learner performs to be judged adequate and the qualitative description of the criteria)

22. PhD Program Example Program Educational Objective : –Sixty percent of graduates from the College of Education will attain positions as college faculty. ( Program) Learning Outcome 1: –“Will prepare students as future researchers” –Graduating students will demonstrate fundamental knowledge of quantitative, qualitative, and mixed research designs.” Procedure 1.1 (Classroom learning outcome): –“Course EDP 382 will focus on the fundamental knowledge necessary to conduct quantitative research.” –“Given an exam administered in EDP 382, at least 80% of students will demonstrate a fundamental knowledge of quantitative research designs by achieving a score at or above an 80 out of 100. The exam will test the extent to which students can compute common statistics, identify assumptions of research designs, evaluate….

23. Writing Critical Thinking Exam Questions For critical thinking: –Focus on higher levels of Bloom’s taxonomy

24. Bloom’s taxonomy for the cognitive domain Remember –recall or memorize information Understand –translate from one form to another (e.g., restate in own words) Apply –apply knowledge to a novel situation Analyze –examine a concept and break it down into parts Evaluate –make judgments using standards of appraisal Create – put information together in an unique or novel way

25. Writing Critical Thinking Exam Questions For critical thinking: –Focus on higher levels of Bloom’s taxonomy Behavior component of learning outcome –Ex: Given an exam administered in in EDP 382, at least 80% of students will demonstrate a fundamental knowledge… –Action verbs (handout)

26. Writing Critical Thinking Exam Questions For critical thinking: –Focus on higher levels of Bloom’s taxonomy Behavior component of learning outcome –Ex: Given an exam administered in __ course, at least 80% of students will demonstrate a fundamental knowledge… –Action verbs (handout) Outcome to exam questions (and instruction) –Can use the action verb »Ex: Compute the standard deviation of…. –Test blueprint (handout)

27. Objective and Subjective Exam Questions Objective questions –Select a correct response from several alternatives Subjective questions –Present an original answer Strengths/weaknesses of each Assumptions (True/ False/ or Undetermined) 1.Essay exams require more thorough student preparation and study time than objective exams. 2.Essay and objective exams can be used to measure the same content or ability. Examples of objective, critical thinking exam questions (packet)

28. Application Activity In groups or pairs and with using the Emerson content from Dawn’s previous presentation, 1.Construct one objective and one subjective exam question. 2.Explain why you believe your questions are assessing critical thinking.

29. Application Activity Objective examples: (Analysis) Emerson might be characterized as any of the following EXCEPT: (A) a transcendentalist (B) an American Renaissance writer (C) a public speaker (D) a political prophet (E) a literary critic (Evaluate) Determine whether the following statement is true or false: –Melville and Hawthorne were both disillusioned transcendentalists.

30. Final thoughts At the program/course level, what do you want your students to learn/develop (outcomes)? Are your exams (methods) matching these objectives? Use multiple assessment methods –Ex: Lower stakes testing Can use rubrics for subjective exam questions

31. Agenda 1.Overview of SACS 2.Mapping Assessments to Program Outcomes 3.Writing Critical Thinking Exam Questions 4.Creating Rubrics

32. 1.What is a rubric? 2.When should you use a rubric? 3.What are the elements of a rubric? 4.How do you create a rubric? Creating Rubrics

33. What is a rubric? -A scoring guideline that measures student achievement systematically: –specifies instructor expectations –lists performance criteria –describes levels of quality

34. When should you use a rubric? On assignments that require subjective grading –Places structure on a subjective process Essays Presentations Multi-media

35. What are the elements of a rubric? What the student needs to do Descriptors and criteria that identify performance expectation for each point on the scale I. Example: Include a compelling, descriptive, and persuasive conclusion that summarizes arguments set forth in paper

36. Standards that identify the range of quality or performance levels How well the student does it II. Example: -- no or poor conclusion or summary of argument -- some summary of points made, but nothing beyond summary; no broad conclusions/lessons -- a conclusion going beyond summary of what was written in the body of the essay

37. A scale of points awarded to specific elements of student performance III. What score the student will receive Example: 5 = outstanding 3= acceptable 1 = poor

38. Critical Thinking Rubric A.Clearly and precisely analyzes key information, questions, and problems B.Uses inference to reason carefully from clearly stated premises to important implications and consequences C.Uses deductive and inductive reasoning and problem-solving skills consistently and with ease A.Is unable to analyze information, questions, and problems or does so superficially B.Is unable to or infrequently uses inference to reason from clearly stated premises or recognize implications and consequences C.Is unable to or infrequently uses deductive and inductive reasoning and problem-solving skills 1 [2 3 4] 5

39. Analytic Rubrics Holistic Rubrics

40. How do you create a rubric? Develop assignment Identify learning objectives Decide which criteria to evaluate Decide what kind of rubric to use Define performance indicators Identify standards of performance Decide on grading or scoring scale Review rubric to ensure it measures what you intend

41. Determine key criteria If the assignment objectives are … … then consider these criteria Creative products“Wow” factor, novelty Technical quality Adherence to conventions of the field WritingGrammar/mechanics Development of ideas Clarity Oral presentationVoice projection Body language Organization

42. Identify Indicators Concrete sign or evidence of a criterion being met—not meant to be exhaustive Example: Learning objective: Students will be able to write in an engaging and persuasive manner Criterion: Student clearly and precisely analyzes key information, questions, and problems Indicator: Identifies main tension in story and relates it to sub-plots

44. Any Questions?