1. CAS EXAMINATION PROCESS 2006 CAS Annual Meeting – Session C4 San Francisco CAS Exam Committee presents: Steve Armstrong, FCAS Nasser Hadidi, FCAS Derek Jones, FCAS Daniel Roth, FCAS

2. 2 Audit Findings (2001) The CAS Does Many Things Well: Good Communication with CandidatesGood Communication with Candidates Sound Procedures for Maintaining Confidential InformationSound Procedures for Maintaining Confidential Information Exams are Administered with Appropriate Controls and Standardized ProceduresExams are Administered with Appropriate Controls and Standardized Procedures

3. 3 Audit Findings Several Areas for Improvement: Need Better Link Between Learning Objectives and Exams/ReadingsNeed Better Link Between Learning Objectives and Exams/Readings Learning Objectives and Exam Blueprints Should be PublishedLearning Objectives and Exam Blueprints Should be Published Need Better Training of Item WritersNeed Better Training of Item Writers Need to Consider Alternative Processes for Selecting Pass MarksNeed to Consider Alternative Processes for Selecting Pass Marks

4. 4 Major Milestones 2001 – Chauncey Began Facilitating Meetings to Write Learning Objectives 2001 – Chauncey Began Facilitating Meetings to Write Learning Objectives 2001 – Pass Mark Panels Pilot 2001 – Pass Mark Panels Pilot 2002 – Item Writer Training Pilot 2002 – Item Writer Training Pilot 2003 – Executive Council Agrees to Fund Item Writer Training and Pass Mark Panels as Ongoing Processes 2003 – Executive Council Agrees to Fund Item Writer Training and Pass Mark Panels as Ongoing Processes 2003 – Executive Council Approves New Learning Objectives 2003 – Executive Council Approves New Learning Objectives

5. 5 Recent Improvements Evolution of CBT Evolution of CBT Improvement to Pass Mark Panel Process Improvement to Pass Mark Panel Process Expanded Sample Answer Sets Expanded Sample Answer Sets Increased Communication Increased Communication CAS Board White Paper CAS Board White Paper

6. 6 Learning Objectives The way things were –What topics should successful candidates understand –What readings should they know? The way things are now –What should successful candidates be able to DO?

7. 7 Learning Objectives The way things were –Individual topics and readings were the basis for assigning the writing of exam questions The way things are now –Learning Objectives are the basis for assigning the writing of exam questions

8. 8 Learning Objectives The CAS Syllabus Committee has developed Learning Objective Documents for CAS Exams 3, 5, 6, 7-US, 7-Canada, 8 and 9 There are more general Learning Objectives for the joint Exams 1, 2, & 4

9. 9 Learning Objective Documents Five Elements Overview Statement for a Group of Learning ObjectivesOverview Statement for a Group of Learning Objectives Learning ObjectivesLearning Objectives Knowledge StatementsKnowledge Statements Syllabus ReadingsSyllabus Readings WeightsWeights

10. 10 Learning Objective Documents Overview Statements Certain Syllabus sections can have multiple Learning Objectives (e.g., ratemaking)Certain Syllabus sections can have multiple Learning Objectives (e.g., ratemaking)

11. 11 Learning Objective Documents Learning Objectives What successful candidates should be able to doWhat successful candidates should be able to do Learning Objectives should:Learning Objectives should:  Clearly state a main intent  Reflect a measurable outcome  Support an attainable behavior  Relate to the learner’s needs or job function

12. 12 Learning Objective Documents Knowledge Statements Support Learning ObjectivesSupport Learning Objectives In order to accomplish the objective, what does the candidate need to know?In order to accomplish the objective, what does the candidate need to know?

13. 13 Learning Objective Documents Readings An individual reading may be listed under more than one learning objectiveAn individual reading may be listed under more than one learning objective Readings listed under multiple objectives may facilitate exam questions that involve more synthesisReadings listed under multiple objectives may facilitate exam questions that involve more synthesis

14. 14 Learning Objective Documents Weights (by Learning Objective) Shown as rangesShown as ranges Guidelines - not intended to be absoluteGuidelines - not intended to be absolute Should end practice of candidates calculating de facto weights by reading or topic from past examsShould end practice of candidates calculating de facto weights by reading or topic from past exams

15. 15 Learning Objectives and the Syllabus Learning Objective documents provide high level guidance Learning Objective documents provide high level guidance –Review of current syllabus material –Identification of topics requiring new syllabus material Weights help Syllabus and Exam Committees target specific objectives Weights help Syllabus and Exam Committees target specific objectives

16. 16 Future Changes to Learning Objective Documents These are living documentsThese are living documents –Never perfect –Subject to change Updates – When and how often?Updates – When and how often? –Not yet determined –Annually per exam seems reasonable –At least disruptive time for candidates

17. 17 Future Changes to Learning Objective Documents CAS Executive Council (VP-Admissions) will perform oversight and final approval of any changes CAS Executive Council (VP-Admissions) will perform oversight and final approval of any changes –Just as it does with changes to the Syllabus –Just as it has with the current Learning Objective Documents

18. 18 Learning Objective Summary Transition to published Learning Objectives should help the CAS achieve: Transition to published Learning Objectives should help the CAS achieve: –Better syllabus content and exam questions –More transparent basic education process –Better model for evaluating future changes to the syllabus –Better model for evaluating future changes to the desired education of casualty actuaries

19. 19 Writing Exam Questions A. One-Best-Answer Questions B. Essay Questions

20. 20 For Each Type: What makes a good question? The questions  Should measure students’ mastery of stated Learning Objectives, and not merely recalling factual information  Should be clearly stated and not subject to different interpretations  Should be perceived by students to be fair  Should be possible to answer by a typical student in the allotted time

21. 21 A: One-Best-Answer Questions Questions Should focus on an important concept Should focus on an important concept Should assess skills in practical applications Should assess skills in practical applications Should avoid flaws that provide special benefits to “test wise” students Should avoid flaws that provide special benefits to “test wise” students Should not start with “Which of the following statements is correct” – heterogeneous options Should not start with “Which of the following statements is correct” – heterogeneous options

22. 22 Distracters-Incorrect Answers Distracters directly affect the difficulty of a question Distracters directly affect the difficulty of a question Should be plausible – selected by some students Should be plausible – selected by some students Common misconceptions and faulty reasoning provide good source of plausible distracters. Common misconceptions and faulty reasoning provide good source of plausible distracters. None should stand out as being obviously incorrect None should stand out as being obviously incorrect

23. 23 Example – Bad Distracters Who was the primary author of the Declaration of Independence? A. Abraham Lincoln B. Thomas Jefferson C. Franklin Roosevelt D. King George II E. Catherine the Great

24. 24 Distracters Should be homogeneous in content Should be homogeneous in content Plausible and attractive to the unprepared student Plausible and attractive to the unprepared student Should not be totally wrong Should not be totally wrong Similar to the correct answer in construction and length Similar to the correct answer in construction and length Grammatically consistent and logically compatible with the correct answer Grammatically consistent and logically compatible with the correct answer “None of the above” usually not a good distracter “None of the above” usually not a good distracter

25. 25 One-Best-Answer Questions Format All relevant information should be given in the question All relevant information should be given in the question No additional information in the options No additional information in the options Options should be relatively short Options should be relatively short

26. 26 One-Best-Answer Questions Summary Item Flaws “Test Wise” issues: grammatical, logical, absolute terms, long correct answer, word repeats “Test Wise” issues: grammatical, logical, absolute terms, long correct answer, word repeats Irrelevant Difficulty issues: inconsistent, long/complicated, vague, “None of the above”, language not parallel Irrelevant Difficulty issues: inconsistent, long/complicated, vague, “None of the above”, language not parallel

27. 27 One-Best-Answer Questions Construction Guidelines Item can be answered without looking at options Item can be answered without looking at options Include as much information as possible – but avoid extra words - Options short Include as much information as possible – but avoid extra words - Options short Avoid superfluous information Avoid superfluous information Avoid “tricky” and overly complex items Avoid “tricky” and overly complex items Options grammatically consistent Options grammatically consistent Avoid using absolute terms – always, never Avoid using absolute terms – always, never Focus on important items – Do not waste time testing trivial facts Focus on important items – Do not waste time testing trivial facts

28. 28 B: Essay Questions Basically two types of Essay Questions: Extended Response - Students free to determine content, format of response Extended Response - Students free to determine content, format of response Restricted Response – Limit both content and form, this is the appropriate form when content is to be tested Restricted Response – Limit both content and form, this is the appropriate form when content is to be tested

29. 29 Constructing Essay Questions Allow adequate preparation time – Effective items need deliberation, reflection Allow adequate preparation time – Effective items need deliberation, reflection Limit to Learning Objectives that can not be measured by One-Best-Answer items Limit to Learning Objectives that can not be measured by One-Best-Answer items Design to test only one or just a few Learning Objectives per item Design to test only one or just a few Learning Objectives per item Select items that can be answered briefly-Several short essay items better than one long answered Select items that can be answered briefly-Several short essay items better than one long answered Clearly indicate task – both content and process Clearly indicate task – both content and process Explicitly state the time or number of points Explicitly state the time or number of points

30. 30 Essay Questions -Avoid Open ended broad type of items Open ended broad type of items Vague poorly worded items subject to multiple interpretations Vague poorly worded items subject to multiple interpretations Items asking simply for “Opinion” – rather ask for evidence in support of a “position” Items asking simply for “Opinion” – rather ask for evidence in support of a “position” Items where different students can give answers to different questions. Items where different students can give answers to different questions.

31. 31 The Time Factor  In both type of questions and especially essay questions, consider the allotted time  Time should not be a major factor  The rule of thumb is that if more than 10% of the students fail to complete the test in the allotted time, Time has become a factor in test scores

32. 32 Checklist for Effective Test Items Item matches Leaning Objective? Item matches Leaning Objective? Item relates to what was actually covered in the Syllabus? Item relates to what was actually covered in the Syllabus? Item relates to important concept rather than trivia? Item relates to important concept rather than trivia? Item measures ability to make practical application, rather than simply measuring recall? Item measures ability to make practical application, rather than simply measuring recall? Item free from vaguely defined terms, ambiguous wording, irrelevant information, unintentional clues? Item free from vaguely defined terms, ambiguous wording, irrelevant information, unintentional clues?

33. 33 References B.B. Zimmerman et. al., How to prepare better tests: Guidelines for University Faculty B.B. Zimmerman et. al., How to prepare better tests: Guidelines for University Faculty National Board of Medical Examiners, Constructing Written Test Questions for Basic and Clinical Sciences National Board of Medical Examiners, Constructing Written Test Questions for Basic and Clinical Sciences W. Cashin, Improving Essay tests W. Cashin, Improving Essay tests

34. 34 Grading Exam Questions Assignments for grading will be distributed immediately after the exam as to which question(s) you are assigned to grade. Questions are graded in pairs, just like writing exam questions. A sample solution and a copy of the exam will be sent to the grading pair for review prior to getting the actual papers. The actual papers come to the grading pair in the week or two following the examination.

35. 35 Grading Exam Questions Graders are encouraged to develop a grading key that accounts for the different combinations and permutations of answers that can be provided. Graders are encouraged to use this grading key for a random set of questions (between 20-40) to ensure consistency in application of the grading key. The graders should meet after grading this set of random questions to reconcile any differences and change the grading key if necessary. CONSISTENCY AND ACCURACY IS VITAL

36. 36 Grading Exam Questions In the subsequent weeks, individual graders will grade every candidate’s response and log the points in a Grading Program. Graders are encouraged to reconcile scores along the way so that this work does not monopolize the time spent at the on-site Grading Session. The on-site Grading Session is now being conducted in Las Vegas and lasts for two days in June and December.

37. 37 Grading Exam Questions On-Site Grading Session The on-site Grading Session allows the following: Allows those on the committee to meet with one another. Small allowance of time to reconcile all scores between graders to no more than a ¼ point difference (differs by Part Chair). Establishment by the committee of an appropriate passing score. Re-grading (no point discrepancy at all) of those candidate’s questions that are within a range of the passing score. Discussion on how to make the exam even better for next year.

38. 38 Grading Exam Questions On-Site Grading Session After the first day, a group activity occurs at night to allow those on the committee to interact with one another in a less formal manner. The on-site Grading Session is concluded when the group has established a mutually agreed upon passing score by inspecting all the relevant statistics and grading/re- grading those candidates around the passing score to ensure that their score is correct. All final scores by candidate, including grading keys and model candidate solutions are left with the Part Chair to create the exam report to be sent to the Exam Officers.

39. 39 Setting the Pass Mark Identify Purpose of the Pass Mark Identify Purpose of the Pass Mark Convene Pass Mark Panel Convene Pass Mark Panel Analyze Exam Statistics Analyze Exam Statistics Prepare Recommendation Prepare Recommendation Proceed through Approval Process Proceed through Approval Process

40. 40 Purpose of the Pass Mark Pass Minimally Qualified (or better) Candidates Pass Minimally Qualified (or better) Candidates –Those who have demonstrated a sufficient grasp of the syllabus material Fail Others Fail Others There is no predetermined pass ratio There is no predetermined pass ratio

41. 41 Purpose of the Pass Mark Failers Passers Minimally Qualified Candidate

42. 42 Pass Mark Panel Panel includes: Panel includes:  New Fellows (1-3 years)  Fellows experienced in practice area  Officers of exam committee Recommends a pass mark independent of the normal exam committee procedures Recommends a pass mark independent of the normal exam committee procedures

43. 43 Pass Mark Panel Defines Minimally Qualified Candidate Defines Minimally Qualified Candidate  What he or she should will know  What he or she will not know  What he or she will be able to demonstrate on the exam Relates Criteria to Learning Objectives for defining the minimally qualified candidate. Relates Criteria to Learning Objectives for defining the minimally qualified candidate.

44. 44 Pass Mark Panel Each panelist independently estimates how 100 minimally qualified candidates will score on each question (and sub-part of each question). Each panelist independently estimates how 100 minimally qualified candidates will score on each question (and sub-part of each question). Scores are assembled and shared in a group format. Scores are assembled and shared in a group format. Group discusses ratings and may change estimates Group discusses ratings and may change estimates Facilitator compiles ratings and shares results with exam committee officers Facilitator compiles ratings and shares results with exam committee officers

45. 45 Analyze Exam Statistics (back at the Grading Session) Collect Initial Scores for All Candidates Collect Initial Scores for All Candidates Review/Discuss Key Measures Review/Discuss Key Measures  High, Low, Mean  Percentiles, Percentile Relationships  Pass Mark Panel Recommendation  Prior statistics from previous exams  CAS Board goal, “…that 40% or more of the candidates should get a score of 70% or more on any given exam; and all candidates that get such a score should pass.” Pick an initial pass mark and re-grade candidates within certain range of pass mark (+/- 3 points, for example) Pick an initial pass mark and re-grade candidates within certain range of pass mark (+/- 3 points, for example)

46. 46 Prepare Recommendation Recollect scores if any have changed and review all relevant statistics again. Recollect scores if any have changed and review all relevant statistics again. Review borderline candidates (tighter threshold now – no discrepancy between graders) and re- grade/review for consistency with grading key. Review borderline candidates (tighter threshold now – no discrepancy between graders) and re- grade/review for consistency with grading key. Repeat process until only looking at the 5 exams above and the 5 exams below the recommended pass mark. Repeat process until only looking at the 5 exams above and the 5 exams below the recommended pass mark. Justify Recommended Pass Score Justify Recommended Pass Score

47. 47 Approval Process Part Chair Part Chair General Officer (Spring / Fall) General Officer (Spring / Fall) Exam Committee Chair (Arlie Proctor) Exam Committee Chair (Arlie Proctor) VP-Admissions (Jim Christie) – The final decision on the pass mark is the responsibility of this position. VP-Admissions (Jim Christie) – The final decision on the pass mark is the responsibility of this position.

48. 48 Appeal Process In the event of a candidate appeal, a grader may be called upon to review the appeal and reconcile the score with the grading key. In the event of a candidate appeal, a grader may be called upon to review the appeal and reconcile the score with the grading key.

49. Questions and Comments More info on the Web – www.casact.org/admissions/process/