1. CAS EXAMINATION PROCESS 2005 CAS Annual Meeting – Session C8 Baltimore CAS Exam Committee presents: Steve Armstrong, FCAS Daniel Roth, FCAS Manalur Sandilya, FCAS Tom Struppeck, FCAS

2. 2 CAS Examination Process Update Fall 2000 – CAS Issues RFP for External Review of Admissions Processes Fall 2000 – CAS Issues RFP for External Review of Admissions Processes The Chauncey Group (Subsidiary of ETS) Selected The Chauncey Group (Subsidiary of ETS) Selected Spring 2001 – Chauncey Group Conducts Audit of CAS Admissions Processes Spring 2001 – Chauncey Group Conducts Audit of CAS Admissions Processes

3. 3 Audit Findings The CAS Does Many Things Well: Good Communication with CandidatesGood Communication with Candidates Sound Procedures for Maintaining Confidential InformationSound Procedures for Maintaining Confidential Information Exams are Administered with Appropriate Controls and Standardized ProceduresExams are Administered with Appropriate Controls and Standardized Procedures

4. 4 Audit Findings Several Areas for Improvement: Need Better Link Between Learning Objectives and Exams/ReadingsNeed Better Link Between Learning Objectives and Exams/Readings Learning Objectives and Exam Blueprints Should be PublishedLearning Objectives and Exam Blueprints Should be Published Need Better Training of Item WritersNeed Better Training of Item Writers Need to Consider Alternative Processes for Selecting Pass MarksNeed to Consider Alternative Processes for Selecting Pass Marks

5. 5 Major Objectives The Chauncey Group Engaged to Help CAS With Three Issues: Write Better Learning Objectives and Establish Links to Readings/ExamsWrite Better Learning Objectives and Establish Links to Readings/Exams Develop a Process for Training Item WritersDevelop a Process for Training Item Writers Pilot an Alternative Process for Selecting Pass MarksPilot an Alternative Process for Selecting Pass Marks

6. 6 Major Milestones 2001 – Chauncey Began Facilitating Meetings to Write Learning Objectives 2001 – Chauncey Began Facilitating Meetings to Write Learning Objectives 2001 – Pass Mark Panels Pilot 2001 – Pass Mark Panels Pilot 2002 – Item Writer Training Pilot 2002 – Item Writer Training Pilot 2003 – Executive Council Agrees to Fund Item Writer Training and Pass Mark Panels as Ongoing Processes 2003 – Executive Council Agrees to Fund Item Writer Training and Pass Mark Panels as Ongoing Processes 2003 – Executive Council Approves New Learning Objectives 2003 – Executive Council Approves New Learning Objectives

7. 7 Learning Objectives The way things were –What topics should successful candidates understand –What readings should they know? The way things are now –What should successful candidates be able to DO?

8. 8 Learning Objectives The way things were –Individual topics and readings were the basis for assigning the writing of exam questions The way things are now –Learning Objectives are the basis for assigning the writing of exam questions

9. 9 Learning Objectives The Syllabus Committee has developed Learning Objective Documents for CAS Exams 3, 5, 6, 7-US, 7-Canada, 8 and 9 and also the VEE exams There are simpler and more direct Learning Objectives for the Joint Exams 1, 2, & 4

10. 10 Learning Objective Documents Five Elements Overview Statement for a Group of Learning ObjectivesOverview Statement for a Group of Learning Objectives Learning ObjectivesLearning Objectives Knowledge StatementsKnowledge Statements Syllabus ReadingsSyllabus Readings WeightsWeights

11. 11 Learning Objective Documents Overview Statements Certain Syllabus Sections Can Have Multiple Learning Objectives (e.g., Ratemaking)Certain Syllabus Sections Can Have Multiple Learning Objectives (e.g., Ratemaking)

12. 12 Learning Objective Documents Learning Objectives What successful candidates should be able to doWhat successful candidates should be able to do Learning Objectives Should:Learning Objectives Should:  Clearly state a main intent  Reflect a measurable outcome  Support an attainable behavior  Relate to the learner’s needs or job function  Have a definitive time frame

13. 13 Learning Objective Documents Knowledge Statements Support Learning ObjectivesSupport Learning Objectives In order to accomplish the objective, what does the learner need to know?In order to accomplish the objective, what does the learner need to know?

14. 14 Learning Objective Documents Readings An individual reading may be listed under more than one learning objectiveAn individual reading may be listed under more than one learning objective Readings listed under multiple objectives may facilitate more synthesis/reasoning/cross-topic Exam questionsReadings listed under multiple objectives may facilitate more synthesis/reasoning/cross-topic Exam questions

15. 15 Learning Objective Documents Weights (by Learning Objective) Will be shown as rangesWill be shown as ranges The ranges are guidelines and are not intended to be absoluteThe ranges are guidelines and are not intended to be absolute Advantages of old-style “blueprints” without disadvantagesAdvantages of old-style “blueprints” without disadvantages Will (perhaps) end practice of candidates calculating de facto weights by reading or topic from past ExamsWill (perhaps) end practice of candidates calculating de facto weights by reading or topic from past Exams

16. 16 Learning Objectives and the Syllabus Learning Objective Documents Provide High Level Guidance Learning Objective Documents Provide High Level Guidance –Review of Current Syllabus Material –Identification of Topics Requiring New Syllabus Material Weights help Syllabus Committee Target Specific Objectives Weights help Syllabus Committee Target Specific Objectives

17. 17 Future Changes to Learning Objective Documents These are Living Documents These are Living Documents –Never Perfect –Subject to Change Updates – When and How Often? Updates – When and How Often? –Not Yet Determined –Once a Year Per Exam Seems Reasonable –At Least Disruptive Time for Candidates

18. 18 Future Changes to Learning Objective Documents CAS Executive Council (VP-Admissions) Will Perform Oversight and Final Approval of Any Changes CAS Executive Council (VP-Admissions) Will Perform Oversight and Final Approval of Any Changes –Just as it does with changes to the Syllabus –Just as it has with the initial Learning Objective Documents

19. 19 Learning Objective Summary Transition to Published Learning Objectives Should Help the CAS Achieve: Transition to Published Learning Objectives Should Help the CAS Achieve: –Better Syllabus Content and Exam Questions –More Transparent Basic Education Process –Better Model for Evaluating Future Changes to the Syllabus –Better Model for Evaluating Future Changes to the Desired Education of Casualty Actuaries

20. 20 Integrated Syllabus Material Ratemaking for Catastrophes Ratemaking for Catastrophes Multiple issues Multiple issues Multiple papers Multiple papers Feedback from candidates Feedback from candidates Need for Integrated Study Material Need for Integrated Study Material RFP process RFP process

21. 21 Learning Objectives related to Ratemaking for Catastrophes Two Learning Objectives Two Learning Objectives Operational Issues Operational Issues Ratemaking Issues Ratemaking Issues 0 – 10 weight spread 0 – 10 weight spread Two or more readings; Candidate feedback Two or more readings; Candidate feedback Exam Committee feedback Exam Committee feedback Can we create an integrated study note Can we create an integrated study note

22. 22 Learning Objectives to Integrated Study Notes Outline begins with Learning Objectives Outline begins with Learning Objectives Expand the outline based on Knowledge Statements Expand the outline based on Knowledge Statements Add new Knowledge Statements where appropriate Add new Knowledge Statements where appropriate Review the flow of ideas Review the flow of ideas Finalize the Study Notes Finalize the Study Notes

23. 23 Writing Exam Questions What makes a good exam question? 1) Should be easy to grade. 2) Should be answerable in a reasonable amount of time. 3) Should measure the student’s mastery of the material, ideally by doing something. Not easy to achieve all of these.

24. 24 Question 1 According to the errata, the “4.2” on page 93, line 15 should read “7.9”. According to the errata, the “4.2” on page 93, line 15 should read “7.9”.

25. 25 Question 1 According to the errata, the “4.2” on page 93, line 15 should read “7.9”. According to the errata, the “4.2” on page 93, line 15 should read “7.9”. This satisfies two of the criteria.

26. 26 Question 2 Struppeck gives six example test questions, list them. Struppeck gives six example test questions, list them.

27. 27 Question 2 Struppeck gives six example test questions, list them. Struppeck gives six example test questions, list them. This is better than Question 1, but it still isn’t giving the student a chance to show mastery of the material, only recall.

28. 28 Question 3 Struppeck gives six example test questions, list them and describe them. Struppeck gives six example test questions, list them and describe them.

29. 29 Question 3 Struppeck gives six example test questions, list them and describe them. Struppeck gives six example test questions, list them and describe them. This is better than Question 2.

30. 30 Question 4 Struppeck claims that Question 3 is better than Question 2, explain why. Struppeck claims that Question 3 is better than Question 2, explain why.

31. 31 Question 4 Struppeck claims that Question 3 is better than Question 2, explain why. Struppeck claims that Question 3 is better than Question 2, explain why. This would actually be a pretty good question.

32. 32 Question 5 Write Question 7 and use Struppeck’s three criteria to evaluate it. Write Question 7 and use Struppeck’s three criteria to evaluate it.

33. 33 Question 5 Write Question 7 and use Struppeck’s three criteria to evaluate it. Write Question 7 and use Struppeck’s three criteria to evaluate it. This might be a bit too open-ended to be graded easily.

34. 34 Question 6 Use Struppeck’s three criteria to evaluate Question 6. Use Struppeck’s three criteria to evaluate Question 6.

35. 35 Question 6 Use Struppeck’s three criteria to evaluate Question 6. Use Struppeck’s three criteria to evaluate Question 6. 1) Easy to grade. OK

36. 36 Question 6 Use Struppeck’s three criteria to evaluate Question 6. Use Struppeck’s three criteria to evaluate Question 6. 1) Easy to grade. OK 2) Can be done quickly. OK

37. 37 Question 6 Use Struppeck’s three criteria to evaluate Question 6. Use Struppeck’s three criteria to evaluate Question 6. 1) Easy to grade. OK 2) Can be done quickly. OK 3) Illustrates that we can use the three criteria to do something. OK

38. 38 Grading Exam Questions Assignments for grading will be distributed immediately after the exam as to which question(s) you are assigned to grade. Questions are graded in pairs, just like writing exam questions. A sample solution and a copy of the exam will be sent to the grading pair for review prior to getting the actual papers. The actual papers come to the grading pair in the week or two following the examination.

39. 39 Grading Exam Questions Graders are encouraged to develop a grading key that accounts for the different combinations and permutations of answers that can be provided. Graders are encouraged to use this grading key for a random set of questions (between 20-40) to ensure consistency in application of the grading key. The graders should meet after grading this set of random questions to reconcile any differences and change the answer key if necessary.

40. 40 Grading Exam Questions In the subsequent weeks, individual graders will grade every candidate’s response and log the points in a Grading Program. Graders are encouraged to reconcile scores along the way so that this work does not monopolize the time spent at the on-site Grading Session. The on-site Grading Session is now being conducted in Las Vegas and lasts for two days.

41. 41 Grading Exam Questions On-Site Grading Session The on-site Grading Session allows the following: Allows those on the committee to meet with one another. Small allowance of time to reconcile all scores between graders to no more than a ¼ point difference (differs by Part Chair). Establishment by the committee of an appropriate passing score. Re-grading of those candidate’s questions that are within a range of the passing score. Discussion on how to make the exam even better for next year.

42. 42 Grading Exam Questions On-Site Grading Session After the first day, a group activity occurs at night to allow those on the committee to interact with one another in a less formal manner. The on-site Grading Session is concluded when the group has established a mutually agreed upon passing score by inspecting all the relevant statistics and grading/re-grading those candidates around the passing score to ensure that their score is correct. All final scores by candidate, including grading keys and model solutions are left with the Part Chair to create the report to be sent to the Exam Officers.

43. 43 What Have We Learned From The Chauncey Initiatives? Questions should be focused on learning objectives, rather than individual papers Questions should be focused on learning objectives, rather than individual papers Triple True/False is not the only kind of multiple choice question Triple True/False is not the only kind of multiple choice question Art of selecting good “wrong” multiple choice answers Art of selecting good “wrong” multiple choice answers

44. 44 What Changes Should The Candidates See On The Exams? Better questions Better questions Questions with many possible full-credit answers Questions with many possible full-credit answers Less “according to” and “based on” questions Less “according to” and “based on” questions Heavy “list” papers may become open-book Heavy “list” papers may become open-book

45. 45 Setting the Pass Mark Identify Purpose of the Pass Mark Identify Purpose of the Pass Mark Convene Pass Mark Panel Convene Pass Mark Panel Analyze Exam Statistics Analyze Exam Statistics Prepare Recommendation Prepare Recommendation Proceed through Approval Process Proceed through Approval Process

46. 46 Purpose of the Pass Mark Establish Objective Pass/Fail Criterion Establish Objective Pass/Fail Criterion Pass Minimally Qualified (or better) Candidates Pass Minimally Qualified (or better) Candidates Fail Others Fail Others Ensure Consistency between Sittings Ensure Consistency between Sittings

47. 47 Purpose of the Pass Mark Failures Passers Minimally Qualified Candidate

48. 48 Pass Mark Panel Panel includes: Panel includes:  New Fellows (1-3 years)  Fellows experienced in practice area  Officers of exam committee Recommends a pass mark independent of the normal exam committee procedures Recommends a pass mark independent of the normal exam committee procedures

49. 49 Pass Mark Panel Defines Minimally Qualified Candidate Defines Minimally Qualified Candidate  What he or she will should know  What he or she will not know  What he or she will be able to demonstrate on the exam Relates Criteria to Learning Objectives for defining the minimally qualified candidate. Relates Criteria to Learning Objectives for defining the minimally qualified candidate.

50. 50 Pass Mark Panel Each panelist independently estimates how 100 minimally qualified candidates will score on each question (and sub-part of each question). Each panelist independently estimates how 100 minimally qualified candidates will score on each question (and sub-part of each question). Scores are assembled and shared in a group format. Scores are assembled and shared in a group format. Group discusses ratings and may change estimates Group discusses ratings and may change estimates Facilitator compiles ratings and shares results with exam committee officers Facilitator compiles ratings and shares results with exam committee officers

51. 51 Analyze Exam Statistics (back at the Grading Session in June) Collect Initial Scores for All Candidates Collect Initial Scores for All Candidates Review/Discuss Key Measures Review/Discuss Key Measures  High, Low, Mean  Percentiles, Percentile Relationships  Pass Mark Panel Recommendation  Prior statistics from previous exams  CAS Board goal that 40% or more of the candidates should get a score of 70% or more on any given exam; and all candidates that get such a score should pass Pick an initial pass mark and re-grade candidates within certain range of pass mark (+/- 3 points, for example) Pick an initial pass mark and re-grade candidates within certain range of pass mark (+/- 3 points, for example)

52. 52 Prepare Recommendation Recollect scores if any have changed and review all relevant statistics again. Recollect scores if any have changed and review all relevant statistics again. Review borderline candidates (tighter threshold now) and re-grade/review for consistency with answer key. Review borderline candidates (tighter threshold now) and re-grade/review for consistency with answer key. Repeat process until only looking at the 5 exams above and the 5 exams below the recommended pass mark. Repeat process until only looking at the 5 exams above and the 5 exams below the recommended pass mark. Justify Recommended Pass Score Justify Recommended Pass Score

53. 53 Approval Process Part Chair Part Chair General Officer (Spring / Fall) General Officer (Spring / Fall) Exam Committee Chair (Arlie Proctor) Exam Committee Chair (Arlie Proctor) VP-Admissions (Jim Christie) VP-Admissions (Jim Christie)