CAS EXAMINATION PROCESS Julie Stenberg, FCAS CANE Meeting March 20, 2007

2 CAS Admissions Process Audit Fall 2000 – CAS Issues RFP for External Review of Admissions Processes The Chauncey Group (Subsidiary of ETS) Selected Spring 2001 – Chauncey Group Conducts Audit of CAS Admissions Processes

3 Audit Findings The CAS Does Many Things Well: Good Communication with Candidates Sound Procedures for Maintaining Confidential Information Exams are Administered with Appropriate Controls and Standardized Procedures

4 Audit Findings Several Areas for Improvement: Need Better Link Between Learning Objectives and Exams/Readings Learning Objectives and Exam Blueprints Should be Published Need Better Training of Item Writers Need to Consider Alternative Processes for Selecting Pass Marks

5 Major Objectives The Chauncey Group Engaged to Help CAS with Three Issues: Write Better Learning Objectives and Establish Links to Readings/Exams Develop a Process for Training Item Writers Pilot an Alternative Process for Selecting Pass Marks

6 Major Milestones August 2001 – Chauncey Began Facilitating Meetings to Write Learning Objectives October 2001 – Piloted Pass Mark Panel Process for Exams 6 & 9 March 2002 – Piloted Item Writer Training Classes for Exams 6 & 9 April 2002 – Pass Mark Panels Meet for Exams 5, 7 & 8 June 2002 – Item Writer Training for Exams 5, 7 & 8

7 Major Milestones October 2002 – Pass Mark Panels Meet for Exams 6 & 9 February 2003 – Executive Council Agrees to Fund Item Writer Training and Pass Mark Panels as Ongoing Processes April 2003 – Executive Council Approves New Learning Objectives for Exams 3, 5-9

8 Recent Improvements Evolution of CBT Improvement to Pass Mark Panel Process Expanded Sample Answer Sets Increased Communication CAS Board White Paper

9 Learning Objectives The way things were –What topics should successful candidates understand –What readings should they know? The way things are now –What should successful candidates be able to DO?

10 Learning Objectives The way things were –Individual topics and readings were the basis for assigning the writing of exam questions The way things are now –Learning Objectives are the basis for assigning the writing of exam questions

11 Learning Objectives The way things were –Syllabus “blueprints” were the documents governing the review of the Syllabus and the construction of Exams The way things are now –Learning Objective Documents are the basis for the review of the Syllabus and the construction of Exams

12 Learning Objectives The Syllabus Committee has developed Learning Objective Documents for CAS Exams 3, 5, 6, 7-US, 7-Canada, 8 and 9

13 Learning Objective Documents Five Elements Overview Statement for a Group of Learning Objectives Learning Objectives Knowledge Statements Syllabus Readings Weights

14 Learning Objective Documents Overview Statements Certain Syllabus Sections Can Have Multiple Learning Objectives (e.g., Ratemaking)

15 Learning Objective Documents Learning Objectives What successful candidates should be able to do Learning Objectives Should:  Clearly state a main intent  Reflect a measurable outcome  Support an attainable behavior  Relate to the learner’s needs or job function  Have a definitive time frame

16 Learning Objective Documents Knowledge Statements Support Learning Objectives In order to accomplish the objective, what does the candidate need to know?

17 Learning Objective Documents Readings An individual reading may be listed under more than one learning objective Readings listed under multiple objectives may facilitate more synthesis/reasoning/cross-topic Exam questions

18 Learning Objective Documents Weights (by Learning Objective) Shown as ranges The ranges are guidelines and are not intended to be absolute Ended practice of candidates calculating de facto weights by reading or topic from past Exams

19

20 Learning Objectives and the Syllabus Learning Objective Documents Provide High Level Guidance –Review of Current Syllabus Material –Identification of Topics Requiring New Syllabus Material Weights help Syllabus Committee Target Specific Objectives

21 Future Changes to Learning Objective Documents These are Living Documents –Never Perfect –Subject to Change Updates – When and How Often? –Once a Year Per Exam Seems Reasonable –At Least Disruptive Time for Candidates

22 Future Changes to Learning Objective Documents CAS Executive Council (VP-Admissions) Performs Oversight and Final Approval of Any Changes –Just as it does with changes to the Syllabus –Just as it has with the current Learning Objective Documents

23 Learning Objective Summary Transition to Published Learning Objectives Should Help the CAS Achieve: –Better Syllabus Content and Exam Questions –More Transparent Basic Education Process –Better Model for Evaluating Future Changes to the Syllabus –Better Model for Evaluating Future Changes to the Desired Education of Casualty Actuaries

24 Writing Exam Questions Question 1 – According to Miller, “Writing Exam Questions”, which of the following is true? I.Writing exam questions is the same now as it was 6 years ago II.Writing exam questions is easier now than it was 6 years ago III.Writing exam questions is harder now than it was 6 years ago A.II only B.I and II only C.I and III only D.II and III only E.I, II and III

25 Writing Exam Questions I.Writing exam questions is the same now as it was 6 years ago True – Question writers have always wanted to write good, fair, high quality questions. 1.Still takes the same time commitment 2.Still requires studying assigned readings 3.Still involves choosing the areas you want to test 4.Still involves peer review by others

26 Writing Exam Questions II. Writing exam questions is easier now than it was 6 years ago True – We have more tools to work with 1.We have identified objectives 2.They identify the readings tied to those objectives 3.Question writing skills are taught – what to do and what to avoid 4.We have a common language with which to make constructive criticisms

27 Writing Exam Questions III. Writing exam questions is harder now than it was 6 years ago True – We have been conditioned by years of studying old questions 1.Triple True-False are often easier to write than short answers for Multiple Choice 2.The easiest questions to write may not always fit the objectives 3.The bar is higher and we don’t like to fail

28 Writing Exam Questions Question 1 – Solution: E A.Some candidates will think nothing has changed B.Some candidates will think the new process could not possibly make it harder to write questions C.Some candidates always guess C when they don’t know the answer D.I used to guess D E.Correct answer

29 What Have We Learned From The Chauncey Initiatives? Questions should be focused on learning objectives, rather than individual papers Triple True/False is not the only kind of multiple choice question Art of selecting good “wrong” multiple choice answers

30 What Changes Should The Candidates See On The Exams? Better questions Questions with many possible full-credit answers Less “according to” and “based on” questions Heavy “list” papers have become open- book

31 Setting the Pass Mark Identify Purpose of the Pass Mark Convene Pass Mark Panel Analyze Exam Statistics Prepare Recommendation Proceed through Approval Process

32 Purpose of the Pass Mark Pass Minimally Qualified (or better) Candidates –Those who have demonstrated a sufficient grasp of the syllabus material Fail Others There is no predetermined pass ratio

33 Purpose of the Pass Mark Failers Passers Minimally Qualified Candidate

34 Pass Mark Panel Panel includes:  New Fellows (1-3 years)  Fellows experienced in practice area  Officers of exam committee Recommends a pass mark independent of the normal exam committee procedures

35 Pass Mark Panel Defines Minimally Qualified Candidate  What he or she should will know  What he or she will not know  What he or she will be able to demonstrate on the exam Relates Criteria to Learning Objectives for defining the minimally qualified candidate.

36 Pass Mark Panel Each panelist independently estimates how 100 minimally qualified candidates will score on each question (and sub-part of each question). Scores are assembled and shared in a group format. Group discusses ratings and may change estimates Facilitator compiles ratings and shares results with exam committee officers

37 Analyze Exam Statistics (back at the Grading Session) Collect Initial Scores for All Candidates Review/Discuss Key Measures  High, Low, Mean  Percentiles, Percentile Relationships  Pass Mark Panel Recommendation  Prior statistics from previous exams  CAS Board goal, “…that 40% or more of the candidates should get a score of 70% or more on any given exam; and all candidates that get such a score should pass.” Pick an initial pass mark and re-grade candidates within certain range of pass mark (+/- 3 points, for example)

38 Prepare Recommendation Recollect scores if any have changed and review all relevant statistics again. Repeat process until only looking at the 5 exams above and the 5 exams below the recommended pass mark. Justify Recommended Pass Score

39 Approval Process Part Chair General Officer (Spring / Fall) Exam Committee Chair (Arlie Proctor) VP-Admissions (Jim Christie) – The final decision on the pass mark is the responsibility of this position.

40 Appeal Process In the event of a candidate appeal, a grader may be called upon to review the appeal and reconcile the score with the grading key.

41 Join the Exam Committee Fill out the annual CAS participation survey or Contact the exam committee recruiter directly Rhonda Walker

42 Questions and Comments Contact Julie Stenberg at Or Arlie Proctor at