Presented by Denise Sibley Laura Jean Kerr Mississippi Assessment Center Research and Curriculum Unit

 Requirement established by Perkins IV  Assessments have been developed  Standard setting needs to be established

The proper following of a prescribed, rational system of rules and procedures that result in determining:  cut scores and performance levels  what’s tested  proper statistical analyses

 Ensures that decisions are based on high- quality data  Ensures that data are combined in a systematic, reproducible, objective, and defensible manner  Increases understanding and trust on the part of the stakeholders

StepDescription 1Choose standard-setting method 2Prepare descriptions of the performance categories 3Select subject matter experts (SMEs) from each program area to conduct standard-setting 4Prepare materials for standard-setting meeting 5Train SMEs to use standard-setting method 6Conduct meeting, compile results, and provide feedback 7Conduct an evaluation of the standard-setting process 8Document the standard-setting process and evidence on the validity of the resulting performance standards

 Researched standard setting methods  Limited literature for CTE assessments – lots of academic literature  Method used had to present a strong validity argument for the results  Method used had to be suitable for the CTE content  Method used had to be straight-forward

 Performance level descriptors (PLDs) need to developed  Descriptions of the performance levels into which examinees will classified (e.g., Minimal, Basic, Proficient, or Advanced )  Participants rate items according to PLDS  Items with agreement less than.67 are re-rated  Use results to distinguish between the four performance level categories.

 PLDs should be specific to each standard in the curriculum  Challenging because there were over 100 curricula  Developed Career and Technical Education PLDs

 Advanced : An advanced student evaluates and integrates transferable academic/workplace knowledge and skills in multiple situations as a productive contributor in the workplace.  Proficient : A proficient student acquires and applies academic/workplace knowledge and skills to become a productive contributor in the workplace.  Basic : A basic student has limited acquisition and comprehension of academic/ workplace knowledge and skills that are necessary to become a productive contributor in the workplace.  Minimal : A minimal student has not developed academic/workplace knowledge and skills that are necessary to become a productive contributor in the workplace.

Included program descriptions and standards from the curricula as a reference for participants to ensure they aligned their ratings with the curriculum

 Used stratified random sample method for selecting participants–geographic location based on 15 community colleges  Randomly selected teachers from each geographic location by program area  Had to be able to attend standard setting meeting  Approximately 400 subject matter experts participated

 PLDs  Standards from each curriculum framework  Program description from each curriculum framework  Test item booklet with only field-tested items  Item difficulty is used to determine the range for each performance level  Sign-in sheets with logins and passwords  Bubble sheets for backup if site networks went down

 Facilitators were trained prior to the meetings to insure that meetings were conducted in a consistent manner  Meetings were held across the state over the course of 3 days  A Web interface was developed for participants to enter their responses  Scheduled participants from each program area to meet on the same day  Participants received instructions on using the materials and the Web interface  Participants went through test item booklets and matched items to PLDs and also were able to write comments about the items  When all participants submitted their responses, they were calculated and any items that didn’t have a.67 agreement were re-rated.  Participants completed a survey after they completed the meeting

 Conduct Standard Setting over long distances  Coordinate Standard Setting for the best use of Subject Matter Experts time  Keep up with ever-changing needs of Career and Technical Education  Organize and compile data

 Consistency is the justification for subjective limitations  Training of Subject Matter Experts  Standard Setting environment  Policies and procedures  Consistency contributes to quality data

 Communication  “The PowerPoint would have been helpful if I had seen it before today.”  “I did not fully understand the descriptors. I probably over estimated the students' ability.”  “The performance level indicators are open to many interpretations.”  Lesson learned:  Communicate with SMEs early and often.

 Meeting Environment  “It was the room and atmosphere where the training was held. It was in a room off the cafeteria. Which means...guess what...too much traffic. We could hear the phone conversations next door from an office, when the phone was ringing and etc.”  “Could use a break for the 200 questions and refreshments.”  “People who arrive late should not be able to enter.”  Lesson Learned:  Every aspect (even the smallest detail) of the testing environment should be taken into consideration.

 Technology  “I would like to see the questions online with the performance descriptors. I believe the process would go a lot faster and would allow for easier comments.”  “Website does not allow you to go back and alter responses on previous pages in event of mistakes.”  “Love doing this on computer!”  Lesson learned:  Incorporated these comments into development of new interface

 Trust and Understanding  “I appreciate being able to give feedback on the questions. This is the first time I have been invited to do so.”  “I think that this is a step in the right direction for standardized testing.”  “Good program...would like to know more about how the process works with our peers, etc.”  Keeping stakeholders in the “loop” fosters trust

 Compile data  Provide feedback to stakeholders  Document process  Improve process  Publish results

Denise Sibley Interim Coordinator of Assessment and Accountability Laura Jean Kerr Assessment Specialist