McCann Associates Presented by: Michael Childs, Barbara Dyer, Ira Taylor, Bruce Nugyen Chad Warner & Joe Koury, McCann Associates

McCann Associates Introduction: The Florida Association of College Test Administrators (FACTA) is an organization, at the institutional level, comprised of testing professionals. The mission of FACTA is to enhance professional growth and communication among test administrators employed by regionally accredited postsecondary institutions in the State of Florida. FACTA actively advocates and supports the National College Testing Association's (NCTA) Professional Standards and Guidelines. To that end, the Testing Center Leadership Certification program was devised.

McCann Associates Purpose: The purpose of the Testing Center Leadership Certification is to provide a means whereby individual test center staff can chart a path of professional growth and development. Through the certification process, testing staff can gauge growth and be rightfully recognized for their accomplishments and expertise. Goals: 1. The certification program will create and set documentable standards to measure professional growth of test center staff. 2. Establish benchmarks that will serve as a guide for achievement. 3. To professionally grow test center staff in the testing profession. 4. Offer a mechanism that documents and certifies accomplishments that can be exhibited to superiors.

McCann Associates Level 1 Criteria: FACTA Certified Testing Specialist:  1 to 4 years’ experience in the testing field.  High School Diploma/Associates Degree – include transcripts  Letter of recommendation by your superior indicating your training and work as a testing proctor.  Individualized statement by your superior explaining your role as a testing professional and the importance of following proper testing procedures.  Copy of evaluation and/or job description indicating work as a testing proctor.  Signed documented reading of procedure manual.  Attendance to regional, state, national testing related conference  Level 1 Certification Assessment

McCann Associates Level 2 Criteria: FACTA Certified Testing Professional:  4 to 8 years of experience in the testing field  Associates/Bachelor’s degree – include transcripts  Letter of recommendation from your supervisor  Evaluations and/or job descriptions showing breadth of experience in testing  Attendance to regional, state, national testing related conference  Work as a committee member and/or FACTA meeting attendance and/or other proof of work on a testing related committee/project  Level 2 Certification Assessment

McCann Associates Level 3 Criteria: FACTA Certified Testing Manager:  8+ years of experience in testing and working as a testing supervisor/coordinator/director/manager/trainer/certifier  Bachelor’s Degree or higher – include transcripts  Letter of recommendation from your supervisor  Copy of job description  Proof of work with FACTA/NCTA or other testing professional organization at the committee level or higher.  Level 3 Certification Assessment

Custom Certification Development Process McCann Associates

Task Documentation Review/Stakeholder Interviews Job Task Analysis Meetings, Design the Survey Analysis of Survey and Participant Demographic Data Conclusion Setting and Concrete Task Development Job Task Analysis Translate Concrete Tasks to Blueprint Standards Deliver Survey to Amass Data GOAL: To identify the appropriate job requirements to be measured, and how their appropriateness should be characterized and quantified. The primary elements typically identified are tasks, objectives, knowledge, skills and abilities. McCann Associates

Standard Standard 1 Standard 2 Standard 3 Standard 4 Standard 5 Description for Standard 1 Description for Standard 2 Description for Standard 3 Description for Standard 4 Description for Standard 5 Description Define Standards Standard 6 Standard 7 Standard 8 Description for Standard 6 Description for Standard 7 Description for Standard 8 Standard 3 Standard 6 Standard 7 Description for Standard 3 Description for Standard 6 Description for Standard 7 GOAL: To identify the standards that will be delivered to determine candidate proficiency. McCann Associates

Develop Certification Assessment Pathways GOAL: To design assessment profiles that determine candidate pathways. McCann Associates Certification Assessment Tiered AchievementPass/Fail Score Determined by single score or average score of weighted standards Sample Certification Pathway Decision Maker(s) or Grit Assessment ?Background Questions ?

Standard Determine Test Blueprint Standard 1 Standard 2 Standard 4 Standard 5 Standard Items Delivered 20Total GOAL: To determine the weight of each standard, which will drive the number of test items to deliver against each standard. Often desired test times will drive test length. 20% 15% 25% 20% Weight 100% McCann Associates

Determine Depth of Item Pool N Items Standard Standard 1 Standard 2 Standard 4 Standard 5 Standard Items Delivered 20Total 20% 15% 25% 20% Weight 100% GOAL: McCann Assessment & Content Specialists, with support from the project team, will review item pools to determine the number of items available for each blueprint standard, based on a number of criteria including range of difficulty, cognitive complexity, key variance, and content coverage in accordance with approved exemplar items. McCann Associates

Build Certification Assessment McCann Associates To build an assessment configuration file that controls assessment length, adheres to blueprint specifications, contains test items that measure blueprint competencies, and applies meaningful scoring and reporting characteristics and decision output. GOAL:

Measure and Analyze Certification Assessment Results GOAL: To work with Subject Matter Experts (SMEs) to determine scores that determine candidate’s success based on performance levels. McCann Associates Test Performance Certification 1 (Pilot Testing of 30 Candidates) Certification Requirements Met 100% 70% 0% Performance at 70% or above determines candidate has met Certification Requirements Performance at 69% or below determines candidate has not met Certification Standards 50 0 (Scale)(%) Certification Standards not Achieved

Sample of individual candidate assessment report The diagnostic roster report in College Success™ will provide stakeholders with a detailed report of candidates performance in each strand measured on the assessment. This report will be of great value when reviewing a candidate’s strengths and weaknesses, thereby assisting in the decision making process. Listed below is an example of standard measures for a certification assessment. McCann Associates DescriptionScore Overall Score13/20 (65%) Knowledge of planning and scheduling techniques4/5 (80%) Knowledge of computer-based testing and scoring5/5 (100%) Database management skills4/5 (80%) Knowledge of regulations, policies, and standards0/5 (0%) Percent Correct 100% 80% 60% 40% 20% 0% Knowledge of planning and scheduling techniques Knowledge of computer-based testing and scoring Database management skills Knowledge of regulations, policies, and standards

Post-Certification Performance McCann Associates GOAL: To review candidate performance on pre-determined standards and further to establish benchmarks for success. Standards are pre- established in the development of the blueprints for each certification assessment. Standard # of Candidates Standard Score Standard % Standard % Standard % Standard % Standard % Standard % Standard % Standard % Total 30568%