Elizabeth L. Shoenfelt, Ph.D. Alicia Turner, M.A. Candidate Patricia Slack, M.A. Candidate David Normansell, M.A. Candidate Department of Psychology Western Kentucky University ASSESSMENT of the CONTENT VALIDITY of the WATER TRAINING INSTITUTE (WTI) CURRICULUM

OVERVIEW Content Validation – Training Program Evaluation Methods Criterion Phase  Participants  Procedure Content Phase  Participants  Procedure Analysis

Background: WTI 2 year associate program 2 certification programs (Water and Wastewater) Curriculum: general requirements and 2 specialized tracks (Water and Wastewater) 4 projected entry level jobs that WTI graduates will most likely enter: o Water Treatment Operator o Wastewater Treatment Operator o Distribution Systems Operator o Collection Systems Operator

Background: NSF Grant Grant evaluation includes an assessment of the content validity of the WTI Curriculum Will examine the 4 courses currently in place to ensure they are preparing students with the Knowledge Skills and Abilities (KSAs) they need to successfully enter the workforce in any of the 4 identified entry level jobs

Content Validity CONTENT VALIDITY: The extent to which the material taught in the training course reflects the actual KSAs required for effective job performance The more similar the WTI training program content is to the job, the more effective it should be in preparing WTI program graduates.

Content Validation – Training Evaluation Methods Matching Technique ( Ford & Wroten, 1984) Analyzed the effectiveness of a Police department training program by identifying Knowledge, Skills, and Abilities (KSAs) considered important to job performance and the KSAs that currently taught in the training program (i.e., time spent) Used a matching matrix to graph the data to identify Hits and Misses Hit: training emphasis reflects training needs (i.e., high importance KSA receives high emphasis in program) Miss: training emphasis does not reflect training needs o Deficiency: high importance KSA receives low emphasis o Excess: low importance KSA receives high emphasis

Matching Matrix

Content Validation – Training Evaluation Methods Linking Technique (T eachout, Sergo, & Ford, 1997) Used the matching matrix but also linked training emphasis to difficulty of learning More effective picture: High Importance KSA with Low Emphasis – is it a Deficiency or does it have Low Difficulty of Learning?

Current Evaluation Will examine the content validity of the WTI curriculum Will use combination of the Matching and Linking Techniques 2 Phases: Criterion (Job) Content (WTI Curriculum)

Criterion Phase: Procedure Developed preliminary Job Knowledge Surveys (JKS) based on materials received from the Associate Director of the Center for Water Resource Studies at WKU that included the KSAs needed for the 4 entry level jobs For each element of job information, JKS included ratings for: a) Time Spent on job b) Importance to job c) Difficulty of Learning d) When learned (before hire/formal trn/on the job) e) Should it be taught in WTI f) Is it needed for certification

Job Knowledge Survey: Sample

Criterion Phase: Participants WTI Steering Committee Members Knowledgeable individuals about Water and Wastewater Industry for both Kentucky and Tennessee. From a variety of organizations from Kentucky and Tennessee such as Kentucky Rural Water Association and Tennessee Association of Utility Districts JKSs were distributed to steering committee members during their WTI workshop/conference in Louisville, KY. The members pilot tested the surveys and provided feedback to refine surveys

Criterion Phase: Procedure The JKSs were refined based on Steering Committee feedback Steering Committee members collectively identified 40 water and wastewater incumbents to serve as job content Subject Matter Experts (SMEs) to complete the JKSs (Approximately 10 per entry level job) The JKS were mailed to the Steering Committee Members for distribution to the identified job incumbent SMEs with a return deadline of October 30, 2009 The JKS data have been entered. We are now in the process of analyzing the JKS data.

Content Phase: Participants Second Phase is a content analysis of the WTI Curriculum We will develop Course Content Surveys to determine KSAs taught and training emphasis (time) WTI Curriculum Content SMEs: o Bowling Green Community College WTI Instructor o Teaching Assistant for the Center for Water Resource Studies at WKU o The 4 current WTI students also will complete the Course Content Surveys

Content Phase: Procedure The SMEs will develop a list of KSAs that are taught in each of the current 4 courses through a “brainstorming” panel discussion At the end of the discussion, Curriculum SMEs will be provided with the course syllabi and any relevant training materials to further identify KSAs taught The KSAs identified will be used to develop a Course Content Survey. Each KSA taught in the WTI course will be rated on the time spent teaching the KSA. The SMEs (including students) will complete the survey

Analysis Matching Matrix will be used to compare: Importance ratings vs. Time Spent Training ratings Difficulty of Learning ratings vs. Time Spent Training ratings Identify hits and misses in the courses: Hits – appropriate emphasis in curriculum Deficiencies – where more emphasis is needed Excesses – where time on topic can be devoted elsewhere

Matching Matrix

Formative Evaluation Data from the Content Validation can be used: To revise current WTI courses (if needed) to ensure appropriate emphasis of job-related KSAs In developing other WTI courses to ensure they contain appropriate emphasis of job-related KSAs

References Primary References Bownas, D., Bosshardt, M., & Donnelly, L. (1985). A Quantitative Approach to Evaluating Training Curriculum Content Sampling Adequacy. Personnel Psychology, Ford, J., & Wroten, S.(1984). Introducing New Methods for Conducting Training Evaluation and for Linking Training Evaluation to Program Redesign. Personnel Psychology, Sproule, C. Rationale and Research Evidence Supporting the Use of Content Validation in Personnel Assessment. Retrieved from International Personnel Assessment Council Website: Teachout, M., Sego, D., & Ford, J. (1997). An Integrated Approach to Summative Evaluation for Facilitating Training Course Improvement. Training Research Journal,

Questions?

Content Validity Ratio ((N imp – N not-imp) / N)