Common Errors in Data Collection

Purpose of this Presentation  To point out the importance of accurate, consistent data collection  To illustrate trends in student outcomes that then may be utilized to improve services to meet students’ needs  To give examples of how evaluation data may be utilized to improve programs and influence policymakers

Important Concepts in Evaluation: Validity  Does the instrument measure what it is supposed to measure?  If valid, the interpretation of the results are sound and the results can be generalized.

Important Concepts in Evaluation: Reliability  Does the instrument measure the same thing repeatedly over time?  If reliable, instrument responses are consistent each time they are measured and the scoring is consistent.

For example: It is important to correctly identify CTP and CUP/CTP completers at the secondary level in order to track them at the postsecondary level. If this is not accomplished consistently, you will not have reliable data for comparisons from year to year (comparing apples to oranges).

Examples of Data Tracking & Comparisons If these data were not collected consistently, we may be indicating change that is not really happening. Therefore, any policy decisions based on faulty data will be pointless.

Review Slide Tracking Enrollment Trends at a Community College Over Time CUP completers show a decreasing enrollment while CTP completers show an increasing enrollment. This illustrates change at the consortium level.

Tracking Post-Secondary Intention Trends Over Time at the State Level

Remediation Rates 2004: Comparing National and State Statistics

Receive Feedback from Stakeholders for Program Improvement Utilizing Senior Survey results, business partner input and faculty and staff feedback for program improvement

Sample Senior Survey Item: “What Pathway Did You Follow?”

Top 10 Huskins and Dual Enrollment Classes Taken ClassN% Criminology Criminal Law Intro to Criminal Justice388.1 Juvenile Justice347.2 Law Enforcement Operations326.8 Ethics and Community Relations265.5 Music Appreciation204.3 Community Policing183.8 Expository Writing183.8 Organized Crime183.8

Receive Feedback about your CTE Work: Items from a Staff and Faculty Survey QuestionsYesNoNot Sure Do you believe that every student’s goal should be to attend a four-year University? 1.8%97%1.2% Based on your experience, do you think CTE lowers the dropout rate and keeps students in school? 86.5%4.1%9.4% Do you believe a community college education is adequate for some students? 98.8%1.2%

Open-ended Item Responses from Staff and Faculty Survey: “If CTE was removed from your curriculum, what impact do you think it would have on your school?”  A lot more dropouts. Many students only stay in school in order to take our CTE courses. Our CTE courses are rigorous, but allow for hands-on application. Our students learn a great deal in CTE and accordingly must be motivated to succeed.  I believe that removing the CTE program would leave a lot of students who lack the funding, family support and desire to attend a 4 year college lacking the skills they need to find gainful employment.  I shudder to think that would ever happen. I have students in my classes who would not continue in school if they did not have tech classes to keep them motivated. They identify with the tech programs.

Receive Feedback that May Impact Policy Decisions Accurate data presented to powerbrokers has power to influence decisions

Senior Survey Item: “Do you have a computer with internet access in your home?”

Senior Survey Item: “If you had had to complete two years of a foreign language, would you have completed your high school diploma?” (By Course of Study) CTPCUP/CTPCUP Yes39.3%87.5%96.9% No60.7%12.5%3.1% Chi-Square test found differences in cells statistically significantly different at.000 level

When planning your evaluation, plan to collect data that are:  Accurate  Reliable  Valid  Useful for your consortium  Useful for State and Federal reporting

Thank You! Praxis Research, Inc (704) 523 – 2999 Website: