1. Rising to the Occasion: Systematic STEM Academy Assessment Practices Virginia Department of Education Office of Career and Technical Education Services October 25, 2012 “Using Data for Program Planning and Continuous Improvement”

2. Rising to the Occasion: Systematic STEM Academy Assessment Practices Nathan Carter, Ph.D. Research Consultant

3. Background Information In 2008, the Virginia Board of Education approved the criteria for establishing Governor’s STEM Academies. As part of the criteria, the Virginia Department of Education (VDOE) was required to conduct annual evaluations of all established STEM Academies. In order to evaluate STEM Academies effectively VDOE needed to: Acquire current descriptions of each career pathway offered at a STEM Academy Develop an informative template that STEM Academies could use to complete an Annual STEM Academy Program Assessment Report

4. Typical Problems with Program Assessments  Pressure for doing a comprehensive program assessment can cause a “rush to completion syndrome” and produce results that are not very useful.  When there is not a team-oriented approach to completing a comprehensive program assessment important voices can be left out of the discussion.  In some situations comprehensive program assessment is not based on a clear understanding of core program values or agreement on what the short and long-term goals of the program are.

5. Another Common Problem: DATA OVERLOAD!!!!! If I’d known they wanted me to use all of this quantitative data – I would NEVER have asked for it!!!

6. Four General Purposes of Quality Comprehensive Program Assessment  To improve – The assessment process should provide feedback to stakeholders on how a school program can be improved.  To inform – The assessment process should inform decision makers and program stakeholders of the continual contributions and impacts of a program on meeting short and long-term goals.  To prove – The assessment process should demonstrate what the program is accomplishing and how these accomplishments are changing overtime.  To support- The assessment process should support faculty, administrators, and others as they think about what actions need to be taken to continue to support students.

7. Requirements for a Complete Assessment Process  It must be comprehensive – must cover the full range of learning outcomes that are connected to program values and goals  It must include multiple judgments – multiple sources of evidence must be used  It must include multiple dimensions – different facets of student performance must be included  It must collect direct evidence – direct measures of student attainment must be used  It must include qualitative interpretations of the data and evaluations of patterns overtime.

8. Iterative Model of the STEM Academy Annual Assessment Process Plan Collect Data Implement Analyze Data Reflect & Revise

9. STEM Academy Career Pathway Description Sheet Grade Level Core Course Name State Course Code Content Area Career Pathway Course Options (indicate all that apply for each course) EnglishMathScience History & Social Science CTE Other Course Requirement For CTE Program Completion Course Requirement For Advanced Studies Diploma Course Requirement For Advanced Technical Diploma Industry Certificatio n Available Course Available To Be Taken As A Dual Enrollment Course Non- Traditional Career Preparation Course (M or F) Career Pathway Name___________________________ Workplace Experiences Offered With Career Pathway Job Shadowing Internship Mentorship Cooperative Education Student Apprenticeship Service Learning Other

10. Sections of the STEM Academy Assessment Report SECTIONDATA ELEMENTS INCLUDED Background Information Name of STEM Academy, year STEM Academy opened, STEM Academy director’s name & contact information, STEM Academy’s purpose, STEM Academy’s current short & long-term goals. STEM Academy Acceptance Rates Number of applications received, number of applications accepted, number of students placed on a waitlist STEM Academy Experiences Number of elementary or middle school students in STEM Academy summer activities, number of elementary or middle school students in STEM Academy classes during the school year STEM Academy High School Enrollment Total number of high school students enrolled, newly enrolled, and number who were expected to return but did not (by grade level and by career pathway) Each section has a quantitative and qualitative data collection requirement

11. Sections of the STEM Academy Assessment Report (cont.) SECTIONDATA ELEMENTS INCLUDED STEM Academy AP Course & AP Test Participation Number of STEM Academy students enrolled in AP courses, number enrolled in AP courses and taking AP tests, number taking AP tests without being enrolled in AP courses. STEM Academy Dual Enrollment Participation Number of STEM Academy students taking 1, 2, or 3+ dual enrollment courses, number of students passing 1, 2, or 3+ dual enrollment courses STEM Academy SOL Achievements Number of STEM Academy students scoring Advanced Proficient, Proficient, or Not Proficient on the different SOL exams STEM Academy Credential Production Number of STEM Academy graduates/completers taking and passing Workplace Readiness Skills Assessments, or Occupational Assessments, State Licensure Exams, or National Industry Certification Exams

12. Sections of the STEM Academy Assessment Report (cont.) SECTIONDATA ELEMENT INCLUDED STEM Academy Graduation Data Number of STEM Academy graduates/completers earning Board approved diplomas with a special emphasis on whether STEM Academy graduates earn advanced studies diplomas and/or CTE credentials upon graduation STEM Academy Post-graduation Data Number of STEM Academy graduates/completers who enrolled in credit-bearing courses, or earned college degrees, or transitioned into high-skill, high-wage, high- demand careers, etc. Summary of STEM Academy Progress 1.STEM Academies compile their qualitative and quantitative data and assess to what extent they are meeting their short and long-term goals. 2.STEM Academies identify their strengths as well as what areas need attention. 3.STEM Academies adjust their short and long-term goals where needed.

13. VDOE provided each STEM Academy with a PDF document that showed which tables and available data in the template could be populated with state- collected data. rt. STEM Academies will organize their own data management team and set deadlines for populating tables in the STEM Academy Assessment Report using locally stored data sources. STEM Academies will compare the data tables provided by the VDOE with their own data tables on the different areas team. STEM Academies will create a list of all the relevant data inaccuracies between the two sources and document possible reasons for the differences. In the case of discrepancies, STEM Academies will use their own locally validated data to populate the tables in the Assessment Report and complete their annual evaluation. START START END END STEM Academies will submit their Annual Assessment Report and a list of any discrepancies uncovered during the data validation process to VDOE along with descriptions of career pathways Data Validation Flow Chart

14. Important Questions To Think About As You Move Forward How are you currently embracing a data-driven decision- making process that leads to actionable results?

15. Consider including people from various stakeholder groups to work colloboratively on the comprehensive program assessment plan. Who is going to be a part of the program assessment team?

16. When will we administer the assessment? When will we collect the data? When will we analyze the data? When will we reflect upon the data? When will we make recommendations? When will we make the decisions about the recommendations? When will we provide written documentation about the decisions? When will we share the data with other stakeholders? What is a realistic schedule for completing the program assessment?

17. The next CTE Directors Video Streaming Session will be held November 8, 2012 Selecting Appropriate Industry Certification Examinations Presenters: Gordon Creasy Joseph Wharff Continuing Professional Development via Video Streaming

18. George R. Willcox, CTE Coordinator Office of Career and Technical Education Virginia Department of Education P. O. Box 2120 Richmond, VA 23218-2120 804 225-2052 804 530-4560 cte.@doe.virginia.gov For Additional Information Contact