© 2014, Florida Department of Education. All Rights Reserved. VAM and DJJ Accountability Rating System Update Assessment and Accountability.

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Agenda VAM Files and Model Results Walk Through VAM Rule FAQ VAM 2016-17 Timeline DJJ Accountability Rating System Background DJJ Measure Walkthrough DJJ Simulation Results DJJ Rule Development Timeline

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. VAM Files Generally available by the end of the 1 st week in August. Uploaded to sharefile for retrieval by district users authorized to view personally identifiable data at both teacher and student level. Each district receives a package of 74 data files containing student, teacher, school, district, statewide and model-level results. There is also a “Read Me” file. The “FileContents” tab provides an overview of all of these files. Other tabs in the “Read Me” file contain layouts and descriptions of the variables contained in each of the major file types.

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Primary VAM Files Scores that are required to be used under SBE Rule 6A-5.0411, or that provide optional scores for unapproved models, have file names that are in bold on the “FileContents” tab of the “Read Me” file. These include: Teacher_Aggregation_3yr Teacher_Alg_Grade9 Teacher_Alg_Grade8 - optional School_Aggregation_3yr (1yr and 2yr scores may be appropriate for administrators who have not been at the school for all 3 years)

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Teacher_Aggregation_3yr VAM File Primary fields of interest for use in annual evaluations include: Agg_vam_combined_cattxt Agg_vam_combined_catscr Fewer_than_10 HOSS Flag_1314, Flag_1415, Flag_1516

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Teacher_Alg VAM files Primary fields of interest for annual evaluations include: Teacher_vam_estimate_catttxt Teacher_vam_estimate_catscr Fewer_than_10 HOSS

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. School_Aggregation VAM files Primary fields of interest for annual evaluations include: Agg_vam_combined_cattxt Agg_vam_combined_catscr HOSS Flag_1314, Flag_1415, Flag_1516

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. After reviewing if you still have questions about how to use these files, Call (850) 245 - 0411 or email VAM@fldoe.orgVAM@fldoe.org

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Teacher Ratings Model Highly EffectiveEffective Needs ImprovementUnsatisfactory FSA15%56%15% 9th Grade Algebra 1 2%88%8%1% 8th Grade Algebra 1 4%80%14%3%

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. School Ratings Model Highly EffectiveEffective Needs ImprovementUnsatisfactoryNA FSA 3 Year30%32%11%25%2% FSA 2 Year29%34%12%23%2% FSA 1 Year24%43%12%19%2% 9th Grade Algebra 1 0%70%<1%0%30% 8th Grade Algebra 1 0%45%<1%0%54%

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Impact data show the relationship of the teacher score to various classroom characteristics There are several ways to interpret a non-zero relationship, including the following: The model does not fully account for non-random assignment of students to teachers Classroom characteristics affect the rate of student learning There are real differences in teacher effectiveness, which are correlated with classroom characteristics Impact Data Results 16

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Required Components of the Evaluation Section 1012.34, F.S., Personnel evaluation procedures and criteria Section 3 – Evaluation procedures and criteria; Must include 1.Performance of students (at least 1/3) 2.Instructional practice/leadership (at least 1/3) 3.Other indicators of performance 8

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Rule Authority and Content for 6A-5.0411 Section 1012.34, F.S., Personnel evaluation procedures and criteria Section 8 – Rulemaking; Establish specific, discrete standards for each performance level (Highly Effective, Effective, Needs Improvement/Developing, and Unsatisfactory) based on student learning growth models approved by the commissioner; Ensure clear and sufficient differentiation in the performance levels; Note: This rule applies only to the portion of the performance of students component of educator evaluations that relate to courses assessed by statewide, standardized assessments where there are growth models that have been approved by the Commissioner.

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Required Measures (Section 3 of the Rule) 1.3 year aggregate combined VAM score - ELA/Mathematics 2.Individual grade and subject VAM score - Algebra I 24

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Standard Error An estimate of a teacher’s impact on student learning (the score) contains some Variability. The standard error is a statistical term that describes that variability. Using the standard error to construct a confidence interval around a score (like the +/- 3 points in an opinion poll) is a good statistical practice that can assist in increasing the accuracy of classification decisions. 26

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Value-Added Results and Scores The formula produces a value-added score for a teacher, which reflects the average amount of learning growth of the teacher’s students above or below the expected learning growth of similar students in the state, using the variables accounted for in the model. A score of “0” indicates that, on average, students performed no better or worse than expected based on the factors in the model A positive score indicates that students, on average, performed better than expected A negative score indicates that students, on average, performed worse than expected 25

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Section 4 - Performance Level Standards Use the standard error to construct confidence intervals. Use these confidence intervals to establish performance standards. Give teachers the benefit of the doubt when there is below-threshold statistical uncertainty. Place a larger share of teachers statewide into higher performance categories than student-level metrics using the same assessment results. 28

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Frequently Asked Questions What measures are required to be used? Can other “performance of students” measures besides VAM be included? Do Districts have to use the ratings the Department provides? Can Districts continue to use “percent meeting expectations”? Can Districts use school-wide VAM anymore? Are there circumstances where the score doesn’t have to be used?? How do we combine with our local measures or other components if they are not on a common scale?

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. What Measures Are Required to Be Used? Teachers: ELA & Math = Agg_vam_combined_catscr from the teacher_agg_3yr file 9th grade Algebra 1 = Teacher_vam_estimate_catscr from the teacher_alg_grade9 file Administrators: Agg_vam_combined_catscr from the school_aggregation_3yr, school_aggregation_2yr, or school_aggregation_1yr as appropriate

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Can Other Measures Be Used? Yes, but restrictions apply. If the teacher has only taught VAM courses during the past 3 years, at least 1/3 of the evaluation must be based on the VAM score required. Other measures may be used in these cases, but the cannot supplant any of the required minimum 1/3. If the teacher has taught both VAM and non-VAM courses, other measures can be included and weighted based on the teacher’s teaching assignment over the period of time. VAM would reflect the relative proportion among the 1/3 that it did of the teaching assignment. Administrators may have measures related to other subjects taught at the school incorporated into the minimum 1/3 student performance component of the evaluation, provided they are weighted appropriately.

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Are Districts Required to Use the Ratings Provided? Yes. Districts may convert them to another scale to combine with their IPS and other measures included in the evaluation, but the rating must be preserved across any such conversion without overlap, and in such a way that still allows for 4 different levels of performance.

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Can “Percent Meeting Expectations” Still Be Used? Yes, but only in addition to, and not instead of, the required VAM score described earlier. If it is used, it cannot replace a portion that would otherwise be required to be covered by a VAM score.

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Can School-Wide VAM Be Used Anymore? Yes, but only in specific, limited circumstances. Must be for an educator or administrator who serves all students at the school such as a guidance counselor or media specialist.

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. How Do We Combine With Our Local Measures Or Other Components if They Are Not On A Common Scale? You may either convert scales for the other components and measures into the 4 point scale used for the VAM scores, or Convert the VAM score to your other scale prior to combining with the other components and metrics, provided there is no overlap and 4 different performance levels can still be generated from the final scores

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Are There Circumstances Where The Score Doesn’t Have to Be Used? Yes, there are 4 of them: The Teacher taught one or more students with expected scores above the HOSS Fewer than 10 assessments were used to compute the score The teacher was not present for more than 50% of the school days associated with the course The teacher taught Advanced Academics courses (7755040 or 7855040) In these situations, use of the score is not required, but is optional.

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. VAM 2016-17 Timeline DateActivity October 28, 2016  All 2014-15 evaluations due via Survey 5 Staff Database submission for inclusion in annual legislative report. Week of November 30, 2015  Fall Survey 2 RVT Opens Mid-March, 2017  Spring Survey 3 RVT opens. April 14, 2017  Survey 2/3 match request list files from 2013-14 posted to sharefile for districts to update. April 28, 2017  All districts requesting any changes from the prior year to their Survey 2-3 match conditions for VAM calculation purposes must have updated the document in their sharefile directory to reflect changes.

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. VAM 2016-17 Timeline (continued) DateActivity May 12, 2017  Fall Survey 2 and Spring Survey 3 Roster Verification Tools close.  Districts electing to use Survey 2 and 3 data instead of RVT for VAM purposes must have those data files finalized. Week of June 2, 2017  FDOE posts files to sharefile site with Survey 2-3 requests applied for district review.  Districts verify that Survey 2-3 requests were applied correctly.  The district accountability contact in every district replies to FDOE that files were compiled accurately. July 2017  Processes VAM calculations for FSA ELA,FSA Mathematics, and Algebra I. First Week of August, 2017  FLDOE posts VAM results for teachers to school districts via sharefile website.

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Rule Authority Section 1003.52(16), F.S., Educational services in Department of Juvenile Justice programs When establishing this rule, the Department of Education must consult with Department of Juvenile Justice District school boards Providers

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. System Requirements Rating must be based on objective and measurable student performance measures that evaluate a student’s educational progress Limited to prevention, day treatment and residential programs Measures must be based on appropriate outcomes for all students in juvenile justice education programs Must take length of stay into consideration

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Measure Requirements Performance measures shall include outcomes that relate to: Student achievement of career education goals Acquisition of employability skills Receipt of a high school diploma or its equivalent Grade advancement The number of CAPE industry certifications earned

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Rating System Up to 11 components included Only components for which a program has sufficient data are included All are based on a percentage calculation For most components students must have been in the program for at least 40 days to be included in a measure (s. 1003.52(5), F.S.) Additional requirements for some components

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. 1 - Attendance Rate The percentage of students who showed improvement in their attendance rate following re-enrollment Based on an exit cohort Only includes students who were in the program for at least 40 days Must have been enrolled in school prior to and following release from the DJJ school Based on days attended of those enrolled Credit awarded if at 95% upon return, regardless of prior rate

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. 2 - Graduation Rate The percentage of students whose last school attended was a DJJ school who graduated or earned a GED within 5 years of entering 9 th grade Based on a 9 th grade entry cohort Only includes students who were in the program for at least 40 days DJJ school must have been the last school they attended

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. 3 - Certified Teachers The percentage of core courses taught by teachers certified for the subject area Only includes core courses as identified by the Course Code Directory

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. 4 - Postsecondary Enrollment The percentage of students attending postsecondary institutions Based on an exit cohort Only includes students who were in the program for at least 40 days Only includes students with a diploma or GED at the time of exit Enrollment can be anytime within one year of exit

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. 5 - Employment The percentage of students employed Based on an exit cohort Only includes students who were in the program for at least 40 days Only includes students who were at least 16 at the time of release and not attending a post-secondary institution Employment can be anytime within one year of exit

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. 6 - Learning Gains – ELA The percentage of students who make learning gains on ELA assessments Only includes students who were in the program for at least 40 days prior to the assessment Learning gain measured as defined in the earlier presentation on school grades Data will not be available for this measure until next year

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. 7- Learning Gains - Mathematics The percentage of students who make learning gains on Mathematics assessments Only includes students who were in the program for at least 40 days prior to the assessment Learning gain measured as defined in the earlier presentation on school grades Data will not be available for this measure until next year

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. 8 - CAPE The percentage of students who earn a CAPE Industry Certification (High School) or Digital Learning Tools (Middle School) Certificate Only applies to programs with a designed LOS of 9 months or longer Based on an exit cohort Only includes students who were in the program for at least 40 days Certificate can be earned anytime within one year of exit Because digital learning tools reporting is new this year, data will not be available for this measure until next year to allow for the full 1 year follow up

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. 9 - Common Assessment Learning Gains - Reading The percentage of students who make learning gains on the DJJ Common Assessment Based on an exit cohort Only includes students who were in the program for at least 40 days Student must have both a pre- and a post- assessment Learning gain means student increases a level or maintains if already at the highest level Because new assessment goes live on July 1, data for this measure will not be available until 2016-17

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. 10 - Common Assessment Learning Gains - Math The percentage of students who make learning gains on the DJJ Common Assessment Based on an exit cohort Only includes students who were in the program for at least 40 days Student must have both a pre- and a post- assessment Learning gain means student increases a level or maintains if already at the highest level Because new assessment goes live on July 1, data for this measure will not be available until 2016-17

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. 11 - Data Integrity The percentage of students who have pre- and post-test data on the common assessment Based on an exit cohort Only includes students who were in the program for at least 40 days Based on new assessment, so data for this measure will not be available until 2016-17

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. DJJ Ed. Program Accountability Model (A maximum of 11 components) Learning Gains K-12 OutcomesEducator QualityPost-K-12 Outcomes Process Measures FSA ELA (0% to 100%) Increased Attendance (0% to 100%) Core Courses Taught by Certified Teachers (0% to 100%) Postsecondary Enrollment (0% to 100%) Common Assessment Data Quality (0% to 100%) FSA Math (0% to 100%) Industry Certifications (0% to 100%) Employment (0% to 100%) WIN Reading (0% to 100%) 5 Year Graduation Rate (0% to 100%) WIN Math (0% to 100%) Simulation (2014-15)Year One (2015-16)Year Two (2016-17) The rating is based on the percentage of total points earned, and programs are graded based only on the components for which they have sufficient data (WIN learning Gains will be included beginning in 2016-2017) Measures are classified on a 3 point scale before being combined Cut-scores used to determine classification are specific to each program type/measure combination

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Procedures for Calculating Scores and Ratings Three ratings include Commendable, Acceptable and Unsatisfactory Components will be rated on a 3-point scale Rating will be the simple (unweighted) average of the components with sufficient data If a DJJ education program doesn’t have sufficient data to generate a rating for three years in a row, the prior three years of data will be aggregated to produce a rating

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. State Board of Education Will Need to Establish the DJJ Ed. Prog. Grading Scale Measures are classified before computing final rating to standardize scale among measures to promote fairness Classification scale for each measures is based on actual relative performance among program type Classification scale is set for each measure by program type independently

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Attendance Program TypeCommendableAcceptableUnsatisfactory Prevention 100% to 60%59% to 35%34% to 0% Intervention 100% to 60%59% to 35%34% to 0% Non-Secure Residential 100% to 60%59% to 35%34% to 0% Secure Residential 100% to 60%59% to 35%34% to 0%

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Graduation Rate Program Type CommendableAcceptableUnsatisfactory Prevention100% to 50%49% to 15%14% to 0% Intervention100% to 50%49% to 15%14% to 0% Non-Secure Residential 100% to 50%49% to 15%14% to 0% Secure Residential 100% to 50%49% to 15%14% to 0%

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Certified Teachers Program TypeCommendableAcceptableUnsatisfactory Prevention100%99% to 50%49% to 0% Intervention100%99% to 50%49% to 0% Non-Secure Residential100%99% to 50%49% to 0% Secure Residential100%99% to 50%49% to 0%

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Employment Program TypeCommendableAcceptableUnsatisfactory Prevention100% to 80%79% to 50%49% to 0% Intervention100% to 70%69% to 30%29% to 0% Non-Secure Residential100% to 70%69% to 30%29% to 0% Secure Residential100% to 70%69% to 30%29% to 0%

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Overall Ratings Program TypeCommendableAcceptableUnsatisfactory Prevention3.0 to 2.52.4 to 1.61.5 to 1.0 Intervention3.0 to 2.52.4 to 1.61.5 to 1.0 Non-Secure Residential3.0 to 2.52.4 to 1.61.5 to 1.0 Secure Residential3.0 to 2.52.4 to 1.61.5 to 1.0

www.FLDOE.org © 2014, Florida Department of Education. All Rights Reserved. Timeline DateActivity Fall 2016  Develop 2 nd cohort of simulation data including ELA and Math learning gains from FSA and CAPE industry certifications and digital learning tools measures. Spring 2017  Incorporate partial year simulations from DJJ Common Assessment into simulations  Gather requirements for and begin development of data corrections processes and applications Summer, 2017  Present rule for State Board Approval based on simulations and Commissioner’s recommended cut scores. Fall, 2017  Develop initial DJJ Rating cohort Spring, 2018  Districts and DJJ providers begin initial data updates and corrections process for 1 sst year DJJ ratings cohort and measures March, 2018

© 2014, Florida Department of Education. All Rights Reserved. VAM and DJJ Accountability Rating System Update Assessment and Accountability.

Similar presentations

Presentation on theme: "© 2014, Florida Department of Education. All Rights Reserved. VAM and DJJ Accountability Rating System Update Assessment and Accountability."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

© 2014, Florida Department of Education. All Rights Reserved. VAM and DJJ Accountability Rating System Update Assessment and Accountability.

Similar presentations

Presentation on theme: "© 2014, Florida Department of Education. All Rights Reserved. VAM and DJJ Accountability Rating System Update Assessment and Accountability."— Presentation transcript:

Similar presentations

About project

Feedback