1. Taking Stock: Assessing & Improving Early Childhood Learning and Program Quality Board of the Massachusetts Department of Early Education and Care May 13, 2008 Dr. Sharon Lynn Kagan Columbia University & Yale University

2. Overview  Impetus for the Work  Challenges  Proposed System Design  Differing Viewpoints on Using Child Assessment Data for Local Agency Accountability  Action Steps  Implications

3. Part I Impetus for the Work

4. Growing Importance of ECE  Throughout the United States and countries around the world, Early Childhood Education is ascending to prominence.  States are investing more dollars in the field, with growth in virtually every segment of ECE, including: –Direct Services –Professional Development –Quality Rating Systems

5. Growing Investments  These growing investments are matched by calls to assess that the investments are paying off.  Are programs delivering the services they say they are delivering?  Are children achieving the outcomes desired?  Are the investments yielding appropriate returns?

6. Growing Emphasis on ECE  Practitioners also are interested in results, recognizing that: –The early years come only once for a child –The early years are critical to long term development –Teachers are the keys to making programs high quality –Teachers and policy makers can learn from data about children.

7. Interest in Accountability  Interest in Accountability is not limited to ECE alone.  All human service agencies are being asked to take on a results orientation.  Business and industry has provided a lead in this area and fueled efforts toward greater efficiency and quality.  There is a prevailing ideology that accountability provides the information to make progress.

8. Many Factors Provide Impetus  So there is a collision of interest that propels a focus on outcomes and accountability: –New data that attests to the importance of the early years –New investments and the need to see if they are making a difference. –New movement toward accountability as an elixir for quality improvement.

9. Part II Challenges

10. Four Challenges 1.Structural Challenges 2.Conceptual Challenges 3.Technical Challenges 4.Resource Challenges

11. Challenges: Structural Challenges: Structural  Fragmented non-system of programs for preschool-aged children  Disjointed early childhood and public education policies

12. Multiple Standards and Assessments Child CareHead StartState Pre-KSpecial Education Program Quality Standards State Licensing Standards (50 states) Quality Rating Systems (QRS) (13 states + 29 pilots) Program Performance Standards State Program Standards (39 states) IDEA regulations State program standards Assessing Program Quality Licensing Visits QRS Assessments (13 + 29) PRISM ReviewsProgram Monitoring (30 states) State Program Monitoring Standards for Children’s Learning Early Learning Guidelines (49 states) Head Start Child Outcomes Framework Early Learning Guidelines (49 states) 3 functional goals Child Assessments No current requirementsNational Reporting System Pre-K Assessments (13 states) Kg. Assessments (17 states) States report % of children in 5 categories on 3 goals Research/EvaluationsYes Kindergarten to Grade 3 Standards, Assessments, Data

13. Challenges: Structural  Costs, burdens, confusion of multiple standards, assessments, & reports.  Multiple new initiatives all at once.  Pre-K–K-3 disconnects: –Pre-K assessments aren’t transferred to schools. –Standards, assessments, curricula aren’t aligned.

14. Challenges: Conceptual  Early childhood assessment training and practice vs. standards-based assessment and curriculum.  Early childhood reliance on program standards/data vs. expanded interest in children’s learning.

15. Challenges: Technical  Need appropriate assessment tools and methods to report on: –Progress/status of young children in all domains of learning and development –Young ELLs and children with disabilities –Program quality in diverse local agencies

16. Challenges: Resources  Limitations and inequities in funding for: –Programs –Infrastructure  Risk that accountability efforts ignore and exacerbate inequities in resources

17. Task Force Charge  Recommendations for a state accountability system for early education programs for pre- kindergarten children and linkage to standards-based assessments in kindergarten-grade 3.

18. Origins  Conceived by The Pew Charitable Trusts as part of their Advancing Quality Pre-K For All initiative.  Additional funding from the Foundation for Child Development and the Joyce Foundation.  Task Force convened in fall, 2005, report release fall, 2007.  Presentation reflects progress-to-date.

19.  Dr. Sharon Lynn Kagan, Chair  Dr. Eugene Garcia, Vice-Chair –Dr. W. Steven Barnett –Ms. Barbara Bowman –Dr. Mary Beth Bruder –Dr. Lindy Buch –Dr. Maryann Santos de Barona –Ms. Harriet Dichter –Mr. Mark Friedman –Dr. Jacqueline Jones –Dr. Joan Lombardi –Dr. Samuel Meisels –Ms. Marsha Moore –Dr. Robert Pianta –Dr. Donald Rock Task Force Members

20. Part III Proposed System Design

21. Framing Beliefs  Accountability is here to stay.  Programs should be held to performance standards that are documented and verified.  Assessments should inform policy decisions and be tied to program enhancement efforts.  Current approaches to accountability and assessment must be reformed.

22. State Accountability & Improvement System Design Pre-K-Grade 3 Alignment and Linkages Assessment/Program Improvement Approaches APPROACHESI CHILD POPULATION II PROGRAM POPULATION III STATE PROGRAM EVALUATION IV LOCAL AGENCY QUALITY CORE QUESTION How well are all children progressing in learning and development? What is the quality of all early childhood programs? What is the quality and how well are children progressing in specific state programs? What is the quality in local agencies? HOW DATA IS USED - Oversight of state investments/initiatives - Planning new investments/initiatives - Baseline information for K-12 education planning - Oversight of state investments/initiatives - Planning new investments/initiatives - Baseline information for K-12 education planning - Program-wide improvement efforts - Refining standards/policies - Appropriations decisions - Technical assistance to individual agencies. - Awarding incentives and recognition to local agencies for program improvements - Decisions on funding local agencies Infrastructure Early Learning & Program Quality Standards Program Rating & Improvement Professional DevelopmentData Management & Reporting

23. Infrastructure System Infrastructure Early Learning & Program Quality Standards Program Rating & Improvement Professional Development Data Management & Reporting

24. Infrastructure  Early Learning and Program Quality Standards –Alignment between: Standards, assessment systems and curricula Standards between ages and grades State and federal program structures and funding streams Child and program standards

25. Infrastructure  Program Rating & Improvement –Assesses and reports on the quality of all forms of early education programs –Provides technical assistance and professional development to improve quality –May provide public recognition/incentives to reward higher levels of quality

26. Infrastructure  Professional Development System –Links informal training with formal education, provides career pathways, links education and compensation. –Supports training on assessment administration, analysis and use.

27. Infrastructure  Data Management and Reporting –All-in-one place data on: Children Programs Workforce –Unified system of child identification numbers. –Provides for quality assurance of data and assessments.

28. Assessment Approaches  States vary in: –What they want to know –How they plan to use data –Available resources  States may implement one or any combination of options.  Report includes cautions/safeguards for each option.

29. Assessment/Improvement Approaches APPROACHESChild Population Program Population State Program Evaluation Local Agency Quality CORE QUESTION How well are all young children progressing in learning and development? What is the quality of all early education programs? What is the quality and how are children progressing in specific state programs? What is the quality in local agencies?

30. All Four Approaches Discuss: All Four Approaches Discuss:  Questions that can be Answered  Data to be Collected  Designs for Data Collection  Uses of the Data  Challenges and Cautions

31. I. Child Population Approach  How well are all young children progressing in learning and development? –Data: Demographic data collected on representative sample of all young children Comprehensive data on health, well-being, pre- school enrollment –Designs: At Kindergarten Entry At Ages 1, 3, 4 Longitudinal ECLS State Strategy

32. I. Child Population Approach  How well are all young children progressing in learning and development? –Uses: Planning interagency investments/initiatives Legislative oversight Baseline information for public education –Challenges/Cautions: Gaining access to unrolled preschoolers Can’t use data to maker inferences about pre- school programs’ impacts/quality No causal attributions Expensive and not routinely done

33. II. Program Population Approach  What is the quality of services in all early childhood programs? –Data: Program quality Workforce Public investments –Design: Data collected from states Program Improvement (PRS/QRS) System

34. II. Program Population Approach  What is the quality of services in all early childhood programs? –Uses: Planning interagency investments/initiatives Legislative oversight Baseline information for public education –Challenges/Cautions: Need to consider quality of programs in light of investments in them Need to be sensitive to changes in program quality Unable to attribute causality to program quality rankings

35. III. State Program Evaluation Approach  What is the quality and how well are children progressing in specific programs? –Data: Sample of centers that represent the universe of programs to be studied Comprehensive data on children, program, and teachers Data aligned with program mandates and standards –Design: Standard Evaluation Design-program and instrument identification, data collection and analysis Program Rating System Design-uses data for program information from the PRS/QRS

36. III. State Program Evaluation Approach  What is the quality and how well are children progressing in specific programs? –Uses: Program-wide improvement efforts Refining standards/policies Appropriations decisions –Challenges/Cautions: Attribution of single program effects due to many children’s participation in multiple programs (e.g., lack of pure control group) Pre-mature evaluation inappropriate Implementation fidelity to program design

37. IV. Local Agency Quality Approach  What is the quality in local agencies? –Data: Program quality data in relation to state standards Observations of teaching/learning opportunities –Design: Use PRS/QRS data On rotational basis –1/3 annually –Low performing programs

38. IV. Local Agency Quality Approach  What is the quality in local agencies? –Uses: Technical assistance to individual local agencies Awarding incentives and public recognition –Challenges/Cautions: Heavy burden on state agencies Heavy costs to do classroom observations States need to develop technical assistance to address program needs/weaknesses

39. Pre-K—Grade 3 Integration  Align standards, assessments and reporting on: –Children’s progress –Quality of teaching/learning opportunities  “Vertical” teams of teachers/managers to: –Review assessment information –Enrich learning experiences and teaching strategies  Joint professional development

40. Part IV Differing Viewpoints on Using Child Assessment Data for Local Agency Accountability

41. Differing Viewpoints on Using Child Assessment Data on Local Agencies  Task Force reached an easy consensus on: –Infrastructure components –Four assessment/improvement approaches –Linking of pre-kindergarten through grade 3

42. Differing Viewpoints on Using Child Assessment Data on Local Agencies  Some members said this wasn’t enough.  Needed more information: –How well are children in local agencies learning? –How does what they are learning compare with state standards? –Are some local agencies unusually effective in fostering learning progress?

43.  To do this, some members of the Task Force advocated collecting and reporting data on all children's performance and to distill it so it could be used by local agencies for planning and improvement.  They recognized that we can have children in good programs who don’t do well, and they contend that program quality is NOT a proxy for child outcomes. Differing Viewpoints on Using Child Assessment Data on Local Agencies

44.  These Task Force members recognized the huge challenges associated with this: 1.When we disaggregate data, it will be held to a higher standard. 2.The field needs better tools 3.The field needs well-trained reliable data collectors 4.The field needs systematic ways of collecting, cleaning, and analyzing data.

45. Differing Viewpoints on Using Child Assessment Data on Local Agencies  Most importantly, these Task Force members noted, we need to be very careful not to misuse data: –Not to teach to the test. –Not to use data to label or place children. –Not to defend programs.

46. Differing Viewpoints on Using Child Assessment Data on Local Agencies  The Task Force members who support the use of agency-level child assessment data acknowledged the challenges, but contended, over time, states would benefit by using both child and program quality data.  They recommended nine crucial safeguards.

47. Differing Viewpoints on Using Child Assessment Data on Local Agencies 1.Collect enriched data on the programs, including teacher-child interactions, nature of curriculum interventions, and nature of teacher training. 2.Collect detailed data on the children, including prior ECE experience and primary language. 3.Impel the state to align its assessments to all domains in standards.

48. Differing Viewpoints on Using Child Assessment Data on Local Agencies 4.Select approaches that include direct observation. 5.Base reporting on progress. 6.Collect data at more than one point in time. 7.Stagger strategies for data collection: –One third in each of 3 years –Priority accorded to low-quality programs

49. Differing Viewpoints on Using Child Assessment Data on Local Agencies 8.Do not, under any circumstances, report any data on individual children. 9.Use the data collected to make program improvements. SAFEGUARDS #8 AND #9 ARE, BY FAR, THE MOST IMPORTANT ONES RECOMMENDED.

50. Differing Viewpoints on Using Child Assessment Data on Local Agencies While there was consensus on the safeguards recommended, the Task Force did not reach consensus on offering this a separate approach.

51. Part V Action Steps

52. Action Steps: Legislatures  Provide adequate funding for programs and infrastructure for ongoing assessments and program improvements.

53. Action Steps: State Agencies  Develop a strategic plan for early childhood accountability and program improvement system.  Create a robust, positive, and rigorous culture for early childhood accountability efforts.  Enable local Pre-K–3 partnerships.

54. Action Steps: Federal Government  “Harmonize” information systems.  Fund research and development for better assessment tools.  Support ongoing longitudinal research on children and programs.

55. Action Steps: Local Agencies  Create opportunities for teachers and managers to review assessments and enhance children’s learning opportunities.  Initiate dialogue with local school districts.

56. The Benefits  For Children: Enhanced learning opportunities and improved outcomes  For Legislators: Better data to guide state policies and investments  For Teachers/Directors: Targeted and well-resourced professional development and program improvement efforts  For the Early Childhood Profession: Enhanced public awareness and credibility

57. Action Steps: Follow-Up  The Pew Charitable Trusts approved a grant to the Council of Chief State School Officers (CCSSO) to disseminate the report and work with selected states in implementing recommendations.  Project will build on CCSSO efforts to improve state education standards, assessments, and data systems for all children—preschool through high school and beyond.

58. Part VI Implications

59. Implications  Acknowledge the intensity of sentiment regarding potential misuse of child assessment data.  Attend to the infrastructure and the resources to do the collection.  Think systemically.

60. Implications  Need more work on tools that are aligned with standards and curriculum.  Need more work on hard-to-capture domains in assessment construction.  Need far more work on the considerations regarding English Language Learners and children with disabilities.

61. Implications  Need states to focus on effective accountability systems.  Such accountability systems should be designed to meet the state’s needs.  Such accountability systems should take into consideration state resources (human, technical, and fiscal).  Every state should develop and begin implementing an ECE accountability system that provides data and respects families, teachers, children, and child development principles.