1. 2010 ECO Conference How Will I Know? How Will You Know? Implementation Fidelity Surveys: 2 States’ Experiences

2. 2010 ECO Conference How Will I Know? A Tale of Two States Ohio 619 6 th Largest State 612 School Districts 88 Counties –Education Service Centers –County Bds of DD 16 State Support Teams- regionalized TA/PD World’s largest basket Minnesota C & 619 Xx School Districts 8? Counties Regionalized prof. development & TA Worlds largest ball of twine

3. 2010 ECO Conference How Will I Know? A Tale of Two States Ohio 619 Has used COSF since 2007 Locals report to state 2x per year (fall & spring) State does conversion of COSF scores to progress categories Minnesota C & 619 Started using COSF on April 1, 2006 Ongoing data collection locally Reported to state annually Moving to “real time” collection in future

4. 2010 ECO Conference How Will I Know? A Tale of Two States Ohio 619 Two institutes… Teacher reference sheet Data collection protocols Data checks in the state data system –Funding impact Minnesota C & 619 Early emphasis on data quality as precursor to date use. Pattern checking tool COSF data linked to demographic & service variables plus Part C family outcome data

5. 2010 ECO Conference How Will I Know? Ohio Assessment Policies State statute requires programs to document child progress using research-based indicators prescribed by the Department and report results annually Data/Reporting Videoconferences in September –Policies provided –Reporting dates calendar

6. 2010 ECO Conference How Will I Know? Ohio Assessment Policies Multiple Sources Required Two Sources are consistent statewide for funded programs Preschool Special Education Early Childhood Entitlement Grant Get It! Got It! Go! & Ages/Stages Questionnaire: Social-Emotional (completed with parent) Early Childhood Outcomes Summary Form –Preschool Special Education: 7 point scale Completed with parent and team providing services –9 point scale with ECO Center for nondisabled peers

7. 2010 ECO Conference How Will I Know? Minnesota Assessment Policies Multiple sources of information inform COSF ratings for Part C & Preschool Special Education entry/exit ratings One source must be a tool appropriate for ongoing assessment cross-walked by ECO State Pre-K program has new statutory mandate to conduct ongoing assessment Procedural updates provided annually. E-module release planned for mid-August

8. 2010 ECO Conference How Will I Know? How Will You Know? Child Outcome Summary Form? Decision Tree ? Publisher’s Online Assessment? Team Process? Parent Engagement? Data Entry Protocol? Vigilant Administrative Oversight? VALID DATA? Summary Statements ? Local Reporting?

9. 2010 ECO Conference How Will I Know?

10. 2010 ECO Conference How Will I Know? Fidelity Survey What do we want to know? Who will we need to ask? How will we ask them? How will we analyze the data How will we turn analysis into action? How will we know that the action has addressed the identified issues?

11. 2010 ECO Conference How Will I Know? What Did We Want to Know? Are all contributors to COSF data and Child Outcome Measurement System implementing process with fidelity? If not….who? If not…which component or components? If not…why not? If not…what will it take?

12. 2010 ECO Conference How Will I Know? What Do We Need to Know? Minnesota used its previously developed Child Outcome Summary Process: Quality Review Form as the basis for survey questions. Outstanding assistance from NECTAC & ECO in formulating questions/responses/branching Electronic survey distributed to local providers and administrators of Early Childhood Special Education (C & 619).

13. 2010 ECO Conference How Will I Know? What Do We Need to Know? Ohio has ongoing contract with ECO Center ECO assisted in survey development in partnership with Ohio EC Quality Network Survey sent to Preschool SpED Supervisors ECO conducted Data Quality and Analysis Institutes

14. 2010 ECO Conference How Will I Know? Key Questions: Respondent Demographics Professional license(s) held Length of time in ECSE ECSE “role” during 2008-09 school year –Service coordinator –Direct provider of ECSE or related service –Regular Early Childhood Educator –Administrator Age Level Served (e.g., 0-2; 3-5; 0-5; etc…)

15. 2010 ECO Conference How Will I Know? Key Questions: Respondent COSF Experience How many COSF’s did the respondent contribute to in 2008-09? Adherence to Procedures –Timely –Completed during team meeting –Parent input –Multiple Sources of Data

16. 2010 ECO Conference How Will I Know? MN Fidelity Survey—November 2009 Zoomerang Sent from ECO through ECSE List Serve 617 valid respondents –Administrators (39) –ECSE Teachers (392) –Speech/Language (164) –OT/PT/SW (146) –Other (45)

17. 2010 ECO Conference How Will I Know?

18. 2010 ECO Conference How Will I Know?

19. 2010 ECO Conference How Will I Know? Challenges to Face to Face Meeting TIME (76%) Schedules of team members (66%) Very small team/only 1 provider (33%) Don’t value the process (25%) –Too subjective –Too redundant Other (7%)

20. 2010 ECO Conference How Will I Know?

21. 2010 ECO Conference How Will I Know?

22. 2010 ECO Conference How Will I Know?

23. 2010 ECO Conference How Will I Know? Identified Challenges to Using Decision Tree Time Can’t always find it when needed “What’s a Decision Tree?” “I’m a related service provider and I never received training.”

24. 2010 ECO Conference How Will I Know?

25. 2010 ECO Conference How Will I Know? Identified Challenges to using Crosswalks Consistently Time More training is needed “I’m not sure what a crosswalk is.” “We don’t always know where to find the crosswalks.” Not all tools have been crosswalked.

26. 2010 ECO Conference How Will I Know? Actionable Items Reduce duplication of documentation Increase use of Decision Tree Create defensible procedure for “virtual teaming” Improve access to “just in time” professional development

27. 2010 ECO Conference How Will I Know?

28. 2010 ECO Conference How Will I Know? COSF’s 7-Point Scale 1 2 3 4 5 6 7 Age Expected (AE) Functioning AE with Qualitative Concern Immediate Foundational Skills Foundational Skills

29. 2010 ECO Conference How Will I Know?

30. 2010 ECO Conference How Will I Know? Articulate! E-Module on Child Outcomes Measurement System –Overview –Tabs for specific roles/functions within COMS –State and local pattern-checking Scheduled for release mid-August

31. 2010 ECO Conference How Will I Know? ECO Implementation Survey Purpose of the Survey: Identify challenging practices for local districts and personnel –Areas with approximately 20% or more of the respondents reporting somewhat, very or extremely challenging Implementation of ECOSF in 2007-2008 Preparation since August 2007 Survey May 2010

32. 2010 ECO Conference How Will I Know? ODE Public Preschool Preschool Special Education Even Start Kindergarten Oversight Program Licensing Ready School Initiative Head Start Collaboration Office

33. 2010 ECO Conference How Will I Know? Lessons Learned Multiple sources of information –Required sources Get It! Got It! Go! (outcome 2) Ages and Stages Questionnaire: Social-Emotional (outcomes 1 and 3) –Record types of sources Family Observations Assessments –Ongoing Assessment System

34. 2010 ECO Conference How Will I Know? Inputs Family information on child functioning –Received in team meeting –Collected separately – Incorporated into assessment(s) –Not included Varied assessments –Curriculum based –Criterion referenced –Standardized –Checklists –Work sampling –Portfolios –Interview Varied Observations –Varied social settings Individual Small group Large group –Different interactions/dynamics –Different degrees of choice –Different levels of competence –Different times of day Transitions Circle time Entry Exit Bus Free choice –Child preferences

35. 2010 ECO Conference How Will I Know? Synthesizing Results from Multiple Sources

36. 2010 ECO Conference How Will I Know? Lessons Learned Comparable to same-aged peers Use of Decision Tree Fidelity of Process Data Issues

37. 2010 ECO Conference How Will I Know? Understanding & Applying Knowledge of Functioning of Non-disabled Peers

38. 2010 ECO Conference How Will I Know? Use of the Decision Tree

39. 2010 ECO Conference How Will I Know? Observe Process for Fidelity

40. 2010 ECO Conference How Will I Know? Review to Ensure Fidelity

41. 2010 ECO Conference How Will I Know? Accuracy Issues

42. 2010 ECO Conference How Will I Know? Lessons Learned Core team input –Preschool special education teacher –General education teacher –Parent –Related services personnel Involving parent

43. 2010 ECO Conference How Will I Know? Reasons for Completion without all Members Contributing

44. 2010 ECO Conference How Will I Know? Involving Parents

45. 2010 ECO Conference How Will I Know? Regionally Based Local Needs Professional Development Technical Assistance Assessment Training Observation and Mentoring ELLCO Observations and TA Regional Leadership Meetings

46. 2010 ECO Conference How Will I Know? Regionally Based Local Needs Teacher Leader Project Preschool Literacy Core Curriculum (CORE) SIRI: Pre K/K State Institutes for Reading Instruction ELLCO Observations and TA English Language Learners

48. 2010 ECO Conference How Will I Know? PD Connection IVDL/Higher Education Faculty ELLS Preschool Literacy Core Curriculum (CORE) SIRI: Pre K/K State Institutes for Reading Instruction Teacher Leaders Ohio Early Learning Content Standards –Integrated Curriculum –The IEP and the Early Childhood Curriculum –Mathematics –Science –Social Studies

49. 2010 ECO Conference How Will I Know? Ohio Resource Center for Mathematics, Science ad Reading Resources for early childhood –Early Childhood Bookshelf –Standards-aligned and Peer Reviewed Lessons –Early Childhood Building Blocks- Best Practice Articles http://www.ohiorc.org/

50. 2010 ECO Conference How Will I Know? So What Now? Webinars –Data Accuracy and Analysis Newbies Advanced –Summary Form Source Citation Form –Newbies –Refresher Evidence Summary Form –Newbies

51. 2010 ECO Conference How Will I Know? So What Now? Continue state research efforts –ECO analysis of GGG, ASQSE and COSF –ELLCO –KRA-L –Local readiness projects

52. 2010 ECO Conference How Will I Know? Get It, Got It, Go! (GGG) FY09 8,395 ECE children (79%) had pre- and post measures 11,196 PSE children (70%) had pre-and post measures Children served in ECE and PSE classrooms experienced statistically significant growth from fall to spring

53. 2010 ECO Conference How Will I Know? OEL&SR Program Participant Performance on KRA-L Fall 2007 KRA-L scores ECE participants had significantly higher KRA-L scores than all other K students. PSE participants who are nondisabled in K have higher KRA-L scores than all other K students who are nondisabled. PSE students who are SWD in K are comparable to other K students who are SWD.

54. 2010 ECO Conference How Will I Know? More…. New Web pageWeb Outcomes brochures Outcomes reference card for families Outcomes reference sheet for teachers and related services personnel

55. 2010 ECO Conference How Will I Know? Teacher Reference: PROCESS Implementation

56. 2010 ECO Conference How Will I Know?