1. Copyright © 2010 American Institutes for Research All rights reserved. Evaluating A Multi-Site 21 st Century Learning Program: Strategies and Results Mary Nistler, Manolya Tanyu, Jonathan Margolin November 2010

2. 2 Learning Point Associates Learning Point Associates is an affiliate of American Institutes for Research. Research Evaluation Policy Professional Development

3. 3 Purpose Present a method for organizing messy project information within the context of a broadly defined initiative.

4. 4 Schools of the Future A five-year initiative 18 diverse private schools/school partnerships Non-prescriptive design and implementation requirements High level of support from funder

5. 5 Evaluation Problem 1 Diversity of Participating Schools Size of school Educational orientation Grades served Setting

6. 6 Evaluation Problem 2 Broad Parameters of the Initiative Introduce 21 st century skills into the classroom Introduce technology into the school and classroom Establish professional learning communities

7. 7 Evaluation Problem 3 Expectations for Change Expansion Changes in implementation plans Changes in design

8. 8 What We Needed to Do Organize the information about the projects so that we and our client could understand the projects in a meaningful way.

9. 9 What We Did Sort Study Label

10. 10 To Sort the Information Identified key project dimensions Used the dimensions to develop a conceptual framework Used the framework to develop project templates to describe each project

11. 11 General Framework B. School Settings & Characteristics A. Direct Project Support 1.Project Design 2.Implementation Capacity 3. School Outcomes 4. 21 st Century Skills

12. 12 Project Template Dimensions Project design (goals and purpose, curriculum, instruction, assessment, planned scope) Implementation (timing, scope, PD, PLCs) Project setting (size, locale, affiliation, grades served)

13. 13 Study and Label Examine each dimension for each project to look for commonalities

14. 14 Project Purpose Example 1Example 2Example 3Example 4  Create PLCs  PLCs to focus on supporting instruction  Existing strategies need strengthening  Train all teachers on ed technology  Train all teachers on RIP  100% of students use digital tools for RIP research  Add digital story-telling units to improve student media and tech literacy  Transform and enhance technology use in the school  Support intercultural learning and understanding

15. 15 Types of Purpose Support – strengthen existing model. The instructional model will not change, but be strengthened through the project. Change – make school-wide and notable changes to instruction. All classrooms will look different. Augment – add a course or distinct instructional component.

16. 16 Implementation Approach Early adopters – volunteers adopt first; communication encourages more early adopters. Preparatory year – first year is spent getting ready for full implementation. Immediate implementation – implementation begins right away, with plans driven by school.

17. 17 Summary Table SchoolSOTF RoleImplement. Approach Tech Use PLCAssessmentEvaluation School A SupportEarly adopterHighSchool- wide School- based Proposed School B AugmentPreparatoryModerateSubject/ Grade level FormalNot proposed School C ChangeImmediateLow

18. 18 Value to the Client Client Goal Understand each funded project Identify strengths and weaknesses Identify ways to support the initiative Communicate about the project

19. Copyright © 2010 American Institutes for Research All rights reserved. Online Communities of Learners: Evaluating Participation and Purpose

20. Purpose Present a method for evaluating the usefulness of an online forum for discussing the Schools of the Future projects.

21. 21 Fostering a Community of Learners Community of Learners: educators working together on shared goals Organized by school level and type Face-to-face quarterly meetings Online forum for discussion and sharing (Ning)

22. 22 Ning: A Riot of Data! Free, online platform (like moodle, Facebook) Members and moderator Groups Discussion threads Collection of resources (video, pictures, documents)

23. 23 Community of Learners on Ning

24. 24 The Evaluation Question What is the usefulness of the online communities of learners as indicated by: 1.Level of participation 2.Topics being discussed

25. 25 The Evaluation Challenge Bringing order to the riot of information Deciding what is important and useful Decision: Focus on three discussion groups.

26. 26 Evaluating Participation How to measure participation? Total number of posts Proportion of members posting Variation over time Key decision: exclude “me toos”

27. 27 Presenting the Participation Data

28. 28 Presenting the Participation Data Group AGroup BGroup C Number of members 393335 Number and percent of members who made posts 13 (33%)7 (21%)7 (20%) Number of discussions threads posted 233 Total number of posts 385562

29. 29 Number of Posts by Month

30. 30 Purpose of Ning Use What are members using Ning for? Our approach: Look at purpose and topic area. Critical decision: What is level of analysis—the post or the discussion? We chose the discussion.

31. 31 Purpose of Discussion Seeking or sharing advice/opinion What is the role of IB/AP? Seeking or sharing resource Curriculum or assessment of 21 c. skills Seeking or sharing factual information What are school cell phone policies? Other

32. 32 Frequency of Discussion Purposes

33. 33 Topics of Discussion Curriculum Instruction Assessment Technology use 21st century skills Logistics of school Other

34. 34 Frequency of Discussion Topics

35. 35 Answers to Evaluation Questions Participation is limited to a handful of active individuals The discussions primarily are used to seek advice or express opinions Most posts happen under planned rather than spontaneous discussions

36. 36 Conclusions Focus the scope of data analysis Decide what level of analysis will be most useful Separate wheat and chaff

37. 37 Final Thoughts The value—and challenge—of this evaluation was to add structure and meaning to messy data.

38. 38 Manolya Tanyu P: 312-288-7611 E-Mail: mtanyu@air.orgmtanyu@air.org Mary Nistler P: 312-288-7629 E-Mail: mnistler@air.orgmnistler@air.org Jonathan Margolin P: 312-288-7632 E-Mail: jmargolin@air.orgjmargolin@air.org 20 N. Wacker St, Suite 1200 Chicago, IL 60606 General Information: 800-356-2735 Website: www.learningpt.org