1. Welcome to LIB5080 The School Library Media Program

2. LIB 5080 The School Library Media Program Today’s Learning IMPACT Model – Guests in AETZ Updates Cross Program Collaboration Program Planning and Evaluation

3. Cross Program Collaboration Briefly describe the following: Your CPC group and how you went about the work What did you learn … –About 21 st Century learning, schools, etc –About other programs/positions in K-16 education –About group processes

4. Planning & Evaluating Media Programs Logic Models - Providing focus for planning and evaluating school library media programs Based on SERVE Center’s Capacity for Applying Project Evaluation (CAPE) available online at http://www.serve.org/Evaluation/Capacity/

5. Why Plan and Evaluate? Improve programs Increase program effectiveness Maintain funding Generate additional funding Enhance communication with stakeholders Improve useful data collection Inform decision makers Increase staff and public awareness Assess needs Set program priorities Focus and prioritize data needs O’Sullivan, R. (2001). Utilization-focused evaluation. Presentation at the Siberian Evaluation Institute, Novosibirsk, Russia.

6. What is Evaluation? Evaluation is the systematic investigation of merit or worth. –Thoughtful, intentional, purposeful –Done for clear reasons –Organized and deliberate –Appraisal or judgment based on the best evidence available Guskey, T. (2005). Taking a second look at accountability: Strong evidence reflecting the benefits of professional development is more important than ever before. Journal of Staff Development. 26(1).

7. What is Formative Evaluation? "When the cook tastes the soup, that's formative evaluation. When the guests taste the soup that is summative evaluation." ~ Bob Stake Cited in Patton, Michael Quinn (1997). Utilization-Focused Evaluation: The New Century Text. Edition 3. Thousand Oaks, CA: Sage Publishers, p. 69.

8. What is a Logic Model? A logic model is a graphic representation of the theory or logic behind a project or program. A logic model shows the forces that shape the program as well as the relationships among the key components. Coffman, J. (1999). Learning from Logic Models. Cambridge, MA: Harvard Family Research Project.

9. Logic Models Help to articulate the key elements of a program Provide focus for program evaluation Can lead to program efficiency and effectiveness Promote stakeholder buy-in by helping clarify how the project works

10. Frameworks and Examples Harvard Family Research Council Kellogg Foundation National Science Foundation CAPE

11. Coffman, J. (1999). Learning from Logic Models. Cambridge, MA: Harvard Family Research Project. Retrieved April 2, 2007 from http://www.gse.harvard.edu/%7Ehfrp/pubs/onlinepubs/rrb/learning.html Harvard Family Research Project

12. Kellogg Foundation W.K. Kellogg Foundation. (2001). Logic Model Development Guide. Battle Creek, MI: W.K. Kellogg Foundation. Retrieved April 2, 2007 from http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf

13. National Science Foundation Frechtling Westat, J. (2002). The 2002 User-Friendly Handbook for Project Evaluation. Arlington, VA: National Science Foundation. Retrieved April 2, 2007 from http://www.nsf.gov/pubs/2002/nsf02057/nsf02057_3.pdf

15. Key Elements of a Logic Model Outcomes, Impacts, or Goals –Goals tell the ultimate purpose(s) of the project or program. They answer the question “What difference will the project or program make in the long run?” –In education, goals usually say what impact the program will have on learners and learning.

16. Key Elements of a Logic Model Objectives, Intermediate Outcomes, or Short-term Goals –Focus on program outcomes –Specify what the program will accomplish –Answer the question “How will learning, (teaching, the school, etc.) be improved as a result of this program?” –Describe what you hope will happen as a result of the program –Are measurable

17. Key Elements of a Logic Model Strategies –Identify the major components of the program –Are larger in scope than simple program activities –Provide details about how the program funds are spent –Describe what the program does –Are measurable

18. Key Elements of a Logic Model Inputs or Contexts –Are the contexts or conditions that influence program activities and strategies. –Should be considered when planning program activities, e.g., policies, plans, needs assessment data.

19. STEM-ICT 3D Strategies Objectives/ Goals/ Intermediate Outcomes Long term Outcomes Middle and high school students are motivated to pursue ICT careers. Teachers mentor others. Design and deliver 3D curricular materials to diverse students. Students develop virtual solutions to real world problems. Students are prepared with the skills necessary to succeed in STEM. Students teach teachers 3D technological skills. Students mentor others. Students collaborate via 3D virtual worlds. Design and deliver teacher professional development on pedagogical strategies using virtual 3D worlds. Teachers are engaged in continuous STEM learning. Teachers build and sustain ongoing relationships with colleagues through interacting in the 3-D world about topics related to delivering content through the 3-D world. Teachers engage students in the use of 3D world tools to explore solutions to real world problems. Students pursue STEM careers.

20. Logic Model Exercise Inside the packet are index cards labeled with components of a project. Your tasks are as follows: –organize these cards into the appropriate columns of the logic model template shown on the packet; –tape the cards on the wall chart paper provided; –connect the cards using marker or tape to make a logic model; –review the connections between objectives and strategies using “if –then”.

21. Healthy Living Logic Model

22. Reflection Describe your process for creating the logic model. What worked? What didn’t?

23. Discussion What could be the advantages for creating and using a logic model for the school library media program? What might be the barriers or obstacles that would have to be overcome?

24. Potential Benefits Logic models Provide clues to why the program does or does not work Can provide evidence for attributing outcomes Help improve programs –Expose faulty thinking –Develop a common language and understanding of a program Rogers, P., Petrosino, A., Huebner, T., & Hacsi, T. (2000). Program theory evaluation: Practice, promise, and problems. In Rogers, P., Hacsi, T., Petrosino, A., & Huebner, T. (Eds.) Program Theory in Evaluation: Challenges and Opportunities (pp. 5-13), San Francisco, CA: Jossey-Bass.

25. Used with permission ©ScienceCartoonPlus.com

26. Advantages to Using Logic Models Teachers became more engaged and reported greater program awareness (Finnan & Davis, 1995*) Staff understood the program better (Beckford, 1998*) Teachers were more actively involved in implementation and altered practices as a result of the evaluation (Darling, 1998*) * As cited in Huebner, T. (2000). Theory-based evaluation: Gaining a shared understanding between school staff and evaluators. In Rogers, P., Hacsi, T., Petrosino, A., & Huebner, T. (Eds.) Program Theory in Evaluation: Challenges and Opportunities (pp. 79-89), San Francisco, CA: Jossey-Bass.

27. Logic Model Work in groups Develop a logic model for a school library media program

28. Sharing Logic Models Explain your logic What did you discover as you were creating your logic model?

29. What Happens Next? Define evaluation questions Determine benchmarks Select methods and measures Conduct the evaluation Analyze and draw inferences from data Modify the program

30. Writing Evaluation Questions Consider the purpose of your evaluation, what do you really care about? –Quality of implementation –Impact

31. Writing Evaluation Questions Evaluation questions about strategies ask how well the strategy is implemented. Evaluation questions about objectives ask about the impact of implementing the strategy.

32. Writing Evaluation Questions Try to avoid simple yes or no questions. Consider quantity questions (e.g., “how many,” “how much,” or “how often”). Consider quality questions (e.g., “how well,” “how effectively,” or “in what ways”). Be sure questions are self-contained – avoid putting implementation questions in terms of impacts.

33. Healthy Living Logic Model

34. Writing Evaluation Questions Examples –Implementation questions How many hours of sleep am I getting each week? (quantity) How soundly am I sleeping? (quality) –Impact question How much weight have I lost? (quantity) How has my stress level changed? (quality)

35. Writing Evaluation Questions Using your logic model, select one strategy and brainstorm implementation questions that you might ask, to evaluate how well it is being accomplished. Select one objective and brainstorm impact questions that you might ask, to evaluate how well it is being met. Share and discuss.

36. Strategy: Implementation Questions Data Sources/Analysis Methods Benchmarks Decisions What do you need to find out?How will you find out?What are your intermediate targets? What decisions will you make based on the data? Writing Evaluation Questions 1.Insert a strategy from your logic model into the worksheet 1.

37. Strategy: Implementation Questions Data Sources/Analysis Methods Benchmarks Decisions What do you need to find out?How will you find out?What are your intermediate targets? What decisions will you make based on the data? Writing Evaluation Questions 2.Write at least one implementation question for the strategy. 2.

38. Writing Evaluation Questions 1.Insert an objective from your logic model into the worksheet Objective: Impact Questions Data Sources/Analysis Methods Benchmarks Decisions What do you need to find out?How will you find out?What are your intermediate targets? What decisions will you make based on the data? 1.

39. Writing Evaluation Questions 2.Write at least one impact question for the objective. Objective: Impact Questions Data Sources/Analysis Methods Benchmarks Decisions What do you need to find out?How will you find out?What are your intermediate targets? What decisions will you make based on the data? 2.

40. Data Sources/Analysis Methods What do we mean by data? –“Data” are pieces of information. –Common understanding is that data are numbers – quantitative data or quantities of things. –Data may also be qualitative – about qualities that are not easily measured in numbers. –Both qualitative and quantitative data should be analyzed. – Data are what we use to conduct formative evaluation of projects.

41. What data do you need? –Your data needs are dictated by your evaluation questions. –Ask yourself, “What information do I need to answer this question?” –Gather all of the data you need, and only the data you need, to answer your questions. Data Sources/Analysis Methods

42. Where do you get the data you need? –Check whether there is data already available to you that might help answer your questions. –Determine the data sources you might use to meet remaining data needs. –Note that data sources are not data. Example – teachers’ lessons plans are a rich data source, but it is necessary to do something with them before you have actual data. Data Sources/Analysis Methods

43. What are analysis methods? –The approaches and tools used to pull data out of data sources –It may be helpful to think in four steps: Collecting data Storing and organizing data Analyzing data Interpreting data –Qualitative data may be quantified. Data Sources/Analysis Methods

44. What is “informal data?” –For an evaluation to be purposeful and systematic, data must be relatively formal, collected, stored and organized, and analyzed with some degree of rigor. –Data not handled in this way may be thought of as “informal.” –The trick is making informal data formal. Data Sources/Analysis Methods

45. Strategy: Implementation Questions Data Sources/Analysis Methods Benchmarks Decisions What do you need to find out?How will you find out?What are your intermediate targets? What decisions will you make based on the data? 1.Review each evaluation question you wrote 1. Data Sources/Analysis Methods

46. Strategy: Implementation Questions Data Sources/Analysis Methods Benchmarks Decisions What do you need to find out?How will you find out?What are your intermediate targets? What decisions will you make based on the data? 2.Identify the data source to answer the question. 2. Data Sources/Analysis Methods

47. Benchmarks –Are your targets. –Define levels of success. –Help you stop periodically to examine progress. Specifying Benchmarks

48. Look at the Healthy Living logic model. Listen to the possible benchmarks. What do you notice? –How are data used to specify benchmarks? –Are the expectations reasonable? –Do you think all the benchmarks will be met? –Is it okay if the benchmarks aren’t met?

49. Healthy Living Logic Model

50. Specifying Benchmarks Objective: Lose weight. Benchmarking –Baseline data current weight (June 2007) = 165 lbs. –Set target weight 1 year from now (June 2008) = 145 lbs. –Specify benchmarks periodically stop and examine progress.

51. Specifying Benchmarks Specify benchmarks – periodically stop and examine progress. By September, I will weigh 160 lbs. By December, I will weigh 155 lbs. By March, I will weigh 150 lbs. By June, I will weigh 145 lbs.

52. Benchmarks – one possible format (there are others…) Specifying Benchmarks Possible Format: How many of who (or what) is going to do (or be) what by when? For Example: 70% of teachers will complete a collaborative unit with the media coordinator by mid-year.

53. Strategy: Implementation Questions Data Sources/Analysis Methods Benchmarks Decisions What do you need to find out?How will you find out?What are your intermediate targets? What decisions will you make based on the data? 1.Consider the questions and data sources you have written 1. Specifying Benchmarks

54. Strategy: Implementation Questions Data Sources/Analysis Methods Benchmarks Decisions What do you need to find out?How will you find out?What are your intermediate targets? What decisions will you make based on the data? 2.Specify one or more benchmark for each data source. 2.1. Specifying Benchmarks

55. What baseline data do you have? What if you don’t have baseline data? In order to make adjustments you will want to collect, analyze, and compare data to benchmarks periodically e.g. bi-monthly, monthly, each 9-weeks, at the end of each semester

56. Specifying Benchmarks Be realistic and reasonable. Consider how much time it will take to collect and analyze the data.

57. What decisions might you make if evaluation findings show… –Problems with implementation? –Good quality implementation? –Positive impacts (benchmarks reached)? –Problems with impacts (benchmarks not reached)? Who will make the decisions? Who will the decisions effect? Making Decisions

58. Strategy: Implementation Questions Data Sources/Analysis Methods Benchmarks Decisions What do you need to find out?How will you find out?What are your intermediate targets? What decisions will you make based on the data? 1.Use the questions, data sources, and benchmarks you have written …. 1. Making Decisions

59. Strategy: Implementation Questions Data Sources/Analysis Methods Benchmarks Decisions What do you need to find out?How will you find out?What are your intermediate targets? What decisions will you make based on the data? 2.What decisions will you make based on your findings? 1. Making Decisions 2.

60. Evaluation Plan Assignment Work in your groups Revise your logic model Taking the role of a school MTAC, select one objective the school will focus on for the school year. Complete an Objective worksheet for that objective. Complete a Strategy worksheet for all strategies associated with that objective.

61. Homework Work on evaluation plan Finish Special Topics Project