1. CLIMATE LITERACY AND ENERGY AWARENESS NETWORK PATHWAY CLIMATE LITERACY AND ENERGY AWARENESS NETWORK PATHWAY SUMMARY OF A RIGOROUS REVIEW PROCESS CLN Webinar December 7, 2010

2. Outline  CLEAN project and CLEAN collection  Review Process  Results of first review round and gap analysis  Conclusions and Outlook

3. CLEAN Pathway Project Goals  Stewarding a collection of excellent teaching resources on climate and energy science  Professional and Community Development  Online communities  Facilitate the Climate Literacy Network

4. CLEAN Pathway: Collection  500 excellent digital teaching resources addressing climate science or energy awareness for grades 6-16  Resources scientifically and pedagogically reviewed  Annotations reflect reviewer comments  Resources aligned with  Climate Literacy: Essential Principles of Climate Science  Energy Awareness Principles  National Science Education Standards  AAAS Project 2061Benchmarks for Science Literacy  NAAEE Excellence in Environmental Edu. Guidelines for Learning

5. Framework for collection  Energy Awareness Principles (newly developed) A. Earth System Energy B. Human Energy Sources C. Impacts of Energy Use D. Energy Distribution/Costs E. Energy Access and Equity F. Decarbonization Challenges http://cleanet.org/clean/literacy/energy.html  Every activity included in collection has to address either  Climate Literacy: Essential Principles of Climate Science http://www.climatescience.gov/Library/Literacy/

6. CLEAN review process  Phase 1 of CLEAN focused on teaching activities  Following phases: broadening of scope to videos, visualizations, lab activities, background materials etc.

7. Developing Review Criteria  Review criteria specific to teaching activities  Informed by NSDL and SERC guidelines, Merlot criteria, DLESE, Climate Change collection scorecard  Review of e-learning materials requires additional considerations (multi-media elements, navigation etc.)  Climate and energy science evolving science – faster turn-around and need more media to convey

8. Developing Review Criteria  Initially long list of possible criteria: convergence in phrasing of questions during 7 test review rounds  Test reviews: good agreement for good resources, wide spread in answers for low quality resources  Weekly telecons for resource collectors

9. What is an excellent activity? Definition Teaching Activity: Relatively brief set of instructional materials that is presented as a whole, where all the parts and ideas are linked and part of the same activity Educator should be excited to find this activity when searching for teaching materials.

10. Avenues into collection  Existing resource pools found through online search  Resources suggested by public through online form  Targeted search after gap analysis  Collaboration with projects that submit resources directly to CLEAN team (iterations)

12. Informal vetting  Resource pools assigned to team of 9 collectors  Scanning of resource pools and informal triage  Informal triage: Does this resource meet the CLEAN criteria?

13. Review questionnaire  Initial Vetting  Review  Scientific accuracy  Pedagogic effectiveness  Technical quality / Ease of use → 6-12 questions for each category, overall rating in rubric format, comment box for annotations → Questions help to consider all relevant aspects for each category and lead to overall rating → No quantitative, only qualitative recommendation (low – medium – high priority)

14. Formal vetting step  Basic check for relevance to CLEAN project  Decision: “Will resource likely pass the review?”  Recommendation: Move on to initial review or put in “holding tank”

15. Review: Scientific accuracy  Considerations for initial science review  Solid, current science  Original data cited and data from a quality source  Attribution  Valid concepts  Supporting references

16. Science: Use of digitally available scientific data to teach concept

17. Science: Simplified models of complex Earth system to teach concept

18. Review: Pedagogic effectiveness  Considerations for pedagogic review  Learning objectives  Accommodates diverse learners (learning styles, language, cultural diversity)  Prerequisite skills and understandings  Assessment strategies  Engaging for students in subject and approach  Requires independent/inquiry-based thinking

19. Pedagogy: Inquiry-based labs or experiments

20. Pedagogy: Pen-and-paper graphing exercise

21. Review: Technical quality/ Usability  Considerations for technical/usability review  Ready for use, stands on its own  Clear presentation of content  Software/tools/resources commonly found in classroom  Amount of necessary guidance for students by instructor  Offers comprehensive guide for instructor  Digitally available resource

22. Ease of use: Printable PDFs with background info and instructions

23. Ease of use: Clear steps-by-step instruction

24. Expert science review  External expert with PhD in relevant field reviews scientific quality and accuracy of resource  Activity already passed lower level science review (75 % of resources that passed the CLEAN review were rated scientifically excellent by experts)  Limitation/Challenge:  Grade-level appropriate science  Difficult to find scientists with enough time

25. Panel review  Based on NSF-panel review system and AccessData Workshops  Panel provides necessary range of expertise  Teams of 4 educators and scientists review each teaching material based on prior reviews, final decision about inclusion in collection  Comments of all reviewers are compiled into annotation which includes teaching tips

26. Annotations  All reviewer comments, suggestions and tips are combined in notes to users (annotations)  Annotation draft reviewed during review panels, final clean-up during cataloging process  Annotations add considerable value, insight from scientists or experienced educators

27. Demo on CLEAN Website http://www.cleanet.org

28. Numbers first review cycle  Informal vetting of ~5000 teaching activities  ~200 resources passed first review with medium or high priority > forwarded to second review  142 resources passed second review with medium or high rating > forwarded to panel review  94 resources passed panel review (18 passed on to editorial board)  5-7 different people reviewed each resource

29. Gap Analysis  Holes in collection are apparent – inform targeted search and hopefully future solicitations Total # of activities # of concepts with no match # of concepts with ≤ 3 matches Guiding Principle2225 of 7 EP 11723 of 5 EP 22003 of 6 EP 31523 of 5 EP 42112 of 7 EP 52713 of 5 EP 6815 of 5 EP 71614 of 6 Energy Awareness3401 of 6

30. Conclusions  Rigorous and transparent review process  Ensures reliable and high-quality resources  Framework of Climate and Energy Literacy Principles allow for gap analysis in collection

31. Outlook  Targeted search to fill gaps in collection  Broaden scope of collection to other educational resources  Promote the collection and build a community of educators (Professional Development/Discussions)  Refining Energy Literacy framework for collection

32. Detailed Review Criteria Details about the CLEAN review process: http://cleanet.org/clean/about/review.html Link to Initial Vetting Questionnaire: http://cleanet.org/files/clean/about/clean_vetting_questionnaire.pdf Link to Review Questionnaire: http://cleanet.org/files/clean/about/clean_review_questionnaire.pdf Link to Expert Science Review Questionnaire: http://cleanet.org/files/clean/about/clean_science_review_questionn.pdf

33. CLEAN collection: cleanet.org