CLIMATE LITERACY AND ENERGY AWARENESS NETWORK PATHWAY CLIMATE LITERACY AND ENERGY AWARENESS NETWORK PATHWAY SUMMARY OF A RIGOROUS REVIEW PROCESS CLN Webinar December 7, 2010

Outline  CLEAN project and CLEAN collection  Review Process  Results of first review round and gap analysis  Conclusions and Outlook

CLEAN Pathway Project Goals  Stewarding a collection of excellent teaching resources on climate and energy science  Professional and Community Development  Online communities  Facilitate the Climate Literacy Network

CLEAN Pathway: Collection  500 excellent digital teaching resources addressing climate science or energy awareness for grades 6-16  Resources scientifically and pedagogically reviewed  Annotations reflect reviewer comments  Resources aligned with  Climate Literacy: Essential Principles of Climate Science  Energy Awareness Principles  National Science Education Standards  AAAS Project 2061Benchmarks for Science Literacy  NAAEE Excellence in Environmental Edu. Guidelines for Learning

Framework for collection  Energy Awareness Principles (newly developed) A. Earth System Energy B. Human Energy Sources C. Impacts of Energy Use D. Energy Distribution/Costs E. Energy Access and Equity F. Decarbonization Challenges  Every activity included in collection has to address either  Climate Literacy: Essential Principles of Climate Science

CLEAN review process  Phase 1 of CLEAN focused on teaching activities  Following phases: broadening of scope to videos, visualizations, lab activities, background materials etc.

Developing Review Criteria  Review criteria specific to teaching activities  Informed by NSDL and SERC guidelines, Merlot criteria, DLESE, Climate Change collection scorecard  Review of e-learning materials requires additional considerations (multi-media elements, navigation etc.)  Climate and energy science evolving science – faster turn-around and need more media to convey

Developing Review Criteria  Initially long list of possible criteria: convergence in phrasing of questions during 7 test review rounds  Test reviews: good agreement for good resources, wide spread in answers for low quality resources  Weekly telecons for resource collectors

What is an excellent activity? Definition Teaching Activity: Relatively brief set of instructional materials that is presented as a whole, where all the parts and ideas are linked and part of the same activity Educator should be excited to find this activity when searching for teaching materials.

Avenues into collection  Existing resource pools found through online search  Resources suggested by public through online form  Targeted search after gap analysis  Collaboration with projects that submit resources directly to CLEAN team (iterations)

Informal vetting  Resource pools assigned to team of 9 collectors  Scanning of resource pools and informal triage  Informal triage: Does this resource meet the CLEAN criteria?

Review questionnaire  Initial Vetting  Review  Scientific accuracy  Pedagogic effectiveness  Technical quality / Ease of use → 6-12 questions for each category, overall rating in rubric format, comment box for annotations → Questions help to consider all relevant aspects for each category and lead to overall rating → No quantitative, only qualitative recommendation (low – medium – high priority)

Formal vetting step  Basic check for relevance to CLEAN project  Decision: “Will resource likely pass the review?”  Recommendation: Move on to initial review or put in “holding tank”

Review: Scientific accuracy  Considerations for initial science review  Solid, current science  Original data cited and data from a quality source  Attribution  Valid concepts  Supporting references

Science: Use of digitally available scientific data to teach concept

Science: Simplified models of complex Earth system to teach concept

Review: Pedagogic effectiveness  Considerations for pedagogic review  Learning objectives  Accommodates diverse learners (learning styles, language, cultural diversity)  Prerequisite skills and understandings  Assessment strategies  Engaging for students in subject and approach  Requires independent/inquiry-based thinking

Pedagogy: Inquiry-based labs or experiments

Pedagogy: Pen-and-paper graphing exercise

Review: Technical quality/ Usability  Considerations for technical/usability review  Ready for use, stands on its own  Clear presentation of content  Software/tools/resources commonly found in classroom  Amount of necessary guidance for students by instructor  Offers comprehensive guide for instructor  Digitally available resource

Ease of use: Printable PDFs with background info and instructions

Ease of use: Clear steps-by-step instruction

Expert science review  External expert with PhD in relevant field reviews scientific quality and accuracy of resource  Activity already passed lower level science review (75 % of resources that passed the CLEAN review were rated scientifically excellent by experts)  Limitation/Challenge:  Grade-level appropriate science  Difficult to find scientists with enough time

Panel review  Based on NSF-panel review system and AccessData Workshops  Panel provides necessary range of expertise  Teams of 4 educators and scientists review each teaching material based on prior reviews, final decision about inclusion in collection  Comments of all reviewers are compiled into annotation which includes teaching tips

Annotations  All reviewer comments, suggestions and tips are combined in notes to users (annotations)  Annotation draft reviewed during review panels, final clean-up during cataloging process  Annotations add considerable value, insight from scientists or experienced educators

Demo on CLEAN Website

Numbers first review cycle  Informal vetting of ~5000 teaching activities  ~200 resources passed first review with medium or high priority > forwarded to second review  142 resources passed second review with medium or high rating > forwarded to panel review  94 resources passed panel review (18 passed on to editorial board)  5-7 different people reviewed each resource

Gap Analysis  Holes in collection are apparent – inform targeted search and hopefully future solicitations Total # of activities # of concepts with no match # of concepts with ≤ 3 matches Guiding Principle2225 of 7 EP of 5 EP of 6 EP of 5 EP of 7 EP of 5 EP 6815 of 5 EP of 6 Energy Awareness3401 of 6

Conclusions  Rigorous and transparent review process  Ensures reliable and high-quality resources  Framework of Climate and Energy Literacy Principles allow for gap analysis in collection

Outlook  Targeted search to fill gaps in collection  Broaden scope of collection to other educational resources  Promote the collection and build a community of educators (Professional Development/Discussions)  Refining Energy Literacy framework for collection

Detailed Review Criteria Details about the CLEAN review process: Link to Initial Vetting Questionnaire: Link to Review Questionnaire: Link to Expert Science Review Questionnaire:

CLEAN collection: cleanet.org