1. Webinar for Activities Review Team On the Cutting Edge pre-GSA REVIEW CAMP Please mute your phone by pressing *6 Technical problems? Contact Krista at kherbstr@carleton.edu Webinar begins at: 1pm Eastern | 12 pm Central | 11 am Mountain | 10 am Pacific Review team instructions page at: http://serc.carleton.edu/NAGTWorkshops/review.html September 25, 2013

2. THANK YOU! The Cutting Edge Review Process is: A tremendous community service; An opportunity for you to see a lot of great activities; An opportunity to see what goes into a quality activity to inform your own work; The realization of a decade-long goal of developing an online, peer-reviewed collection of activities

3. The collections  SERC: Science Education Resource Center at Carleton College  SERC office and staff helps develop and manage web resources for many projects through collaboration  Many different collections of activities, submitted for different projects  On the Cutting Edge: the first of the projects hosted by SERC  Most of the activities in the Cutting Edge collections were submitted in connection with workshops.

4. Cutting Edge Reviewed Collection  On the Cutting Edge is conducting a review of activities in the Cutting Edge collections  Each activity reviewed once and ranked:  Exemplary (Part of Reviewed Collection)  Pass (Part of Reviewed Collection)  Keep (Not part of Reviewed Collection)  De-accession

5. Cutting Edge Reviewed Collection  Activities ranked as “Exemplary”  Come up first in searches  Are designated on individual the ActivitySheet as being part of Exemplary Teaching Activities collection

6. Cutting Edge Reviewed Collection  Activities ranked as “Pass”  Come up second in searches  Are designated on individual ActivitySheet as being part of the Peer Reviewed Teaching Activities collection

7. Cutting Edge Reviewed Collection  Activities ranked as “Keep”  Come up last in searches  Have no designation on the ActivitySheet  Worth keeping as catalyst idea  Might be too local or items might be missing or has other problematic aspects

8. Review process  Activities that have been collected across the Cutting Edge collections  Each activity receives 2 reviews  Authors of “Exemplary” and “Pass” activities receive letters  Explains review process  Indicates activity rank  Indicates that reviewer comments are available on request if the author wishes to revise  If author does revise, the activity will be reviewed again

9. Plan for webinar  Explain the web interface  Clarify review criteria  Answer questions  As we go along, please post questions in chat

10. Your list of items  When you click on Review Tool on Review Team Instructions page, your login will take you to a page that lists only your items to review

11. Your list of items  If you have not yet completed the review, you will see:  Click the URL to go to the ActivitySheet and download the actual activity and any supporting materials.

12. The review tool  Click Review It to bring up the Review Tool

13. The review tool  You will evaluate the activity in five categories  Scientific accuracy  Alignment of goals, activity, and assessment  Pedagogic effectiveness  Robustness  Activity description

14. The review tool  For each category, questions plus rubric provide guidance for what to consider

15. The review tool  Summary score will tabulate automatically  Exemplary = 4; Very good = 3; Adequate = 2; Problematic = 1  Comments help the editors understand your ranking – please don’t leave these boxes blank!!

16. The review tool  At the end of the form, you will add your view about what it would take to raise the activity to Exemplary status if it fell short in your review  The editors will use your comments to respond to authors on request. Please phrase your comments in a collegial fashion.

17. Your list of items  Once you have submitted a review, your review list indicates completion for that item

18. Can you revise a review?  Yes – click on the Review It link and then the link to what you submitted previously  Your original rankings will come up, and you can change them and add to/change your comments.  Click submit when done.

19. Pause for questions  Anyone??

20. Your review  Review the activity in the context for which it was designed  Not just whether it’s good for a particular upper level course – many will be for other courses (e.g., intro geo)  Not everything has to be a full lab or major assignment (e.g., a back-of-the- envelope calculation could be Exemplary)  Not every activity needs to be usable by all instructors (e.g., a lab requiring specific software/math/expertise background)

21. Your review  Make a summary list of the activities that you reviewed  List both the total numerical score and your overall assessment  Exemplary  “Exemplary minus”  Pass  Keep  De-accession  Bring with you to review team meeting

22. Your review  Exemplary  Must have good science, good pedagogy, and all materials so that someone else can adapt/adopt, nothing “broken”  Can be “local” if it is also a good template or model for others to emulate  Does not need to have answer key or to provide an instructor with background  We have never required these so cannot ding someone for not including them  Scoring  Exemplary or very good in all categories  Exemplary in at least three of the five.  18 or higher.

23. Your review  Exemplary minus  Could be made Exemplary with only a small amount of work, such as:  fixing a URL  uploading the latest version of the assignment or adding instructor tips  fleshing out the ActivitySheet  This is not a formal category, but it would help us a lot to have your list of “Exemplary Minus” activities  These will be ranked as Pass, but knowing that they are “Exemplary minus” will help the editors craft feedback if authors request it.

24. Your review  Pass – these become part of the Reviewed Collection  Those that aren’t Exemplary but still have value to others  Must be more than just the germ of an idea  Must have all of the components  These must have no scientific errors.  If you think there are errors, confirm this with someone else on the review team.  Those with scientific errors should go into the Keep or De-accession category, depending on the severity of the problem.

25. Your review  Keep – no designation on ActivitySheet, will come up last in a search  Nucleus of a good idea  Insufficient info for someone to adapt or adopt or has scientific errors  Author does not receive a letter

26. Your review  De-accession  Not an activity or very fragmentary  Has truly egregious problems

27. Summary  Review each activity using rubric  Score the activity in each of 5 categories  Write a summary evaluation for each  Remember that these were submitted voluntarily to a community collection  Be kind but clear  Make a summary list  Exemplary  Exemplary minus (technically a Pass)  Pass  Keep  De-accession

28. Your assignment  Each team member has ~20activities to review  Reviews must be completed and submitted using review tool before the Review Camp  At Review Camp we will  Discuss issues  Resolve discrepant reviews  Arrive at final rankings  Hopefully do a few more reviews

29. Final Thoughts  Please write your reviews in the same spirit you would like to receive them  We all teach “out of field”—so don’t be dismayed if you have an activity not related to your immediate interests  Just use your best professional judgment.

30. Final Thoughts  The goal is to have a high quality, coherent, and comprehensive collection for use by all.  Acknowldege the great work that is available; credit and kudos to authors  Help authors improve their activities  Weed out those that are not ready for prime time

31. Thank you!  Email Dave or Krista with any questions  mogk@montana.edu mogk@montana.edu  kherbstr@carleton.edu kherbstr@carleton.edu  Have fun, stay in touch, post messages or questions to the listserv, and we’ll see you in Denver!