1. The Review Process: Where Do We Begin? Jennifer L. Bishoff June 7, 2001

2. The review process is necessary in order to ensure that all material entering the digital library is accurate, appropriate, and useful.

3. Examples of Previous Review Processes MERLOT College Board JOMA NCTM NEEDS for SMETE

4. MERLOT’s Lengthy Criteria Three general categories Quality of content Effectiveness as a teaching-learning tool Ease of use Goal: to provide information for users on the best material in the Mathematics Database

5. Quality of Content Is the material factually correct? Does the material present important mathematical concepts? Does the material help develop conceptual understanding of mathematics? Does the material make effective use of graphics and multimedia? Is the material flexible?

6. Effectiveness as a Teaching- Learning Tool Does the material improve faculty and students’ abilities to teach and learn? Can the material be readily integrated into the mathematics curriculum? Can the material be used in a variety of ways? Are the teaching-learning goals for the material easy to identify? Assessment.

7. Ease of Use Is the general layout of the material consistent and intuitive? Does the material provide effective feedback? Is the material easy to navigate? Is the material documented and does it have useful instructions?

8. College Board: Editorial Guidelines for Calculus For teachers of AP Calculus Designed to be flexible; does not place many restrictions on content or format Submitters previously agree to certain editorial conventions 200-500 word summary/review to evaluate usefulness to AP teachers, and to provide background information

9. CB’s Questions to Consider Is the resource appropriate to AP Calculus? Will it help understanding of course content? Will it help AP teachers to prepare courses? Is it appropriate for classroom use? What are the limitations? Degree of difficulty? What type/level of technology required?

10. CB’s Specific Questions for Online Resources Is the content of high quality and correct? How does the page compare to similar resources on the topic?

11. The JOMA Review Instructions: Read the paper and try the software, then email conclusions about the following: Correctness Significance Appropriateness Usability Recommendation Typos and/or software bugs

12. NCTM Guidelines for Reviewers Not for subject evaluations! Narrative form, addressing the following points: Overview Errors Uses of the material Software Opinions Recommendation

13. NEEDS for SMETE Very precise form See handouts

14. What we have so far… From 02/12/2001: Three categories Metadata Technical Content Rankings Acceptable as is Acceptable with revisions Unacceptable No comment on style or pedagogy Focus on accuracy Reviews not available to user

15. Moving on…a discussion Narrative or checklist style? More categories with subcategories similar to MERLOT: which ones? Review specific to each discipline? Format: email or letter? Rough draft for next meeting…

16. References Editorial Guidelines for Calculus at http://mathforum.com/mathnsdl/CBGuidelines.html http://mathforum.com/mathnsdl/CBGuidelines.html Instructions to JOMA Referees at http://mathforum.com/mathnsdl/Instructions.html http://mathforum.com/mathnsdl/Instructions.html MERLOT Evaluation Standards for Mathematics at http://mathforum.com/mathnsdl/MERLOTeval.html http://mathforum.com/mathnsdl/MERLOTeval.html NCTM Guidelines for Reviewers at http://mathforum.com/mathnsdl/nctm1.html http://mathforum.com/mathnsdl/nctm1.html NEEDS Judging Form at http://www.needs.org/engineering/premier/2000/2000criteria- final.pdf http://www.needs.org/engineering/premier/2000/2000criteria- final.pdf