The Review Process: Where Do We Begin? Jennifer L. Bishoff June 7, 2001

The review process is necessary in order to ensure that all material entering the digital library is accurate, appropriate, and useful.

Examples of Previous Review Processes MERLOT College Board JOMA NCTM NEEDS for SMETE

MERLOT’s Lengthy Criteria Three general categories Quality of content Effectiveness as a teaching-learning tool Ease of use Goal: to provide information for users on the best material in the Mathematics Database

Quality of Content Is the material factually correct? Does the material present important mathematical concepts? Does the material help develop conceptual understanding of mathematics? Does the material make effective use of graphics and multimedia? Is the material flexible?

Effectiveness as a Teaching- Learning Tool Does the material improve faculty and students’ abilities to teach and learn? Can the material be readily integrated into the mathematics curriculum? Can the material be used in a variety of ways? Are the teaching-learning goals for the material easy to identify? Assessment.

Ease of Use Is the general layout of the material consistent and intuitive? Does the material provide effective feedback? Is the material easy to navigate? Is the material documented and does it have useful instructions?

College Board: Editorial Guidelines for Calculus For teachers of AP Calculus Designed to be flexible; does not place many restrictions on content or format Submitters previously agree to certain editorial conventions word summary/review to evaluate usefulness to AP teachers, and to provide background information

CB’s Questions to Consider Is the resource appropriate to AP Calculus? Will it help understanding of course content? Will it help AP teachers to prepare courses? Is it appropriate for classroom use? What are the limitations? Degree of difficulty? What type/level of technology required?

CB’s Specific Questions for Online Resources Is the content of high quality and correct? How does the page compare to similar resources on the topic?

The JOMA Review Instructions: Read the paper and try the software, then conclusions about the following: Correctness Significance Appropriateness Usability Recommendation Typos and/or software bugs

NCTM Guidelines for Reviewers Not for subject evaluations! Narrative form, addressing the following points: Overview Errors Uses of the material Software Opinions Recommendation

NEEDS for SMETE Very precise form See handouts

What we have so far… From 02/12/2001: Three categories Metadata Technical Content Rankings Acceptable as is Acceptable with revisions Unacceptable No comment on style or pedagogy Focus on accuracy Reviews not available to user

Moving on…a discussion Narrative or checklist style? More categories with subcategories similar to MERLOT: which ones? Review specific to each discipline? Format: or letter? Rough draft for next meeting…

References Editorial Guidelines for Calculus at Instructions to JOMA Referees at MERLOT Evaluation Standards for Mathematics at NCTM Guidelines for Reviewers at NEEDS Judging Form at final.pdf final.pdf