Dewey Peer Review System Cal Collins and Shai Sachs Isovera Consulting Dec. 2006

Internet consulting for non-profits 2 Overview The Peer Review System is an add-on to the Dewey Cataloging System which supports the peer review process for a collection of learning objects. All learning objects contributed to the BEN Portal must be peer- reviewed. Currently, many collections conduct peer review “off-line” or via . The Dewey Peer Review System will allow your editors and reviewers to work collaboratively online, with no need to maintain many different versions of Excel spreadsheets, Word documents, etc.

Internet consulting for non-profits 3 Features Major areas of functionality are: –Creation of user roles to facilitate targeted workflow assignments –Workflow management –Review form management –Group reviews, single-person reviews, assignment, feedback –System monitoring Features which are not included: –Segmented workflows and resource grouping –Templated s –Unassignments and rollbacks

Internet consulting for non-profits 4 Technical Components Minimum requirements: –MySQL 4.0 –PHP 4.0 –Apache 1.3 –Works on Windows and Unix servers Designed as a plug-in to the Dewey system. –Plug-ins are mini-applications supported by the Cake PHP framework. –Cake PHP is the framework which supports the Dewey Cataloging System.

Internet consulting for non-profits 5 Timeline First draft of software requirement specification has been released. Final draft of software requirement specification due for release in early Jan Review form management will be released in March Performing reviews, assignments, and feedbacks released in May. –Possible pilot demonstration of the system in May Workflow management and system monitoring released in June. –Beta release commences in June Production release in July 2007.

Internet consulting for non-profits 6 Workflow Management All resources in the system are reviewed through the workflow. The workflow is a chronological series of steps designed by the administrator. It may be revised over time. The first step is always the submission of the resource. Subsequent steps are either: –Assignments: one user orders one or more other users to carry out some other step –Parallel reviews: a set of reviews conducted by one or more people –Single-person reviews: a review conducted by exactly one person

Internet consulting for non-profits 7 Single-person reviews A single-person review results in a decision on the resource. Available decisions include: –Accept –Reject –Feedback / Accept with revisions –Redirect the resource to another step in the process

Internet consulting for non-profits 8 A sample workflow Submitted Assignment by Managing Editor Assignment by Associate Editor Parallel Reviews by Peer Reviewers Single-Person Review by Associate Editor AcceptedRejectedFeedback from Author

Internet consulting for non-profits 9 A sample workflow, Contd User Groups: Managing Editors, Associate Editors, Peer Reviewers 1.Submitted 2.Assignment step: “Assignment by Managing Editor” 1.From: Managing Editors 2.To: Associate Editors 3.Assignment step: “Assign peer reviews” 1.From: Associate Editor 2.To: 1.Lab exercise reviewers, if the resource is a Lab exercise 2.Textbook reviewers, if the resource is a Textbook 4.Parallel Review step: “Perform peer reviews” 1.Lab exercise review 1.Review form: Lab exercise Review Form 2.Textbook review 1.Review form: Textbook Review Form 5.Single-Person review step: “Issue final decision” 1.Available redirection steps: “Peer Review”

Internet consulting for non-profits 10 Review Forms Review forms are the key to substantive quality assessments. They ask reviewers questions like “Is the resource scientifically accurate?”, “Please comment on the originality of the resource.”, and “What is the pedagogical merit of the resource?” Review forms are designed by the administrator and may be revised over time. Review forms consist of a single page of inputs.

Internet consulting for non-profits 11 Review Form Inputs Available input types are single-line text, multiple-line text, drop- down lists, and files. When designing a drop-down list input, administrators specify a controlled vocabulary which contains the options for the drop-down list. Inputs are accompanied by standard Dewey user interface cues: –Prompt (appears to the left of the input in bold) –Brief help (appears below the prompt, not in bold) –Extended help (opens in a separate window)

Internet consulting for non-profits 12 Review Forms The Gory Truth Review form management relies on the metadata management, controlled vocabulary management, and input process management capabilities provided by Dewey. Review form management provides simplified, streamlined access to these capabilities. Each review form corresponds to a single metadata structure, a single input process, and as many controlled vocabularies as are needed to support the various drop-down lists within the form. So administrators and reviewers get all the power of the Dewey cataloging system and none of the limitations of the BEN-LOM.

Internet consulting for non-profits 13 Feedback A Word document version of the specification is available; you can make comments and suggest revisions directly on the document using Track Revisions. (Tools  Track Revisions) You may also send suggestions and feedback directly to While we will try to accommodate as many requested revisions as possible. The workflow management system has been carefully designed to avoid inconsistencies within the system, “matrix management” of the review process, and other potential pitfalls of peer review systems. Some requests may have to be modified to preserve consistency.

Internet consulting for non-profits 14 Conclusion and Next Steps Read through the first draft of the specification. Analyze whether the workflow management, as specified, will facilitate your review process. Analyze whether the review form management system will allow you to design the kinds of review forms that you want your reviewers to complete. If not, tell us why not. If you have questions, and we will try to provide clarifications.