In a new Firefox window, go to http://cb.cete.us. Review Process In a new Firefox window, go to http://cb.cete.us. Firefox ESR is recommended Using other versions could cause functionality and display issues Audio: In a new Firefox window, go to http://cb.cete.us.

Enter your email address and password. Review Process Enter your email address and password. Audio: Enter your email address and password.

3. Click on “Review Management.” Review Process 3. Click on “Review Management.” Audio: Click on Review Management.

Review Process Audio: This is the welcome screen that you will see after logging in. There is contact information for the help desk, should you experience issues with the review system.

Review Process Under “Organization,” click on the organization listed next to the collection name on your checklist. Audio: Under “Organization,” click on the organization listed next to the collection name on your checklist. You may have more than one organization in the tree. The organization is the current location of the item in our system, and not an indicator of where its final placement will be.

Review Process Review assignments will then display below in a table. Please review ALL collections. Only the collections from your selected organization will be displayed. Audio: Review assignments will appear in a table below. In this table, you can see the name of the collection, the number of items, as well as the status. You can sort the collections by these headers. This may be helpful if you know that you have a certain amount of time to review items and wish to complete a collection with fewer items. Please complete the review all of the collections which are assigned to you. Only the collections from your selected organization will be displayed. Please remember to complete the review of all of the collections in all of the organizations. You will have at most two organizations.

Review Process To start reviewing a collection, click the arrow button in the “Begin Review” column. Audio: To start reviewing a collection, click the arrow button in the “Begin Review” column.

Review Process Metadata will be displayed above the item. Please refer to your training materials about how to use this information. Audio: Above the item, you will see the item’s metadata. Metadata is information about the item, such as its name, alignment to the content framework, and depth of knowledge level. Depending on your review panel, you may or may not need this information. Please refer to your training materials.

Use the “Comments Section” to record appropriate feedback. Review Process After reviewing the item, choose either “Accept,” “Revise,” or “Reject.” Use the “Comments Section” to record appropriate feedback. Audio: After reviewing the item, choose either “Accept”, “Revise”, or “Reject.” The system will default to “Accept,” but you must click save in order for the system to record it as feedback. Use the “Comments Section” to record appropriate feedback. Bias and sensitivity reviewers, you will need to put the appropriate code in this section.

Review Process If you choose “Revise” or “Reject,” please provide a very specific explanation for why in the “Comments Section.” Audio: We hope that the “Reject” button is rarely selected, because we would like to make revisions in order to keep the item. Please make sure that you include very specific comments if you think an item is not able to be saved through any amount of revision.

Review Process Informative Bias/Sensitivity Examples 2c – The second and third responses rely on knowledge of photosynthesis. These answer choice are not based on information in the passage. 3c – $800,000 for the price of a house is well above the median home price for a house in KS. Audio: We will share a few examples of feedback that we find informative and some that are less informative. These are examples of informative comments about potential bias and sensitivity issues. An informative comment could point out where an item relies on information not contained in the passage. This is a potential barrier for a student in answering the item, because it relies on prior knowledge that cannot be obtained through careful reading of the passage. Last, comments about an unrealistic context are helpful, too, such as this comment about the price of a home being too high. This is a potential sensitivity issue for students of varying socioeconomic statuses.

Review Process Informative Content Examples This item doesn’t align to the content code/this grade level. It aligns more closely to… because… The key is incorrect. The key should be… Audio: These are informative comments for a Content review. If you find an item that does not align, please state the target to which you believe the item does align. Please also describe why it aligns more closely to the new target than its current target. If you find an item that does not have a correct key, please state what the key should be.

Review Process Less Informative Examples  This question is hard. I wonder if students in rural areas will understand this. We don’t teach this. Some kids don’t like dogs. Audio: These are examples of less informative comments, regardless of the review type. It is hard for us to translate what exactly is meant by these comments. Does the sad face mean you don’t like the item? Does it mean that you were sad when you reviewed the item? Stating that the question is hard does not help us much, either. Is it a good thing or a bad thing that the question is hard? When the comment says, “I wonder…”, we don’t have the information we need to make any possible revisions. Comments about what kids know or what is taught in your classroom must also reflect the ELA and Math Standards. Before making a comment like this please check the standards to verify what the expectations are. If the content is not aligned with the standard then please provide a new appropriate standard and grade level. And lastly, we know that all students won’t like all things. You are seeing a lot of items that won’t necessarily be on the same test, so there is a good chance that items will be mixed up and students won’t only see items about animals.

Review Process Less Informative Content Examples I prefer a semicolon instead of a period. Rephrase the question. This vocabulary word is too hard. Audio: These are less informative comments for a Content review. The items in this review have been edited according to our style guide, so suggestions for minor rewording and conventions are not necessary. Some vocabulary words will be difficult, as they are a part of the tested content.

If you do not click “Save,” your feedback will be lost. Review Process Click “Save.” If you do not click “Save,” your feedback will be lost. Audio: After selecting the appropriate button and entering your feedback in the Comments Section, click the Save button. If you do not click Save, even if you want to select the default “Accept,” your feedback will be lost.

Click “Yes” to continue reviewing items. Review Process Click “Next Item.” Click “Yes” to continue reviewing items. Audio: When you are ready to move on, click Next Item. When you are asked if you would like to continue, click Yes.

Review Process A blue bar will display your progress in the current collection as you save your feedback. Audio: As you save feedback, a blue bar will display your progress within the current collection.

Review Process You can view the key and related media, as well as preview the item similar to how a student would view it by clicking the buttons above the item. Audio: There are three buttons above the item which you may need as a part of your review. You can view the key (or correct answer), related media, for example the text passage that goes with an ELA item, and preview the item similarly to how a student would view it. However, you will not be able to interact with multiple-choice items. Bias and sensitivity reviewers will rarely need to use these buttons.

Review Process You can also navigate to items/passages through the drop-down menu. Click “Save,” then open the drop-down and click on the task/media. Audio: You can also navigate to items and passages through the drop-down menu. Open the drop-down and click on the task/media. It will automatically jump to your selection. You will not be asked if you want to continue, so please make sure that you save any work before moving on.

Review Process After the last task in the collection, there will be a “Summary” screen that shows the completion status of each task. You will not be able to submit your review collection until all of the task feedback has been saved. Use the drop-down navigation to return to items in red. Audio: None

Review Process When all of your task feedback has been saved, click the green “Submit” button. You will then see a screen that displays this message: Audio: When you complete a review of a collection, you will get a screen that displays a completion message.

Review Process In the “My Review Assignments” window, the collection will display “Complete” in the “Status” column. Also, there will no longer be an arrow in the “Begin Review” column. Audio: In the My Review Assignments window, the collection will display Complete in the Status column. There will no longer be an arrow in the Begin Review column. Collections that have not been completed will still be available for you to complete.

Review Process If you must stop reviewing in the middle of a collection, when you start the review again, you will be returned to the first item that does not have saved feedback. After you finish reviewing all of the tasks/media in a collection, you cannot return to that collection. Audio: If you must stop reviewing in the middle of a collection, when you start the review again, you will be returned to the first item that does not have saved feedback. After you finish reviewing all of the tasks and or media in a collection, you cannot return to that collection.

If you have any questions, please do not hesitate to contact us. Thank you! If you have any questions, please do not hesitate to contact us.  Topic Contact Contact Information ELA Content Lauren Adams laurenadams@ku.edu Math Content Alicia Stoltenberg aliciastoltenberg@ku.edu Science Content William Bennett william-d-bennett@ku.edu Bias and Sensitivity Laura Kramer laura.kramer@ku.edu Review Management System or Firefox ESR CETE service desk 785.864.3537 or 855.277.9752 Mon-Fri, 8 a.m. – 5 p.m. CT