1. U.S. Department of Health and Human Services Health Resources and Services Administration APPLICATION REVIEW MODULE (ARM) HRSA Division of Independent Review (DIR) Reviewers Manual

2. U.S. Department of Health and Human Services Health Resources and Services Administration A BOUT ARM About ARM Once you have read and considered each application, ARM is the web- based computer system you will use to record the results of your evaluations. It provides a user-friendly environment to facilitate the HRSA/DIR grant review process while allowing reviewers to participate from remote locations.  The users of ARM that are discussed in this material are the Reviewer and the Summary Statement Operator (SSO). 2

3. U.S. Department of Health and Human Services Health Resources and Services Administration A CCESSING ARM  Log-in under the “Reviewer Logon” tab at the top. 1.Click in or Tab to User Name and enter your username and then to Password and enter your password into the appropriate boxes. Then click the button. Note: Usernames and Passwords are case sensitive.  Each review in ARM has a unique Session ID. You may be a reviewer on more than one active review. Your Username and Password will provide you access to any active review to which you have been assigned. Accessing ARM You will receive an email prior to your review containing your Username and Password as well as the link to the ARM website. 3

4. U.S. Department of Health and Human Services Health Resources and Services Administration S TARTING THE R EVIEW Accessing Your Review 2.Find the Session ID or Session Name that matches the review you will be working on and click on the hyperlinked Session Name. 3.Click on the hyperlinked panel number to select your panel. 4 Note: If the “Program Support Site” column has a Go to Site link visible, clicking on the link will open an outside website in a new window.

5. U.S. Department of Health and Human Services Health Resources and Services Administration S TARTING THE R EVIEW Application Selection Screen 4.Click on the hyperlinked Application Name to advance to the Application Worksheet Screen and begin your review. Note: In the score column, if the score shown is -- you have not entered any scores for this application. If an * is present next to the score, you have not entered all of the individual scores required for this application.  Application Number  Application Name  Status  Total Score  Scores Entered  Comments Added  View Application PDF File (optional)  If there are no applications listed, please contact your HRSA/DIR review administrator. This screen displays six major identifiers of the application. 5

6. U.S. Department of Health and Human Services Health Resources and Services Administration A PPLICATION E VALUATION S CREEN This screen serves as the primary control panel for evaluating that particular application. All applications will be scored by individual criterion. Application Worksheet Screen  The scoring criteria are listed in the first column.  The middle column is where you will click to enter your scores. The score will automatically total as you move through the criteria.  The last column is where you will be able to click and enter comments after you have entered scores.  Application Budget Data 6 These options may apply to your review: Priority Points Question(s) Preference Question(s)

7. U.S. Department of Health and Human Services Health Resources and Services Administration V IEW C RITERIA D ESCRIPTIONS 7 View full Descriptions of Criteria (if applicable) If the review criteria were setup with detailed descriptions, you may view them by clicking on the hyperlinked header Criteria.  Clicking on the header Criteria will open a new window with the full descriptions visible.  You may keep this window open in the background and reference it if you need to.

8. U.S. Department of Health and Human Services Health Resources and Services Administration V IEW P RIORITY P OINTS D ESCRIPTIONS 8 View full Descriptions of Priority Points (if applicable)  Clicking on the header Priority Points Question(s) will open a new window with the full descriptions visible.  You may keep this window open in the background and reference it if you need to. If the review was setup with Priority Points and detailed descriptions were added, you may view them by clicking on the hyperlinked header Priority Points Question(s).

9. U.S. Department of Health and Human Services Health Resources and Services Administration V IEW P REFERENCE D ESCRIPTIONS 9 View full Descriptions of Preference Questions (if applicable) If the review was setup with Preference Questions and detailed descriptions were added, you may view them by clicking on the hyperlinked header Preference Question(s).  Clicking on the header Preference Question(s) will open a new window with the full descriptions visible.  You may keep this window open in the background and reference it if you need to.

10. U.S. Department of Health and Human Services Health Resources and Services Administration A PPLICATION E VALUATION S CORING Application Scoring First, users must enter a score BEFORE any comments can be entered. Note: You will not be able to select a score greater than the maximum defined by the criterion. 5.In the Score column, click on the hyperlinked -- to begin scoring the application. 6.Using the drop-down arrow, select the appropriate score by clicking on the number. 7.Click. 10

11. U.S. Department of Health and Human Services Health Resources and Services Administration A PPLICATION E VALUATION S CORING Entering Comments ONLY AFTER YOU ASSIGN A SCORE to an individual criterion can you include corresponding comments. 8.Click on the hyperlinked word Comment to write a strength or weakness for the respective criteria. Note: If you have not entered all required scores for an application, your Actual Score will be followed by an * indicating that your scoring is not complete.  Again, as seen in the image, the only activated Comment links are those with a corresponding score. The link will automatically appear once you have scored the criterion. 11

12. U.S. Department of Health and Human Services Health Resources and Services Administration A PPLICATION E VALUATION C OMMENTS Entering Comments Clicking the Comment link will open the Comments window. 9.Click on the drop-down arrow to open the menu. Select whether you are describing a Strength or Weakness. 10.Enter a page number or any page identifying information. Note: All fields in this screen are required. 11.Enter your strength or weakness. You can type directly into this field or copy and paste from a word processing program. After entering the comment, it is strongly recommended that you Spell Check your work by clicking on the Spell Check button. 12.Click to save your comment as part of your evaluation. 12

13. U.S. Department of Health and Human Services Health Resources and Services Administration 13 A PPLICATION E VALUATION C OMMENTS Saving and Retrieving Saved Comments  Clicking on the SAVE button will allow you to save data into the ARM system. The data will remain even if you time out or you decide to enter additional text at a later time. It is important that you click on the “Submit” button to finalize the comment even if you have saved the comment. Clicking on the submit button will provide you the opportunity to continue adding additional comments.

14. U.S. Department of Health and Human Services Health Resources and Services Administration 14 Saving and Retrieving Saved Comments  Clicking on the “Retrieve the Last Comment” link will load the comment that was saved most recently. In case of a time out, you should go to the respective criteria and click on the “Retrieve the Last Comment” link in order to retrieve the comment you were working on. A PPLICATION E VALUATION C OMMENTS

15. U.S. Department of Health and Human Services Health Resources and Services Administration A PPLICATION E VALUATION C OMMENTS Entering Multiple Comments You can enter multiple strengths or weaknesses for each criterion.  To enter multiple comments, repeat steps 9-12.  All previous comments will be visible toward the bottom of the page, below the “Add a New Comment” box.  Regardless of the order in which they were entered, comments will group by category (Strength or Weakness). 15

16. U.S. Department of Health and Human Services Health Resources and Services Administration A PPLICATION E VALUATION C OMMENTS Editing Comments You can edit your saved comments at any time. Deleting Comments  Click Delete to permanently remove a comment from your evaluation.  To edit your comments, click on the word Edit next to the comment number you would like to change.  Clicking on the Edit link will open the comment window and you will be able to edit your content. When complete click to save your changes. 16

17. U.S. Department of Health and Human Services Health Resources and Services Administration S ESSION T IME -O UTS AND P REVENTING D ATA L OSS Session Time Out  In order to provide maximum data security, the ARM system sessions are designed to “Time-Out” after 30 minutes of inactivity. After 25 minutes, if you are not clicking between ARM web pages, a warning message will appear. If no action is taken within 5 minutes, you will be timed out. Any unsaved work will be lost. It is Highly recommended that you take advantage of the SAVE button. 17

18. U.S. Department of Health and Human Services Health Resources and Services Administration A PPLICATION E VALUATION S CORING Completed Comments Once you have finished entering all of your comments for a given criterion, you will need to return to the Application Worksheet Screen. 13.Click on the hyperlinked word Evaluation in the bar at the top of your screen to return to the Application Worksheet Screen after you have included all comments for a given criterion. 18

19. U.S. Department of Health and Human Services Health Resources and Services Administration A PPLICATION E VALUATION S CORING Application Worksheet Screen After returning to this screen continue entering comments for the remaining criteria as needed.  Repeat steps 8-12 to add comments for each criterion 19

20. U.S. Department of Health and Human Services Health Resources and Services Administration S CORE A LL C RITERIA IN ONE S CREEN 20 Entering Multiple Scores at Once  The full Add Scores screen will open, and you will be able to click on the drop-down arrow for each criterion and assign a score.  You may also view the full descriptions of the criteria by clicking on the Show Criteria link at the criterion level OR you can view all of the descriptions for all of the criteria by clicking on the header Show All Descriptions  If you would like to enter all of your scores at one time, click on the hyperlinked column header Score.  When you have entered all of the scores, click.

21. U.S. Department of Health and Human Services Health Resources and Services Administration A PPLICATION E VALUATION B UDGET 21 Entering Application Budget Data From the Application Worksheet Screen you will need to enter the Application Budget Data. 14.Click button to enter Budget Data.

22. U.S. Department of Health and Human Services Health Resources and Services Administration A PPLICATION E VALUATION B UDGET 22 Entering Application Budget Data 15.Enter budget information for all fields marked with a red asterisk. 16.Click the button when you are done.  If “As Reduced” is selected a supporting comment is required.  You may also SAVE the comment. Upon returning to this page at a later time, the SAVED comment will be pre-populated into the comment field.

23. U.S. Department of Health and Human Services Health Resources and Services Administration 23 Entering Application Budget Data Note: If your review session does not have any Preference Question(s) and/or Priority Point Question(s) associated with it, then Criteria Comments, Scores, and Application Budget Data will be the only sections you will see. APPLICATION EVALUATION BUDGET

24. U.S. Department of Health and Human Services Health Resources and Services Administration A PPLICATION E VALUATION Q UESTIONS 24 Priority Point Questions After returning to the Application Worksheet Screen, you will need to answer the Priority Point Question(s). You will only see this section if it is applicable to the application you are reviewing. 17.Click the Add/Edit link next to the priority point question. 18.Click on the Priority Met drop down menu and select Yes or No. 19.Click on the Priority Point drop down menu and select a point value. 20.Enter your comments. 21.Click the button once, you are done. You may also SAVE the comment and complete it at a later time.  Don’t forget to Spell Check

25. U.S. Department of Health and Human Services Health Resources and Services Administration A PPLICATION E VALUATION Q UESTIONS 25 Preference Questions After returning to the Application Worksheet Screen, you will need to answer the Preference Question(s). You will only see this section if it is applicable to the application you are reviewing. 22.Click the Add/Edit link next to the preference question. 23.Click on the “Preference Met” drop down menu and select Yes or No. 24.Enter your comments. 25.Click the button once, you are done. You may also SAVE the comment and complete it at a later time.  Don’t forget to Spell Check

26. U.S. Department of Health and Human Services Health Resources and Services Administration S CORING R EPORTS Reviewing your Work You can review your work at any time in the process.  To view all of your strengths and weaknesses for each criterion, click. Click to view your entries for all other sections. Each of these views open in a new window and show information for that individual application, and each view can be printed as a report.  To view your scores by criterion, you can click the at any time. The view will open in a new window from which you can also print the view as a report. 26

27. U.S. Department of Health and Human Services Health Resources and Services Administration S UBMIT R EVIEW Note: The Submit to Chairperson button will not be activated until ALL scores have been entered. Note: Once you submit your application evaluation to your SSO, further edits CANNOT be made. Submitting Your Review When you have completed scoring the application and are satisfied with your comments, you will need to submit your evaluation to your SSO. 26.Click the button to submit your application evaluation to your SSO. 27.Click OK in the confirmation box to finalize your submission. 27  The button will not be activated until all sections are completed.  To evaluate other assigned applications, click “Applications List” located on the blue tool bar at the top of the page.  When all applications have been evaluated and submitted to the chair, log-off of the system.

28. U.S. Department of Health and Human Services Health Resources and Services Administration S CORING AND D OCUMENTS R ULES  You can only see your own scores and comments.  You must enter a score for each criterion BEFORE proceeding to enter strengths and weaknesses. The Comments link will be automatically activated you enter a value in the score field.  The button does not become active until the your review of an application is complete. You cannot submit your review to the SSO until you have scored all criteria, completed all strengths and weaknesses, entered all budget data, and if applicable, answered all preference and priority point questions.  You cannot change or modify your scores, strengths and weaknesses, budget data, and if applicable, priority points and preference questions once an application has been submitted to the SSO. You can, however, still view and print your own scores, comments and entries to other sections.  All Fields marked with an asterisk * are mandatory.  If the Back Arrow or the blue Navigation Bar are available, use one or the other to navigate within ARM. Only close a window when neither the Back Arrow or Navigation Bar are available.  Exit ARM by clicking on Logon on/off in the upper right corner or your screen. Application Worksheet Rules 28

29. U.S. Department of Health and Human Services Health Resources and Services Administration A PPLICATION S TATUS D EFINITIONS  When an application has been assigned and is available to you to enter criteria scores, strengths, weaknesses, budget data, and, when applicable, assign preferences and priorities, that application’s status is In Review.  When you finish entering all your scores, strengths, weaknesses, budget data, and when applicable, assigning preferences and priorities, you can click the button and that application’s status changes to Submitted to Chair.  (Web-Assisted Teleconference Only) When the panel has reached consensus and the SSO has recorded all of the panels recommendations regarding strengths, weaknesses, budget requests, and if applicable, the panel’s recommendations on assigning preferences and priorities, the SSO will return control to all reviewers on the panel so that they may enter or modify their scores for each criteria. That application’s status is then changed to Returned by SSO. Once each reviewer has entered, changed, or decided to leave his or her scores unchanged, they will again click the button, and that application’s status once again becomes Submitted to Chair.  Some HRSA programs require that a panel decide by majority vote whether an application is worthy of consideration for funding. The vote will be called for by the Chair, once the panel has agreed upon consensus statements of criteria strengths and weaknesses, but before any reconsideration of score. If the panel votes not to consider the application for funding, the SSO will change that application’s status to Not Recommended. 29

30. U.S. Department of Health and Human Services Health Resources and Services Administration Note to Field and Face-to-Face Meeting Reviewers  The remaining slides in this presentation discuss a process that occurs when HRSA employs a Web-assisted Teleconference process to score or modify scores of grant applications. However, if you are participating in…  … a Field review (no discussion of each application by a panel of reviewers) you will seldom be asked to reconsider your score of an application; if you are, you’ll be given explicit instructions at that time on how to proceed.  … a Face-to-Face meeting, once the panel has agreed upon all strengths, weakness, budget recommendations, preferences, and priorities for recommended applications, each reviewer will be given a score sheet on which he or she can record their final scores by criterion. These will be passed to the SSO who will enter them into ARM. 30

31. U.S. Department of Health and Human Services Health Resources and Services Administration R ETURNED A PPLICATIONS Entering or Modifying Your Final Scores After your panel finishes their discussion of an application, your SSO will return the application to you so you can enter or modify your scores.  You will know if an application has been returned to you because the status on your Application Selection Screen will read “Returned by Chair”. 28.To begin the process of modifying or entering your score for an application, click on the Application Name, as you did in Step 3, to open the Application Worksheet Screen. 31

32. U.S. Department of Health and Human Services Health Resources and Services Administration R EVISE A PPLICATION E VALUATION To change or enter a criterion score, click the applicable Score link. When you have finished updating or entering, click to send your evaluation back to the SSO. Revising Your Evaluation At the direction of your Chairperson and based on your panel discussions, you will enter or modify your scores. Note: Once you re-submit, you will no longer have access to the Return History screen from this screen. 32

33. U.S. Department of Health and Human Services Health Resources and Services Administration E ND Please follow the instructions in this manual to process all the applications on your panel. Please contact the HRSA ARM Technical Assistance Team if you have any questions at ArmTechAssistance@hrsa.gov ArmTechAssistance@hrsa.gov Thank you. 33