1. Quarterly Reviewer Training Real News for New Reviewers Module 9 June 22, 2017
Division of Independent Review (DIR)
Office of Federal Assistance Management (OFAM)
Health Resources and Services Administration (HRSA)

2. Overview
This Training Module will assist the New HRSA Reviewer in understanding the importance of their role in the Objective Review Process, and what to expect once selected.

3. The Division of Independent Review (DIR)
MISSION:
DIR’s mission is to ensure a fair, ethical and
objective review of each application.
PURPOSE:
DIR ensures that the process for reviewing
each application for funding is fair, open, and
competently performed.

4. So You Are Registered in RRM…What Happens Next?
To Increase the Probability of Being Selected as a HRSA Reviewer:
Be sure that your contact information is current ( , address on record and contact number)
Be sure that your skillset(s) and area of expertise is clearly documented on your CV

5. The Reviewer Role
As reviewers, your role is to fairly, equitably and objectively review and evaluate each application by clearly documenting strengths and weaknesses.

6. Access to Review Materials
HRSA uses an on-line grant review process called The Application Review Module (ARM), available 24/7 during the evaluation cycle to accommodate reviewer flexibility
You will receive an from the Contractor which contains the URL/link for the Program Support Site
All review materials are located on this site. The Application Review Module (ARM) also contains the link for this site

7. Time Commitment Please Check Your Schedule(s)
Before Making the Commitment to Serve on an Objective Review Panel.

8. Average Time Commitment
Independent Evaluation (50%)
Biggest allocation of time
Approx. 6 apps/reviewer
Read and make notes
Write down preliminary scores and comments on worksheet
28 days (avg.) – 2 apps/week
3-5 hours/application
6-10 hours/week total
Submission (15%)
Use worksheet as transfer document into ARM
Record comments (S&W) and scores/criterion
Submit to Chair
Pre-ORC (10%)
Merged summary statements
Review other apps on panel
ORC (25%)
Participate in panel discussion for ALL apps on panel
Score ALL apps on panel
Participate in panel debriefing
Be available the duration of ORC
25%
50%
10%
15%

9. Conflict of Interest and Confidentiality Certification
An individual is considered to have a conflict of interest in an application if that person, or a close relative or professional associate of that person, actually has or has the appearance of having:
Received or expects to receive a direct financial benefit of any amount deriving from an application or proposal under review;
Any other interest or relationship in the application or proposal that is likely to bias the individual’s evaluation of that application or proposal.
A Conflict of Interest or the appearance of a Conflict of Interest is prohibited. The HRSA objective review process must meet the highest ethical standards.

10. Access to Review Materials (continued)
The Program Support Site Contains:
Final NOFO
Reviewer Worksheet (Review Specific)
Scoring Rubric
Reviewer Instruction Guide
Chairperson Instruction Guide (if applicable)
COI form
Links for Chair and Reviewer Feedback (Survey Monkey)
ARM instructions/quick reference
Program Specific Information (if applicable)
Access to the Applications

11. Access to Review Materials (Continued)
Access to the Applications:
PDF(s) of the application(s) are available in the Application Review Module (ARM).
Each eligible application is read by at least 3 reviewers who work independently during the review period.
During the ORC Meeting, the three reviewers then discuss their evaluations and initial scores with other reviewers on their panel.
Reviewers will score ALL applications assigned to their Panel.   

12. Sample Merged Document

13. The Reviewer Worksheet
Working Document
Keep a copy on hand for the ORC Meeting
The Worksheet is where you write your Overview
Place any additional notes that you may want to discuss, but do not want to document as a strength or weakness in the ARM
Document any Recommendations to HRSA

14. Writing Strengths and Weaknesses
As a reviewer you are assessing the technical merit of the applications.
Critical indicators have been developed for each review criterion.
Scoring should be reflective of how well the applicant responded to the critical indicators: Review Criteria

15. Sample Strengths
The application clearly documents a well-planned training in all areas with a focus on separate populations, specific recruiting activities and selection processes.
The applicant demonstrates substantial prior experience in working with large, national health care datasets and conducting rural-urban analyses.
 The applicant organization clearly demonstrates their ability to keep track of past trainees.
The applicant organization has an extensive network of collaborative relationships that contribute to its ability to provide comprehensive and coordinated services for the target population.

16. Sample Weaknesses
The application does not clearly document critical needs at the national, regional, and local level.
The application does not clearly define current gaps in training and technical assistance for health centers serving PNHC’s that will be addressed through the proposed XXX project.
The application does not include a plan for reporting the number of times that each research product is accessed/downloaded from the applicant’s website.
The application does not provide sufficient information to demonstrate familiarity with the broad range of issues facing rural communities and the health care providers that serve residents of those communities.

17. Helpful Tips Enter all comments using complete sentences
Please keep your Strengths and Weaknesses succinct, without losing the intent of the statement
Document true Strengths and Weaknesses by Criterion…remember in addition to your comments, there are two other reviewers providing comments
Enter only one strength/ weakness per text box, do not place all of your bullet points into one text box
 Be sure that statements provided are true strengths and or weaknesses or the application, not a restatement of what the applicant organization has submitted

18. Helpful Tips (continued)
Spell out acronyms the first time it is introduced in the summary
Please carefully edit your S&W statements before submitting to ARM
Reviewers must enter budget recommendations (and Funding Factors, if applicable) When Reviewers complete entering scores and budget recommendations in ARM, they must click on the “Submit to Chair” button
When entering scores in ARM, use whole numbers

19. Suggested Scoring Guidelines Total Point Value for a Review Criterion
Scoring Rubric
Suggested Scoring Guidelines
Total Point Value for a Review Criterion
Outstanding
Very Good
Good
Satisfactory
Poor 5 4 3*
2-0 10 9 8 7
6-0 15 14
13-12 11
10-0 20
19-18
17- 16
15-14
13-0 25
25-24 23
22 -20
17--0 30
30-29
28-27
26-24
23-21
20--0 35
35-34
33-32
31-28
27-25
24-0 40
40-39
38-36
35- 32
27--0 45
45-43
42-41
40-36
35-32
31-0
Approx.%
(Overall) %
95-90%
89-80%
79-70 %
69-0%
*Based on scale not on percentage
[1] This scale is used unless otherwise indicated in the NOFO.

20. Scoring Rubric Ratings - Defined
Outstanding
All elements of the criterion are clearly addressed, well-conceived, thoroughly developed, and well supported. Documentation and required information are specific and comprehensive. The criterion has no deficiencies or weaknesses. All strengths identified should clearly be above and beyond the baseline requirements. No restatements of the application or the FOA requirements.
Very Good
Elements are clearly addressed with necessary detail and the evidence is thoroughly supported. Documentation and required information are specific and comprehensive. Any weaknesses identified will likely have minor impact on the successful implementation and execution of the proposed project.
Good
Elements are addressed, although some do not contain necessary detail and/or support. Most documentation and required information are present and sufficient. Application has some strengths but with at least one weakness identified that will likely have moderate impact on the successful implementation and execution of the proposed project.
Satisfactory
Most elements are addressed, although when addressed, do not contain all the necessary detail and/or support. Documentation and required information are deficient. Application has few strengths and some weaknesses and of the weaknesses identified, only one major weakness. The one major weakness could potentially impact the successful implementation and execution of the proposed project.
Poor
Few, if any, elements are addressed. Documentation and required information are deficient or omitted. Application has very few strengths and numerous major weaknesses.

21. What to Expect During the ORC
Day 1 - General Orientation
(All Attendees are expected to be on the web-assisted meeting)
Day 1 – 3
Convene by Panel
Introduction of Panel Reviewers and Staff (Role Call)
Review of Applications Assigned to the Panel
(Following the Steps in Conducting Committee Review)

22. Roles of the Participants
DIR Review Administrator (RA)
The role of the RA in the Division of Independent Review is to coordinate and oversee the review process and to ensure that a fair, equitable and objective review is performed for each eligible application.
Reviewers
The role of the reviewer is to fairly, equitably and objectively review and evaluate each application by clearly documenting strengths and weaknesses.
Program Staff
It is the role of the Program Official to ensure that program’s policies and NOFO requirements are effectively conveyed and evaluated in accordance with the evaluation criteria.

23. Roles of the Participants (continued)
Program Staff It is the role of the Program Official to ensure that program’s policies and NoFO requirements are effectively conveyed and evaluated in accordance with the evaluation criteria. Program staff will provide you with specific program policies/procedures and any legislative requirements today. Program will also be available to answer any questions or concerns you may have during the review. Grants Management Specialist The role of the Grants Office is to provide grant policy and budget development input applicable to assist you in the budget review process. DIR Contractor The Contractor role is to provide logistical support to DIR such as handling honoraria and transmission of review materials, and any other special needs. There is a Logistics Lead and Summary Statement Operator (SSO) assigned to each review panel. The SSO serves as the panels’ Summary Statement Editor.

24. Steps in Conducting Committee Review

25. Steps in Conducting Committee Review (Continued)

26. Sample Summary Statement

27. HRSA Is Seeking Reviewers
All professionals working in or knowledgeable about Health Care Services are invited to register in our Reviewer Recruitment Module database (RRM) 
HRSA wishes to identify more qualified reviewers who have expertise in social, cultural or health care issues of rural, migrant or Native American populations
If you know of anyone who may be interested in lending their expertise in any of these fields to our application review process, please invite them to register in the RRM
The link to RRM is:
(Reading from Slide)

28. Previous Training Modules 1-8
Copy the link below to access all previous Quarterly Reviewer Training Modules
Scroll to the bottom of the page on the right side of the screen to see them

29. Please send any feedback to our e-mailbox:
We Value Your Feedback!
Please send any feedback to our box:

30. Question and Answer Period
Please Follow the Instructions from the Operator to ask a Question

31. Thank you for Attending The Training