1. NIH Peer Review of Small Business Applications 11th NIH SBIR/STTR Conference July 2009 National Institutes of Health U.S. Department of Health and Human Services Dana Jeffrey Plude, Ph.D. Acting Director Division of AIDS, Behavioral and Population Sciences

2. NIH DUAL PEER REVIEW Scientific Review Group Evaluate scientific and technical merit Recommend level of support, duration Does not make funding decisions Advisory Council Reviews the review Funding recommendations to Institute/Center Evaluate priorities/relevance Advise on Policy Funding decision made by Institute/Center 2

3. Translational and Clinical Sciences Translational and Clinical Sciences Cardiovascular and Respiratory Sciences Cardiovascular and Respiratory Sciences Surgical Sciences, Biomedical Imaging and Bioengineering Surgical Sciences, Biomedical Imaging and Bioengineering Musculoskeletal, Oral And Skin Sciences Musculoskeletal, Oral And Skin Sciences Oncology 2 – Translational Clinical Oncology 2 – Translational Clinical Vascular and Hematology Hematology Physiological and Pathological Sciences Physiological and Pathological Sciences Endocrinology,Metabolism, Nutrition & Reproductive Sciences Reproductive Sciences Endocrinology,Metabolism, Nutrition & Reproductive Sciences Reproductive Sciences Immunology Immunology Infectious Diseases & Microbiology Infectious Diseases & Microbiology Digestive, Kidney & Urological Systems Digestive, Kidney & Urological Systems Neuroscience, Development and Aging Neuroscience, Development and Aging Brain Disorders & Clinical Neuroscience Brain Disorders & Clinical Neuroscience Molecular, Cellular & Developmental Neuroscienc Developmental Neuroscienc e Molecular, Cellular & Developmental Neuroscienc Developmental Neuroscienc e Integrative, Functional & Cognitive Neuroscience Integrative, Functional & Cognitive Neuroscience Emerging Technologies & Training in Neuroscience Emerging Technologies & Training in Neuroscience Biology of Development & Aging Biology of Development & Aging Biobehavioral & Biobehavioral & Behavioral Processes Behavioral Processes Biobehavioral & Biobehavioral & Behavioral Processes Behavioral Processes Risk, Prevention& Health Behaviors Risk, Prevention& Health Behaviors Population Sciences & Epidemiology Epidemiology Healthcare Delivery & Methodologies Healthcare Delivery & Methodologies AIDS & Related Research AIDS & Related Research AIDS, Behavioral and Population Sciences AIDS, Behavioral and Population Sciences Basic and Integrative Biological Sciences Basic and Integrative Biological Sciences Biological Chemistry & MacromolecularBiophysics MacromolecularBiophysics Bioengineering Sciences & Technologies Bioengineering Sciences & Technologies Genes, Genomes & Genetics Genes, Genomes & Genetics Oncology 1 – Basic Translational Translational Cell Biology InterdisciplinaryMolecular & Training InterdisciplinaryMolecular & Training CSR Review Divisions

4. PEER REVIEW IN CSR Scientific Review Groups/Study Sections/Review Committees/Special Emphasis Panels - Small Business applications are reviewed in Special Emphasis Panels. Scientific Review Officer is responsible for the management of the meeting. Chair and 10 - 24 members who are from academia and small businesses 40 - 100 applications are reviewed at each study section meeting (face to face meeting). CSR is experimenting with review platforms – video conferences, asynchronous extended discussion, and others. 4

5. What It Looks Like: Video Enhanced Discussions

6. What It Looks Like: Asynchronous Electronic Discussions

7. SCIENTIFIC REVIEW OFFICER Performs administrative and technical review of applications to ensure completeness Selects reviewers based on broad input Assigns reviewers (generally at least two reviewers and one reader per application) Manages review meeting/Designated Federal Official Prepares Summary Statement Provides requested information about review recommendations to Institutes/Centers and advisory councils 7

8. CRITERIA FOR SELECTION OF PEER REVIEWERS Demonstrated scientific expertise Doctoral degree or equivalent Mature judgment Work effectively in a groups Breadth of perspective Impartiality Interest in serving Adequate representation of women and minority Scientists Geographic distribution Small Business representation 8

9. REVIEW MEETINGS Closed to the public Introductions SRO instructions: Confidentiality, conflict of interest, new NIH policies, procedures Approximately 60% of the applications are Discussed, the ‘less competitive’ 40% of the applications are Not Discussed Applications are discussed in ‘score order’ based on Preliminary “Overall Impact Score” given by the three assigned reviewers Note - any member can nominate an application for discussion 9

10. REVIEW MEETINGS (cont). Individual review of top 60% applications Conflicts excused Assigned reviewers/readers: preliminary scores, evaluations General discussion, including human subjects, vertebrate animals, biohazard issues Private assignment of priority score Discussion of budget, data sharing, resource sharing issues “Mock” Study Section video http://cms.csr.nih.gov/ResourcesforApplicants/Inside theNIHGrantReviewProcessVideo.htm http://cms.csr.nih.gov/ResourcesforApplicants/Inside theNIHGrantReviewProcessVideo.htm 10

11. REVIEW LOGISTICS Reviewers receive applications and assignments one to two months in advance of meeting. Reviewers post preliminary scores and critiques on a secure web site in advance. Reviewers may see other critiques only after their own are posted. Are not aiming for consensus but outliers must explain their views Critiques are modified in light of discussion. SRO prepares Resume and Summary of Discussion. 11

12. Enhancing Peer Review “Fund the best science, by the best scientists, with the least administrative burden…” Elias Zerhouni, MD, Former Director, NIH Enhancing Peer Review “Fund the best science, by the best scientists, with the least administrative burden…” Elias Zerhouni, MD, Former Director, NIH 2008: The Year of Peer Review

13. Priority 4: Continuous Review of Peer Review Priority 1: Engage the Best Reviewers Increase Flexibility to Better Accommodate Reviewers Recruit Reviewers Acknowledge Reviewers more formally Compensate Reviewers Time and Effort Improve Review Quality with Training Priority 2: Improve the Quality & Transparency of Review Modify Rating System to Focus on Specific Review Criteria Align Summary Statement with Review Criteria Shorten and Align Application with Review Criteria Priority 3: Ensure Balanced & Fair Reviews Across Scientific Fields and Career Stages Support for Early Stage Investigators Review of Established Investigators Enhanced Review of Clinical Research Expand Awards Encouraging “Transformative Research” Reduce Need for Resubumissions Recommendations

14. Amended Applications: To speed the funding of meritorious science and minimize reviewer burden: single amendment (A1)As of January 25, 2009, all original new applications (i.e., never submitted) and competing renewal applications will be permitted only a single amendment (A1).

15. Balanced and Fair Reviews Across Career Stages and Scientific Fields New Investigator:New Investigator: Not previously competed successfully as PD/PI for a significant NIH independent research award. Early Stage Investigator:Early Stage Investigator: Within 10 years of completing terminal research degree or is within 10 years of completing medical residency (or the equivalent). These Designations are IRRELEVANT for Small Business applications – only R01 research grant applications are coded for New PI/ESI

16. Scoring To improve the transparency of the scoring process: five review criteria using a scale of 1-9Score applications on five review criteria using a scale of 1-9. overall impact score using 1-9 scalePreliminary overall impact score using 1-9 scale. Discussed applications will receive an overall score from each eligible (i.e., without conflicts of interest) panel member and these scores will be averaged to one decimal place, and multiplied by 10. The 81 possible priority scores will thus range from 10-90. Percentiles will be reported in whole numbers. All applications will receive scores:All applications will receive scores: Not Discussed Not Discussed applications will receive initial criterion scores from the three assigned reviewers

17. Goal: Improve the transparency of the scoring process: Score applications on five review criteria using a scale of 1-9. Preliminary overall impact score using 1-9 scale. oShould not be the average of the criterion scores. oNot Discussed applications will receive initial criterion scores from the three assigned reviewers. Scoring on a 9-Point Scale

18. Scoring Descriptions ImpactScoreDescriptorStrengths/Weaknesses High Impact 1Exceptional Weaknesses 2Outstanding 3Excellent Moderate Impact 4Very Good 5Good 6Satisfactory Low Impact 7Fair 8Marginal 9Poor Strengths

19. Scoring Expectations: Reviewers participate in entire meeting. To rank the applications in order of review based on preliminary score, all applications must be scored; applications with initial scores of 7-9 are those unlikely to be discussed. Once at the meeting, the full range of scores is available to assign an overall impact score for all discussed applications.

20. Scoring (cont.) Criterion scores are considered part of the critique and will not be discussed at the review meeting. They may be changed in the EDIT phase in IAR.

21. Clustering Phase I, Phase II and Fast Track Applications will be clustered together (provided sufficient numbers). Phase I: FEASIBILITY Phase II: EFFICACY Fast Track: Both Components

22. Clustering of Clinical Applications The goal of clustering clinical applications in review is to increase fairness in the process of reviewing clinical applications. A clinical application will be defined as human subjects research minus Exemption 4.

23. Order of Review Goal: To discuss applications in order of average preliminary score. Why? Concern: variation of scores during different times of the meeting. One recommendation was to recalibrate scores at the end of the meeting. The Solution Recalibrate “dynamically” throughout meeting.

24. Order of Review For calibration purposes… Begin meeting by discussing the best scored application (any activity code). oAll Activity codes are clustered if f feasible (if at least 10 discussed, may include R41/43, R42/44 and Fast Track as groups that can be clustered ).

25. Order of Review Summary Discussion order is based on the average of the impact scores from assigned reviewers. Final scores of discussed applications may differ from preliminary scores as re-calibration happens dynamically.

26. Discuss ~ 40% of small business applications. SRO will then ask if there are any other applications that panel wishes to discuss. The remaining applications will not be discussed (applications receive criterion scores only). Not Discussed

27. Final Scores Discussed applications will receive an overall score from each eligible (i.e., without conflicts of interest) panel member and these scores will be averaged to one decimal place, and multiplied by 10. The 81 possible priority scores will thus range from 10-90. Percentiles will be reported in whole numbers.

28. Summary statement will be shorter and more focused. Discussed applications will also have a summary of the panel’s discussion at the meeting. ALL applications will be scored. Not discussed applications will receive criterion scores only. Summary Statements

29. Critiques To improve the quality of the critiques and to focus reviewer attention on the review criteria: Electronic template for critiquesElectronic template for critiques that will prompt for strengths and weaknesses for each criterion. shorterSummary statement that will be shorter and more focused. summary of discussionDiscussed applications only also will have a summary of the panel’s discussion at the meeting. ALL applications will be scoredIn 2009, ALL applications will be scored. criterion scoresNot discussed applications will receive criterion scores only in addition to the reviewers’ critiques

30. Template-Based Critiques Critique template contains a total of 18 boxes. Reviewers should provide text for only those criteria that are applicable. 1. Overall Impact 7. Resubmission13. Biohazards 2. Significance 8. Renewal14. Budget and Period of Support 3. Investigator(s) 9. Revision15. Select Agents 4. Innovation 10. Protection of Human Subjects 16. Applications from Foreign Organization 5. Approach 11. Inclusion of Women, Minorities, and Children 17. Resource Sharing Plan 6. Environment 12. Vertebrate Animals18. Additional Comments to Applicant

31. Template-Based Critiques Goal: Write evaluative statements and to discourage summarizing the application. Comments should be in the form of bullet points or, if necessary, short narratives. Do not record scores on the critique template. The entire template is uploaded to IAR to become part of the summary statement. 1. Significance Please limit text to ¼ page Strengths  Weaknesses 

32. REVIEW CRITERIA Specifically tailored for small business applications: Significance Investigator(s) Innovation Approach Environment Overall Impact – weigh as appropriate, consider impact on scientific field 32

33. SIGNIFICANCE Does the project have commercial potential to lead to a marketable product, process or service? If the aims of the project are achieved, how will scientific knowledge or clinical practice be advanced? Does this study address an important problem and what commercial and societal benefits might be derived from the proposed research? What will be the effect of these studies on the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field? Does the project lead to enabling technologies (e.g., instrumentation, software) for further discoveries? Will the technology have a competitive advantage over existing/alternate technologies that can meet market needs? 33

34. INVESTIGATORS Is the PD/PI appropriately trained and capable of coordinating and managing the proposed project? Are the investigators well suited to carry out this work and does the investigative team bring complementary and integrated expertise to the project (if applicable)? Is the work proposed appropriate to the experience level of the PD/PI and other researchers, including consultants and subcontractors (if any)? Are the relationships of the key personnel to the small business and to other institutions appropriate for the work proposed? 34

35. INNOVATION Are the aims original and innovative? Does the project challenge existing paradigms or clinical practice; address an innovative hypothesis or critical barrier to progress in the field? Does the project develop or employ novel concepts, approaches, methodologies, tools, or technologies for this area? 35

36. APPROACH Are the conceptual or clinical framework, design, methods, and analyses adequately developed, well- integrated, and appropriate to the aims of the project? Is the proposed plan a sound approach for establishing technical and commercial feasibility? Does the applicant acknowledge potential problem areas and consider alternative strategies? Are the milestones and evaluation procedures clearly described and appropriate? For applications designating multiple PD/PIs, is the leadership approach, including the designated roles and responsibilities, governance and organizational structure consistent with and justifies by the aims of the project and the expertise of each of the PD/PIs? 36

37. ENVIRONMENT Is there sufficient access to resources (e.g., equipment, facilities)? Does the scientific and technological environment in which the work will be done contribute to the probability of success? Do the proposed studies benefit from unique features of the scientific environment, or subject populations, or employ useful collaborative arrangements? Is there evidence of institutional support? 37

38. SPECIAL REVIEW CRITERIA Resubmissions Responses to comments from the previous review Improvements in application Phase II Applications Progress in Phase I – objectives met, feasibility demonstrated Commercialization Plan Commercial potential Fast Track Applications Measurable goals/milestones Commercialization Plan Commercial potential Funding commitments/resources 38

39. ADDITIONAL REVIEW CRITERIA Protection of human subjects from research risk Inclusion of women, minorities and children Care and use of vertebrate animals Biohazards Problems in these areas will have a deleterious impact on the review outcome/score 39

40. OTHER REVIEW CONSIDERATIONS Budget – amount, duration Plans for Data Sharing if required Plans for Model Organism sharing Genome Wide Association Studies – plans for sharing data generated These do not contribute to the score 40

41. REVIEW OUTCOME Not recommended for further consideration – rare, usually due to serious ethical or safety concerns Deferral – also rare, study section does not have sufficient information to make a determination Not Discussed 40% (approximately) Scored 60% (approximately) 41

42. SUMMARY STATEMENT Not Discussed/Priority Score Overall Resume and Summary of Discussion if Discussed Essentially unedited critiques Budget recommendations Administrative Notes Available in Commons only to Principal Investigator 42

43. Example 1 – Phase II project PROTECTION OF HUMAN SUBJECTS UNACCEPTABLE RESUME AND SUMMARY OF DISCUSSION: The proposed project is intended to provide web-based education to and collaboration between parents and teachers of children with ADHD. The proposed program addresses a timely and important problem and if effective, this product would be a welcomed tool. The investigative team is exceptionally well qualified, Phase I successfully demonstrated the program’s feasibility and Phase II is well poised to refine the prototype. During discussion, however, the Committee noted several issues that limit the potential of the Phase II project. For example, the proposed assessment of the program’s effectiveness does not rely on parents and teachers of students with ADHD, and it does not assess the actual collaborative experiences of parent-teacher dyads that use it. Another concern centers on the plan to provide parents with clinical tools without proper guidance from mental health professionals. …There is also some question about the feasibility of creating a product that is relevant and useful for the full range of experience with ADHD that characterizes the target audience. 43

44. Example – Fast Track project RESUME AND SUMMARY OF DISCUSSION: The proposed project develops a brief screen for depression in antenatal and postpartum women. Although it is unclear how feasible it is to obtain a reliable screening tool that is shorter than the 10-item Edinburgh Postpartum Depression Scale, if the project is successful the product would provide an important clinical tool. The project is especially significant because there is not an adequate scale to assess depressive symptoms among pregnant women who have physical symptoms. Using item response theory to develop the screener is innovative and will potentially result in a highly marketable product. The application outlines a strong commercialization plan. There are, however, some aspects of the proposed methodology that are questionable and there is concern that all of the members of the expert panel are directly involved in the project. Also Phase II lacks a gold standard assessment of depression for comparison with the final product. … These issues notwithstanding, the proposed product is innovative and, if successful, would be a significant contribution to the field. 44

45. ADVISORY COUNCIL/BOARD CONSIDERATION Second required step in dual peer review process Most applications considered in en bloc action; a few are specifically discussed (budget change, deferral) 45

46. QUESTIONS