On the Use of Probing-Question Approach in Grant Application Peer Reviews 谈项目立项专家评审中探究式提问手段的应用 Ping Sun （孙 平 ) ISTIC / ORI, MOST Dalian,China· May 21-23, 2012

1. Targeted problems 2. The fuci of relevant researches in China 3. Concrete measures are needed 4. Simple strategies might also be effective 5. Summary

1. Targeted problems “As is the case with any process, peer review is not an infallible system and to a large extent depends on the integrity and competence of the people involved and the degree of editorial oversight and quality assurance of the peer review process itself.” —— A UK House of Commons Committee report

Lack of Integrity and competence Agreeing to review an application beyond his/her competence. Entrusting review tasks to someone else without adequate approval. Not disclosing conflict of interest. Irresponsible behaviors, e.g. scores given and comments made are inconsistent; comments made are not based on careful scrutiny of the applications ……

A mail reviewer is always right! Characteristics of the applications Extensive presentation  Brief presentation  Theoretically focused  Details included  Focus emphasized  Many factors listed  Possible comments of the reviewers Not focused Lack of in-depth understanding Lack of details Focus to much on tools and methods Other factors ignored Too ambitious, not focused

The suspicions of an failed applicant (NSFC now provides full-text comments) The first reviewer seems not read the research proposal carefully. The second reviewer is not familiar with the concepts and contents. The third reviewer is an expert in the field and did read the proposal, but the inquiry made (“ previous articles were almost all published on a single journal ) is not reasonable.

Some causes of the problems Insufficient communication between applicants and reviewers The review process is not transparent, and the outcomes are un-appealable. The supervision of the peer review process is not in place. Personal factors of reviewers (COI, etc) The accountability of reviewers is not emphasized.

(Continued) Struggling Reviewers Many peers have to review more than 50 grant applications altogether (Xu Gong). Some reviewers are not familiar with the novice theories, methods, tools, and have to take more time. Some applications are not well presented, so the decisions are hard to make. Some reviewers are not quite sure about their decisions due to various reasons.

2. The Foci of relevant researches in China The development of principles and norms related to peer review The effectiveness and fairness of peer review process and outcomes The criteria and selection of reviewers The assessment of reviewers’ credit (integrity, competence, performance, etc) The control of conflict of interest (COI), including the COI of review organizers

3. Concrete measures are needed To address some major problems —— The mismatch of the criteria and the practical selection of reviewer, resulting in the qualifications of reviewers are not always guaranteed. The credit records of reviewers are scarce and not necessarily fit for use. The unpredictable nature of reviewers’ behaviors in a particular review process.

4. Simple strategies might also be effective The proposed practice —— To encourage reviewer to pose one or two questions to the applicants to clarify or explain something, if necessary. The questions will be sent via an automatic relay system, with address of the sender concealed. The communications between reviewers and applicants will be archived.

( Continued) What can be achieved —— A serious reviewer will be able to obtain additional information (interdisciplinary), without increasing the burden of anyone. A less serious reviewer will be more careful, at least with the questions asked. The records of the communications can be analyzed and examined for different purposes, if desirable.

( Continued) What differences can be made —— A source of reviewers’ credit information, collected and stored centrally The assessment and adjustment of reviewers can be made more easily. The reviewers’ awareness of accountability might be raised. The quality of peer review might be improved with extra information.

1. The use of probing-question approach is simple, but potentially effective in many aspects. 2. The approach will add positive personal experiences for both reviewers and applicants (to be supervised vs. the applications to be taken seriously). 3. The approach can promote integrity and the review quality at the same time. 5. Summary

Any comment? Do the problems listed also exist in Western countries? Is the approach a practical one?

Thank you!