1. The NRF rating system and its appeals: some facts and some fallacies Duncan Mitchell

2. ● I am a theoretical physicist and the panel consists only of experimental physicists, so they can’t assess me ● My h index is better that that of five B-rated researchers I know, but the committee gave me a C ● I was elected Vice-Chairman of the International Society of Orchid Botanists so I must have an international reputation ● My research has improved but my rating has gone down

3. Reviewers ● The only assessors of research output ● Your choices and the specialist committee’s choices ● Excluding potential reviewers Pick a likely reviewer and write your application for her/him

4. The reviewers’ focus ● Your best outputs of the review period ● Peer-reviewed journal articles ● Other legitimate research outputs ● Postgraduate students Be careful out about conference proceedings Teaching textbooks are not research outputs

5. Scoring the reviewers’ reports ● Familiarity with the research field ● Evidence-based assessment of the applicant’s research in the review period ● Bias and hostility ● Halo effect ● Congruence with data in the application Good reports count more

6. Arriving at the rating ● Enough usable reports? ● Search for the language: …… coherent body of work? …… promise, leadership? …… significant international impact? ● Agreement between committee, assessor and chair Count up the A, B, C, P, Y, RU reports (weighting the good reports)

7. The Appeals Committee ● Mandate ● Membership ● Evidence: ……. appeal letter..…... RRAP sheets ……. documents that served before the specialist committee/ Executive Evaluation Committee ……. and it has the right to call for more reports

8. The Appeals Committee’s deliberations ● Is the appeal legitimate? ● What issues in the letter need a response ? ● Did the rating process follow the rules? ● Were there enough usable reviewers’ reports? ● Was the set of reviewers (with usable reports) balanced? Does the Appeals Committee agree with the rating decision?

9. The difficult issues ● Applicants on a cusp ● Established early-career researchers ● Members of large research teams ● Researchers competent in more than one field of research ● Researchers still active but past their peaks

10. ● I am a theoretical physicist and the panel consists only of experimental physicists, so they can’t assess me ● My h index is better that that of five B-rated researchers I know, but the committee gave me a C ● I was elected Vice-Chairman of the International Society of Orchid Botanists so I must have an international reputation ● My research has improved but my rating has gone down