U.S. China Computer Science Leadership Summit Beijing, May 23, 2006 Evaluating Research and Impact Guri Sohi John P. Morgridge Professor and Chair Computer Sciences Department University of Wisconsin-Madison

U.S. China Computer Science Leadership Summit Beijing, May 23, 2006 Process of Evaluation Metrics Mechanisms

U.S. China Computer Science Leadership Summit Beijing, May 23, 2006 Metrics A diversity of metrics, depending upon type of research and area. Funding Publications Artifacts –Hardware or software or other

U.S. China Computer Science Leadership Summit Beijing, May 23, 2006 Funding Sometimes necessary but definitely not sufficient Sometimes confused as an end vs. a means Questionable metric for longer-term direct impact –May have indirect impact, e.g., more students trained

U.S. China Computer Science Leadership Summit Beijing, May 23, 2006 Publications Quality vs. quantity Quantity is easily measured; quality harder –Temptation to rely on quantity Use community to help assess quality –Create and cultivate selective publication venues –Larger community helps with assessment

U.S. China Computer Science Leadership Summit Beijing, May 23, 2006 Impact Short-term vs. long-term Temptation to use short-term –Biases toward flavor of the day Measures of impact –Citations –Adoption in commercial products Citations may be lacking –Success of former members of research group Much longer term

U.S. China Computer Science Leadership Summit Beijing, May 23, 2006 Citations Number of citations vs. quality of citations Citation indices, e.g., Science Citation Index or Is Highly Cited –May be less useful for CS due to publication venues considered Modern versions of citation indices e.g., Google Scholar) Only one measure: highly dependent upon area