Multi-Dimensional Factors in Academic Research Evaluation Xiaodong Zhang Ohio State University

Impact of Strong Research to Universities  A foundation of improving undergraduate and graduate programs  Timely updating contents in classrooms  Creating research environment and opportunities to students  Bringing competitiveness and recognition  New knowledge, discoveries, and innovations.  Continue to attract strong students, excellent faculty, and research grants.

How is Research Measured?  Research Production is Typically Measured by  Publications in refereed conferences and journals  Software and prototypes  Invited lectures  Patents  Research Awards and honors  Input of Research  Research proposal writing, and research project planing  Research grants  Recruiting and forming teams.

How is Research Impact Measured?  Research results are useful  Citations of the publications  Widely used software and prototypes  Technology transfer from original research results  Useful patents  Influence in the research community  Leading new directions in the field  The quality of research is essential  Number of publications should not be a major metric  Our research should be Impact-driven

Where to Publish?  Why publish in top venues?  Each field has its own flagship and leading conferences/journals defined by reputations  Most researchers only read papers there.  Conferences versus Journals  In certain fields, such as systems, architecture, networking, and databases, top conference papers are prestigious, highly visible, with 4-5 or more reviews in depth, and low acceptance rate.  SCI and EI should NOT be used as a guidance  An SCI/EI entry does not necessarily reflect its quality in the field  The quality should be judged by the peers in the field.  The impact factor sometimes is field-size dependent.

Establishing Identity and Reputation  Conducting research on focused topics in depth  Making names behind some important innovations  US NSF Career Program for Junior Faculty  A 5 year research/education plan, aiming at conducting research to solve a small set significant problems.  A strong department should have several identities  Paying attention to building on existing strength.  Rewarding to strong research identity/reputation

What is an ``Acceptable” Research Production?  How to quantify research productions?  Number vs. quality of publications  Research grants/expenditure (mission vs. basic research)  Supervising Ph.D. students: productivity and quality of placements.  Impact of research after many years.  Conference/journal organizations  Not part of the research, but professional service.  Each department has its formula  Resources dependent (teaching load, et. al.)

Effects of University Rankings  Rankings by society have to be a consideration  Rankings significantly affect student recruiting  University rankings influence parents  Faculty recruiting is also affected.  However, these rankings are a reference not an absolute indicator.

Major Sources of University Ranking in US  US News & World Reports (University ranking)  Published every August.  Based on 6 criteria to quantify comprehensive ability.  Quality of undergraduate students is an important consideration.  US News & World Reports (Graduate school ranking)  Published every March.  Different school rankings are based on different disciplines.  Quality of graduate programs is the major factor.  National Science Foundation (funding statistics)  Research expenditure of last year published every August  Ranking the research scale of universities.  National Research Council (Department ranking)  Published every 10 to 15 years  Ranking all the Ph.D. granted departments in all majors

US News & World Reports’ 6 Criteria  Undergraduate student selectivity  SAT scores, high school ranking, and other activities.  Acceptance rate.  Faculty academic reputation  Research activities and accomplishments  Faculty-student ratio  Reflecting the efforts and attention to students, e.g. class size.  Student retention and graduation rate  Seriousness of the students and management of curriculum.  Financial resources  University revenue and endowed funds.  Alumni donation rate  Reflecting graduates’ proud, passion, and care to the university.

Major Themes of Graduate Program Ranking  Scholarships and impacts  Quality of faculty publications and their influence to the field  Citations rates and technology transfers  Ph.D. student production and placement  How many per year, and where they go after graduations.  Ph.D. alumni achievements  The group distinguishing themselves in the fields.  Faculty resources  Award winners, members of the academies, and others.  Research scale  Measured by the amount of expenditure by external grants

Foundations of Academic Excellence  Excellent undergraduate programs  Highly selected students with comprehensive talents.  Providing highly quality education and research environment.  All top U.S. universities have top undergraduate programs.  Excellent research and education infrastructure  Students are able to effectively learn and research.  Excellent faculty  World-class professors who are well established in their fields.  High standards in research and education  Rigorous training to students and doing impact driven research.  Ambitious students who will play leadership anywhere  Enter: grow the wisdom; Depart: serve the better country and the kind.  Should not be too narrowly focused.

Obstacles to Establish World-Class Universities in China  Student selection system has a lot of limits  The exam system narrowly focuses on knowledge learning  Selected ``best students” may not be innovative and ambitious.  Research is too much economy development driven  Too many mission oriented projects with low research values  Faculty selection system is not rigorous and rewarding  Not very selective, and not as prestigious as administrators.  SCI papers driven work will never lead to world class.  Only impact driven work is the foundation of scholarships.  The centralized control limits our vision and steps  Lacking freedom and competitions, and peer reviews.  A selection of university president other leading administrators is tightly controlled  When can we openly select university leaders worldwide?

Mr. Hu’s 4 Conditions of an ``Established University”  In 1947, Hu Shi urged the government to invest 5 existing universities to make them be ``established and independent” in the world within 10 years.  1. Providing first-class education environment.  2. Being able to retain young researchers.  3. Being able to solve country’s problems in science, industries, health, and national defense.  4. Being able to collaborate with foreign scientists to solve open and hard problems.

Considerations of Building World Class Universities  High selectivity of students far beyond exams  Building high quality faculty: life term scholars.  Research standard should be defined by peers not by upper administrations for easy management.  Basic research projects should not be driven by economic development missions.  Basic research grants should be dominant in universities, not mission oriented funds.  University governance should be professors based.  Recruiting presidents, provosts, deans, and department chairs should be open to the world.