1. Amir Ghaisi Grigorios Fountas Panos Anastasopoulos Fred Mannering
Statistical Assessment of Peer Opinions in Higher Education Rankings: The Case of U.S. Engineering Graduate Programs in Journal of Applied Research in Higher Education
Amir Ghaisi Grigorios Fountas Panos Anastasopoulos Fred Mannering

2. Importance of University Rankings
Students seek admission to highly ranked universities
Universities use rankings to attract the best students and faculty, instill alumni pride, improve fundraising, etc.
University funding from governmental and private entities can be influenced by rankings
Because of the above, universities develop policies to support and improve their rankings

3. USNews rankings
Easily the most widely followed university rankings in the U.S.
Rankings are based on quantitative data (research expenditures, number of faculty, test scores of students, etc.), peer and recruiter opinions, and predetermined weightings
In USNews rankings of engineering graduate programs peer opinions account for 25% of the total ranking score

4. Peer assessments
Individuals from universities (typically engineering deans) are asked to rate the quality of specific programs on a scale from 1 (marginal) to 5 (distinguished)
Peer scores a function of experiences with faculty and graduates of the schools they are evaluating, the school’s overall reputation, etc.
Peer impressions may also be influenced by their exposure to past rankings, and by the factual information used to support those past rankings

5. Objective of this Research:
Provide insight into the determinants of peer rankings in higher education by developing a statistical model of the average peer assessment scores of U.S. colleges of engineering from the 2018 Engineering Graduate program evaluations provided in USNews
Findings will provide help guide university policies that may be used to improve peer impressions

6. Methodological Approach
Engineering graduate programs are peer-rated on a scale of 1 (marginal) to 5 (distinguished), and USNews provides the resulting average peer assessment scores in tenths (such as 2.6, 2.7, etc.)
Thus the peer score is reported as average that is continuous variable bounded by 1 and 5
This suggests a lower and upper censored Tobit regression would be appropriate, but statistical estimations indicated an uncensored regression was sufficient since few observations are near the censored boundaries

7. Methodological Approach (cont.)
Using a standard uncensored regression:
where, yn is the dependent variable (average graduate program peer assessment score for university n,
is a vector of estimable parameters,
Xn is a vector of explanatory variables for university n,
εn is a normally and independently distributed error term with zero mean and constant variance σ2.

8. Unobserved Heterogeneity
To account for heterogeneity (unobserved factors that may vary across universities), we allow for the possibility that each university may have its own parameter for one or more of the explanatory variables
Estimable parameters are thus written as:
where β is the mean of the parameter estimate and φn is a randomly distributed term (for example a normally distributed term with mean zero and variance σ2)

9. Unobserved Heterogeneity
We also allow for the possibility for the mean and variance of the parameter to be a function of explanatory variables so (Mannering, 2018)
Models are estimated using simulated maximum likelihood methods using 1,000 Halton draws for numerical integration
Observation-specific mean parameters estimated using the simulated Bayesian approach proposed by Greene (2004)

10. Data
peer assessment scores are gathered from USNews’ rating of U.S. engineering graduate programs
USNews-related data provided with their 2018 graduate program rankings including:
- Engineering graduate student enrollment; - Total number of engineering faculty; - Number of faculty in the National Academy of Engineering; - Average math Graduate Record Exam score of students; - Percent of engineering applications accepted; - Number of doctoral students graduated in the past year; - Research expenditures per faculty

11. Data (cont.) Additional data:
- University membership in the American Association of Universities (AAU); - Whether the university is public or private; - National merit scholars admitted, - Incoming students’ average SAT scores, - Number post-doctoral appointees; - Citation information of engineering faculty and university faculty overall:
Google Scholar including total citations, number of documents, h-index, and number of papers cited 10 times or more, etc.

12. Estimation Results 139 universities
13 statistically significant variables (10 produce parameters that are fixed across observations and 3 that vary across observations)
Random parameters linear regression produced an R- squared value of and an adjusted R-squared value of (which accounts for the number of estimated parameters in the model)

13. Observed vs. Model-Predicted

14. College-related attributes
Statistically significant variables and their effect on mean peer assessment scores:
College-related attributes

15. Graduate Record Examination (GRE) scores of incoming graduate students
Positively increases the average peer evaluation
an 8 point increase in the math GRE scores of graduate students would result in roughly a 0.1 point increase in a college of engineering’s average peer assessment

16. Graduate Admissions
Universities that admit more than 31% of their graduate applicants tend to have average peer assessment scores that are less than universities that admit 31% or less

17. Number of doctoral students graduated
Parameter estimate for this variable suggests that graduating 34 more doctoral students would result in a 0.1 increase in the average peer assessment score

18. Number of faculty
Total number of faculty, was also found to positively influence average peer assessment scores, with larger faculty size more likely to result in a higher score
For public universities, the model’s parameter estimate suggests that nearly 60 faculty would need to be added for a 0.1 increase in the average peer assessment score
For private universities, the model’s parameter estimate suggests that about 35 faculty would need to be added for a 0.1 increase in the average peer assessment score

19. Faculty-size threshold
Effects of having less than 200 faculty vary across universities
Having fewer than 200 faculty has negative effect for roughly 68% of the universities and a positive effect for 32%.
This dichotomy is likely picking up the fact that the faculty effect is not just about size, but about quality as well

20. Graduate student enrollment per engineering faculty member
Was found to have a negative effect on average peer assessment scores
This variable produced a statistically significant random parameter
While negative for all universities, the fact that the values vary across universities suggests that unobserved factors relating to student and faculty quality are affecting the influence that this ratio has on average peer assessment

21. Graduate student enrollment per engineering faculty member
Was found to have a negative effect on average peer assessment scores
This variable produced a statistically significant random parameter
While negative for all universities, the fact that the values vary across universities suggests that unobserved factors relating to student and faculty quality are affecting the influence that this ratio has on average peer assessment

22. National Academy Members
Percent of NAE members positively influences peer scores, but effect is highly variable across universities
The effect ranges from a 0.1 increase in the average peer assessment score due to a 1.5% increase in NAE membership, to virtually no effect on the average peer assessment score
NAE effect is likely highly variable due to NAE faculty being at different stages in their career, the variance in academic credentials of NAE members

23. Faculty h-index
Google Scholar h-index of the 10th most cited engineering faculty member has a positive effect on average peer assessment scores
By definition, an author’s h-index value indicates that the author has published h papers, each of which has been cited at least h times
While many colleges of engineering in the U.S. have been slow to adopt citation information such information has a statistically significant effect on average peer assessment scores

24. Overall university-related attributes
Statistically significant variables and their effect on mean peer assessment scores:
Overall university-related attributes

25. American Association of Universities (AAU) membership
AAU membership of the University improved peer assessment score of holding all else constant
AAU membership requires that a wide variety and high level of research and scholarly output be achieved by the university

26. Highly-cited university faculty
Universities having 5 or more faculty at the university, as a whole, with more than 50,000 Google scholar citations enjoy higher peer rankings (an average higher peer assessment scores) than those universities that do not have as many of these faculty
Reflects highly-cited faculty and the university’s position in highly citable fields

27. Number of post-doctoral researchers
The average number of annual postdoctoral researchers employed at the university as a whole (averaged over the past 15 years) positively influences the average peer assessment score of the college of engineering
The number of post-doctoral students is a factor in AAU membership and other measures of university reputations

28. SAT scores of incoming students
The past 10-year average of math plus verbal (1600 maximum) Scholastic Aptitude Test (SAT) scores of students entering the university positively influence the average peer assessment scores of engineering programs
A 150 point increase in the average SAT score of all university students will result in a 0.1 increase in the average peer assessment score of the university’s college of engineering

29. Summary and Conclusions
This paper explored the factors influencing the USNews average peer assessment scores of U.S. engineering programs by estimating a random parameters linear regression using data extracted from a number of sources
Estimation results show that both College and University factors influence peer assessment scores, and the influence of some of these factors varies across universities

30. Summary and Conclusions (cont.)
Research funding itself was found to be statistically insignificant (although it is needed to generate many of the variables found to be significant)
This is a wake-up call to the many universities who blindly pursue research dollars without carefully thinking about the scholarly productivity that such dollars can potentially produce

31. Summary and Conclusions (cont.)
U.S. engineering colleges have tended to use promotional brochures, website enhancements, and other means to influence how peers assess them
But our statistical analysis suggests that peer assessments are rooted much deeper in the measurable accomplishments of the college’s faculty, the quality of its students, and the quantifiable achievements of its university as whole