1. [R ESAMPLED R ANGE OF W ITTY T ITLES ] Understanding and Using the NRC Assessment of Doctorate Programs Lydia Snover, Greg Harris & Scott Barge Office of the Provost, Institutional Research Massachusetts Institute of Technology 2 Feb 2010

2. Overview 1.Background & Context 2.Approaches to Ranking 3.The NRC Model: A Modified Hybrid 4.Presenting & Using the Results O VERVIEW 2 *NB: All figures/data in this presentation are used for illustrative purposes only and do not represent a known institution.

3. Background & Context A.History of NRC Rankings B.MIT Data Collection Process I NTRODUCTION 3

4. Participating MIT Programs Aeronautics and AstronauticsComputer Science Applied BiosciencesEconomics Applied MathematicsElectrical and Computer Engineering Astrophysics and Astronomy and Planetary Science Geology and Geochemistry and Geophysics Atmospheres, Oceans & Climate including MIT/WHOI Oceanography History, Theory and Criticism Biological Engineering, Health Science and Technology Linguistics Biology/Biochemistry and BiophysicsMaterial Sciences and Engineering Biology/Cell and DevelopmentalMathematics Biology/Genetics and GenomicsMechanical Engineering Chemical EngineeringNeuroscience ChemistryOperations Research Civil and Environmental EngineeringPhilosophy Cognitive SciencePhysics Computer EngineeringPolitical Science 4 I NTRODUCTION

5. Section 2 2.Approaches to Ranking A PPROACHES TO R ANKING 5

6. How do we measure program quality? Use INDICATORS (“countable” information) to compute a rating – Number of publications – Funded research per faculty member – Etc., Try to quantify more subjective measures through an overall PERCEPTION - BASED RATING – Reputation – “Creative blending of interdisciplinary perspectives” A PPROACHES TO R ANKINGS 6

7. Section 3 3.The NRC Approach T HE NRC A PPROACH 7

8. So how does NRC blend the two? The NRC used a modified hybrid of the two basic approaches: In total, a 4-step process, indicator based, by field Process results in 2 sets of indicator weights developed through faculty surveys: – “Bottom up” –importance of indicators – “Top-down” – perception-based ratings of a sample of programs Multiple iterations (re-sampling) to model “the variability in ratings by peer raters.” * T HE NRC A PPROACH 8 *For more information on the rationale for re-sampling, see pp. 14-15 of the NRC Methodology Report

9. So how does NRC blend the two? STEP 1: Gather raw data from institutions, faculty & external sources on programs. Random University (RU) submitted data for its participating doctoral programs. T HE NRC A PPROACH 9 RU Physics IndicatorValue # publications/fac1.07 # citations/article1.17 Median GRE746 Gender diversity44% Time to degree5.67 … RU ChemE IndicatorValue # publications/fac1.07 # citations/article1.17 Median GRE746 Gender diversity44% Time to degree5.67 … RU Economics IndicatorValue # publications/fac1.07 # citations/article1.17 Median GRE746 Gender diversity44% Time to degree5.67 …  NRC

10. So how does NRC blend the two? STEP 2: Use faculty input to develop weights: – Method 1: Direct prioritization of indicators-- “What characteristics (indicators) are important to program quality in your field?” T HE NRC A PPROACH 10 Program Faculty QualityMost Impt Indicator (Mark 4) Top 2 Indicators Number of publications per faculty member Number of citations per publication Racial/ethnic diversity of the student population Avg. # of Ph.D.s granted over last 4 years Gender diversity of program faculty ……… Direct Weights Indicator 1= 0.2 Indicator 2= 0.0 Indicator 3= 0.1 Indicator 4= 0.1 Indicator 5= 0.2 … Calculations

11. So how does NRC blend the two? STEP 2: Use faculty input to develop weights: – Method 2: A sample of faculty each rate a sample of 15 programs from which indicator weights are derived. T HE NRC A PPROACH 11 Program #2: Yale University Economics Some Facts About the Program: # of Ph.D.s 2001-2006: _____Faculty % Female: _____ Median Time to Degree: _____Current Faculty List, etc. Program #2: Yale University Economics Program #2: Ivy University Economics Program #1: Land Grant University Economics Some Facts About the Program: # of Ph.D.s 2001-2006: XXFaculty % Female: YY% Median Time to Degree: Z.ZCurrent Faculty List, etc. On a scale from 1 to 3, indicate your familiarity with this program? ___ 1 (Little or none) ___ 2 (Some) ___ 3 (Considerable) On a scale from 1 to 6, how would you rate this program? ___ 1 (Not adequate for doc educ.) ___ 2 (Marginal) ___ 3 (Adequate ___ 4 (Good) ___ 5 (Strong) ___ 6 (Distinguished) ___ 9 (Don’t know well enough) Regression- based Weights Ind. 1= 0.3 Ind. 2= 0.04 Ind. 3= 0.2 Ind. 4= 0.15 … Principle Components & Regression

12. So how does NRC blend the two? STEP 3: Combine both sets of indicator weights and apply them to the raw data: T HE NRC A PPROACH 12 Direct Weights Ind. 1= 0.3 … Regression- based Weights Ind. 1= 0.2 … Combined Weights Ind. 1= 0.25 … DATA IndicatorValue # publications/fac1.07 # citations/article1.17 Median GRE746 Gender diversity44% Time to degree5.67 … X = Rating RANKED LIST 1. Ivy Univ (98) 2. Random Univ (94) 3. Private Univ (91) 4. Land Grant U (88) 5. Univ of State (87) … 

13. So how does NRC blend the two? STEP 4: Repeat steps 500 times for each field T HE NRC A PPROACH 13 A) Randomly draw ½ of faculty “important characteristics” surveys C) Randomly draw ½ of faculty program rating surveys B) Calculate “direct” weights D) Compute “regression- based” weights E) Combine weights F) Repeat (A) – (E) 500 times to develop 500 sets of weights for each field G) Randomly perturb institutions’ program data 500 times* H) Use each pair of iterations (1 perturbation of data (G) + 1 set of weights (F)) to rate programs and prepare 500 ranked lists I) Toss out the lowest 125 and highest 125 rankings for each program and present the remaining range of rankings *For more information on the perturbation of program data, see pp. 50-1 in the NRC Methodology Report

14. Section 4 4.Presenting & Using the Results R ESULTS 14

15. What are the indicators? Program Faculty QualityStudent CharacteristicsProgram Characteristics # of publications per faculty member Median GRE of entering students Avg. # Ph.D.s granted in past 5 years # of citations per faculty member % students receiving full financial support % entering students who complete Receipt of extramural grants for research % students with portable fellowships Time to degree Involvement in interdisciplinary work Racial/ethnic diversity of student population Placement of students after grad Racial/ethnic diversity of the program faculty Gender diversity of student population % students with individual work space Gender diversity of the program faculty High % of international students % of health insurance premiums covered Reception of peers of a faculty member’s work as measured by honors/awards # of student support activities provided 15 R ESULTS

16. What will the results look like? TABLE 1: Program values for each indicator plus overall summary statistics for the field R ESULTS 16 RU EconAll Economics Programs (n=117) IndicatorValueMin25 th %tile 75 th %tile MaxStd. Dev. # publications/fac1.07.049.369.6551.257.246 # citations/article1.17.153.6841.7715.4851.002 Median GRE74635374079080055 % female students44%0%28.6%42.9%76.9%12% % female faculty12.5%0%10.5%21.1%66.7%9.9% Time to degree5.673568.8 …

17. What will the results look like? TABLE 2: Indicators and indicator weights – one standard deviation above and below the mean of the 500 weights produced for each indicator through the iterative process (and a locally calculated mean) R ESULTS 17 IndicatorMinus 1 SDPlus 1 SDCalculated Mean # publications/fac0.1300.134 0.132 # citations/article0.2940.267 0.2805 Median GRE0.0910.089 0.09 % female students-0.029-0.043 -0.036 % female facultyn.s.* Time to degree-0.026-0.031 -0.0285 … *n.s. in a cell means the coefficient was not significantly different from 0 at the p=.05 level.

18. What will the results look like? TABLE 3: Range of rankings for RU’s Economics program alongside other programs, overall and dimensional rankings R ESULTS 18 InstitutionOverallResearch Activity Diversity of Acad Environ. Student Supp/Outcomes 25 th %tile 75 th %tile 25 th 75 th 25 th 75 th 25 th 75th Ivy Univ3036313237412831 Univ of State45544042 504546 Random Univ4556384247514347 Private Univ4857414240474549 Land Grant U5563596448505461 Total # of ranked programs = 117

19. What will the results look like? TABLE 4: Range of rankings for all RU’s programs R ESULTS 19 ProgramOverallResearch Activity Diversity of Acad Environ. Student Supp/Out comes 1995 NRC Ranking 2009 USNWR Ranking 25 th 75 th 25 th 75 th 25 th 75 th 25 th 75 th Linguistics 4556…4038 Material Sciences and Engineering 252624 Mathematics 2123 25 Mechanical Engineering 323633 Neuroscience 34353435 Operations Research 5456 53 Philosophy 4344… 43

20. Q&A 20

21. For more information… The full NRC Methodology Report http://www.nap.edu/catalog.php?record_id=12676 Helpful NRC Frequently Asked Questions Page http://sites.nationalacademies.org/pga/Resdoc/PGA_ 051962 R ESOURCES 21