National Research Council Assessment of Research Doctorate Programs Summary of Methodology Michigan State University

What is it? National Academies of Science “rate” research doctorate programs approx. every 13 years (1982, 1995, 2009) was largely a reputational (only) survey. In 2003, NRC published a study on the methods used in 1995 with recommendations on changes for data collection (for 2009 ratings). 2

Scope and Coverage of the Current Study 222 institutions 61 fields (MSU ranked in 54 fields) 5006 programs across the 61 fields Each field of study had produced at least 500 Ph.D.s in the 5 years prior to To be included, each University program must have granted 5 Ph.D. degrees in the 5 years prior to

Current NRC study The taxonomy of fields was hotly debated. Used CIP codes (US Dept of Ed taxonomy), NSF fields, & info from scholarly societies. It is what it is! Not very useful for interdisciplinary programs as faculty effort was sub-divided. Universities use different names for similar programs. Learn more: 4

54 MSU programs in current NRC study Agricultural Economics*Electrical EngineeringMechanical Engineering American Studies*EnglishMicrobiol & Molec Genetic Animal ScienceEntomology*Music Education AnthropologyEnvironmental ToxicologyNeuroscience Astrophysics & Astronomy*Fisheries & WildlifePathobiol & Diagnostic Inv Biochem & Molecular BiologyFood Science*Pharmacology/Toxicology Biosystems EngineeringForestry*Philosophy Cell & Molecular BiologyGeneticsPhysics Chemical EngineeringGeographyPhysiology ChemistryGeological SciencesPlant Biology Civil & Environmental EngHistoryPlant Breeding & Genetics CommunicationsHorticulturePlant Pathology Comm Arts Media & Info StudiesHuman NutritionPolitical Science Computer ScienceKinesiologyPsychology Criminal JusticeLarge Animal Clinical SciSociology Crop & Soil SciencesLinguisticsSpanish Lang, Lit, Culture Ecology, Evol Bio, & BehaviorMaterials Science & EngStatistics EconomicsMathematicsZoology 5

Current NRC study Faculty effort assignment to program(s) (not always based on tenure/salary home unit): –CORE (supervise dissertations and/or on admissions/curriculum committees for PhD) –NEW (hired in previous 3 years & expected to become Core) –ASSOCIATED (not Core in program, but regular faculty at institution) Note: A faculty member’s program assignment may be divided between various programs, but will always total 100% Dept chairs made final decisions. 6

Current NRC study Data collection was in Data points collected over various spans of time (Question G) Approx. 25%+ of MSU faculty changed between and present Approx. 94% questionnaire participation rate by MSU faculty 7

Five Questionnaires 1.Institutional Questionnaire: U practices and a list of doctoral programs 2.Program Questionnaire: Characteristics of students, faculty, program 3.Faculty Questionnaire: Faculty work history, grants, pubs, and important characteristics of a quality program (Question G) 4.Student Questionnaire: Students ( post comps in English, Chem Eng, Economics, Physics, Neuroscience ) background, faculty interactions, & post-graduation plans 8

Questionnaires 5. Rating Questionnaire: Asked faculty to rate programs in their field Raters classified by rank, geographic region, faculty size in program. Each faculty member rated 15 programs with data provided. Not asked about basis for rating. 9

Current NRC study In addition to the questionnaires, publications and citations data were collected thru ISI database Humanities CVs submitted with faculty questionnaire were used to count books and publications. Honors and awards data came from 224 scholarly societies. 10

The 20 Variables (Question G on Faculty Questionnaire) Publications per Allocated Faculty*, (going back to 1986 for faculty in humanities fields) Average Citations per Publication (citations in to articles dating back to 1981—for all fields except the humanities) Number of Grants per Allocated Faculty* Percent Interdisciplinary (% Associated Faculty) Percent Non-Asian Minority Faculty for Core or New Faculty, 2006** Percent Female Faculty for Core or New Faculty, 2006** Awards per Allocated Faculty* Average GRE, (Verbal measure for the humanities, Quantitative measure for all other fields) Percent students receiving full support in the first year, (fall 2006) Percent first year students with external funding, 2006 *Faculty members who served in more than one program were allocated to those programs based on whether they were core in the program and the share of that program of total dissertations supervised. 11

The 20 Variables (continued) Percent Non-Asian Minority Students, 2006 Percent Female Students, 2006 Percent International Students, 2006 Average annual PhDs graduated 2002 to 2006 Average completions (8 year completion percentage for humanities fields, 6 years for other fields) Time-to-Degree (for Full and Part Time graduates) Percent PhDs with definite plans for an academic position, (including postdoctoral fellowships) Student Work Space [1 = 100% of students with workspace, -1 if <100% of students with workspace] Health Insurance [1= provides health insurance, -1 = does not provide health insurance] Student Activities (number offered from a list of 18) ** “Core” Faculty are those whose primary appointment is in the doctoral program. “New” faculty are faculty with tenure track appointments who were appointed in the past 3 years. 12

2009 NRC Methodology report MSU task force of statistical/survey experts to study the report and provide input: Brian Silver -- Director, Center for Statistical Consulting, Political Science Neal Schmitt -- Chair, Department of Psychology Les Manderscheid -- Ag, Food and Resource Economics and the Graduate School Mary Black -- Assistant Director, Office of Planning and Budgets Kyle Sweitzer -- Data Analyst, Office of Planning and Budgets CONVENER: Karen Klomparens -- Dean, Graduate School Methodology was complex and valid, with sources of uncertainty dealt with appropriately. PROGRAMS WILL PROVIDE FURTHER INTERPRETATION OF THE DATA 13

NRC 2006 Methodology 14

Ranges of overall ratings two methods 1.Directly: Faculty chose the most important quality characteristics from Question G 2.Reputational rating: statistically related faculty ratings to the Question G variables. Combined the direct and reputational weights, rank ordered. Used first and third quartiles of these ratings to yield a RANGE OF RATINGS for each program 15

Additional information No student outcomes are reflected in the overall rating, so 3 supplemental measures added NRC preparing a way for us to weight variables differently and run own ratings Not reporting data with less than 5 per cell Combined small progs with others to calculate weights Preparing a way for students to use data when considering programs 16

Supplemental Ratings Research Impact –Publications/faculty member –Citations/publication –Percent of faculty holding grants –Honors and awards per faculty member Student Support and Outcomes –Percent of students having full support in first year –Percent of students with portable fellowships in first year – Percentage of students with RAships – Percentage of students with TAships –Time-to-degree –Percent who complete in 6 years (sciences) or 8 years (humanities) – Placement in an academic position or postdoc after graduation Source: C. Kuh, NAS 17

Supplemental Ratings (2) Diversity of the Academic Environment –Percent of students who are female –Percent of faculty who are female –Percent of students from underrepresented minority group –Percent of faculty from underrepresented minority group –Percent of students who are international Source: C. Kuh, NAS 18

Ranges of Ratings for a Field Programs will be arranged alphabetically and the range of ratings will be given for each. Ranges overlap other ranges for most programs. This means that there may be a number of programs of roughly the same quality. You should identify those similar programs in discussing the quality of your programs. Source: C. Kuh, NAS 19

What information will I receive about the rating calculation for my programs? 1)A list of the values of variables that your program supplied to the NRC or that was calculated from those variables 2)The normalized values for those variables 3)The median combined coefficient (statistical + direct) for each variable and its standard deviation 4)The range of the normalized variable values 5)The range of the combined effects of the coefficients in the random halves calculation Source: C. Kuh, NAS 20

21 TABLE 5-1 Data and Coefficient Table for a Program in Economics Standardized Program Values and Range of Combined Coefficients Institution Name: xxx Program Name: yyy Publications per Allocated FacultyV Cites per PublicationV Percent of Faculty with GrantsV325.50% Percent of Faculty InterdisciplinaryV45.90%-0.641n.s.# Percent of Non-Asian Minority FacultyV57.70%0.547n.s.# Percent Female FacultyV612.50%-0/440n.s.# Awards per allocated facultyV Average GRE-QV Percent 1 st yr. Students w/full supportV % Percent 1rst yr. Students w/portable fellowships V100.00% Percent non-Asian Minority StudentsV %0.069n.s.# Percent Female StudentsV % Percent International StudentsV %-0.509n.s.# Average PhDs 2002 to 2006V Percent Completing within 6 yearsV %-0.638n.s.# Time to Degree full and Part timeV Percent students in Academic PositionsV % Student Work SpaceV1811n.s.# Health InsuranceV1911n.s.# Number of Student Activities OfferedV Column 1 Column 2 Column 3 Column 4 Column 5 Column 6 Combined Coefficients** Program Program Value Description Variable Value Standardized Minus 1 SD TO Plus 1 SD *Col 3 is based on data submitted by the program or calculated from these data. + Col 4 is standardized across all program values in the field, with mean of 0 and variance of 1. ** Col 5 is Minus 1 Standard Deviation from the Mean for the combined coefficients for the field as a whole ** Col 6 is Plus 1 Standard Deviation from the Mean for the combined coefficients for the field as a whole # n.s. in a cell means the coefficient was not significantly different from 0 at the p=.05 level.

22

23

24

25

Analyzing the data by program 1.Identify variables with the largest contribution. These had the greatest effect on the range of ratings by program. 2.Compare your variable values with programs in other institutions. These will be available in an online database. 3.Consider any additional relevant comments about your program and the NRC methods. 26

Next steps The data will be made available hours before they are publically released, and programs may review their data during this time. COMMENTS TO KK! Additional analyses will be conducted upon release of the data (by CIC, AAU, and by you!) This study improved MSU’s ability to have data readily available in a consistent fashion about students and committee memberships. Data collection has expanded to include ALL graduate programs (Master’s and Doctoral) via the GradInfo database. See next slide… 27

28

29

Weights for Supplemental Measures Research Activity Pubs per Cites per Pct FacAwards Broad Field Faculty Pub w grants per fac Biological Sciences Health Sciences Engineering Physical Sciences Agricultural Sciences Social Sciences Humanities Source: C. Kuh, NAS 30

Weights for Supplemental Measures Student Support & Outcomes Pct full Finish Median Student Collect Broad Field Supp 6 Yrs TTD Placmt Outcome Biological Sciences Health Sciences Engineering Physical Sciences Agricultural Sciences Social Sciences Humanities Source: C. Kuh, NAS 31

Weights for Supplemental Measures Diversity of Academic Environment Pct fac Pct fac Pct stud Pct stud Pct stud Broad FieldMinority Female Minority Female Int’l Biological Sciences Health Sciences Engineering Physical Sciences Agricultural Sciences Social Sciences Humanities Source: C. Kuh, NAS 32