1. Columbia University :: Office of the Provost :: Planning and Institutional Research 2006-7 NRC Assessment of Research-Doctoral Programs October 27, 20061 The National Research Council Assessment of Research Doctorate Programs, 2006-07 Assessment Overview and Implications for Columbia Lucy Drotning Associate Provost for Planning and Institutional Research John Scanlon Data Manager, Planning and Institutional Research

2. Columbia University :: Office of the Provost :: Planning and Institutional Research 2006-7 NRC Assessment of Research-Doctoral Programs October 27, 20062 Background 1.Collection of quantitative data through questionnaires 2.Data on faculty publications, citations, and research activity and student dissertation keywords 3.Faculty opinion on the relative importance of measures of program quality Who is doing what? NRC Mathematica Policy Research (MPR) Columbia Goal The stated goal of the study is to “help universities improve the quality of these programs through benchmarking; provide potential students and the public with accessible, readily available information on doctoral programs nationwide; and enhance the nation's overall research capacity.” The study will consist of:

3. Columbia University :: Office of the Provost :: Planning and Institutional Research 2006-7 NRC Assessment of Research-Doctoral Programs October 27, 20063 End results: what will the study produce? A quantitatively based methodology for rating graduate programs Online database of graduate programs available to public Expert analyses and reports

4. Columbia University :: Office of the Provost :: Planning and Institutional Research 2006-7 NRC Assessment of Research-Doctoral Programs October 27, 20064 Process and Timeline NRC StagesScheduleStatus @ Columbia 1. Collection of Program DataDecember 15, 2006*In process 1.a. Collection of the program listsAugust 2006Submitted 1.b. Collection of Faculty names by programDecember 1, 2006In process 1.c. Collection of the names of admitted to candidacy students in 5 fields. December 1, 2006In process 2. Collection of Institutional DataDecember 15, 2006In process 3. Collection of Faculty Data Questionnaire sent to all faculty identified in 1.b. December-February 2006 To be administered by MPR 4. Collection of Student Data Questionnaire sent to all students identified in 1.c. December-February 2006 To be administered by MPR 5. Rating Questionnaire Questionnaire administered to a random sample of faculty in each field March-April 2007To be administered by MPR

5. Columbia University :: Office of the Provost :: Planning and Institutional Research 2006-7 NRC Assessment of Research-Doctoral Programs October 27, 20065 Programs Program Questionnaires What’s being collected? Faculty Lists List of all doctoral graduates from the past three years. General program characteristics and policies. Admissions, enrollment and degree completion data. Detailed financial support data for full-time doctoral students in fall 2005. How can programs help with the process? Confirm/correct data that can be collected centrally Provide answers to questions that cannot be answered centrally Program List: Submitted in September to MPR; 56 total programs submitted. See website for all programs and field designations.

6. Columbia University :: Office of the Provost :: Planning and Institutional Research 2006-7 NRC Assessment of Research-Doctoral Programs October 27, 20066 Programs- Faculty Lists Collection of faculty names by program For each participating program, we have to submit a list of faculty members. Faculty members must be classified in one of 3 categories: CORE has served on a dissertation committee in the past 5 years OR is a member of admissions or curriculum committee is currently and formally designated as faculty in the program NEW is currently and formally designated as faculty in the program, but does not meet the Core requirements has been hired in a tenured or tenure-track position in the past 3 academic years. ASSOCIATED has served on a dissertation committee in the past 5 years is not currently and formally designated as faculty in the program Note: Faculty members can be reported in more than one program

7. Columbia University :: Office of the Provost :: Planning and Institutional Research 2006-7 NRC Assessment of Research-Doctoral Programs October 27, 20067 Programs-Faculty Lists, cont’d What’s being collected? Name Street address and email address Category (i.e. Core, New, Associated) Highest degree awarded Rank and tenure status Number of dissertation committees “chaired” (i.e. “Sponsored”) in the past 5 academic years Total number of dissertation committees served on in the past 5 academic years. How will the data be used? NRC will use email addresses to contact faculty members directly for faculty questionnaire. NRC will collect citation and publication data for faculty members. NRC will use a formula to allocate faculty headcount and productivity (publications and citations) for faculty serving in more than one graduate program. How can programs help with the process? Confirm/correct list: are all the appropriate faculty members listed? Is anyone missing? Is anyone on the list who shouldn’t be there? Is anyone categorized incorrectly? Confirm/correct faculty name, contact, rank, tenure, and degree information, especially email address For each program, we will indicate all other programs in which the faculty are listed; we will also list anyone not included at all but who is appointed in the program.

8. Columbia University :: Office of the Provost :: Planning and Institutional Research 2006-7 NRC Assessment of Research-Doctoral Programs October 27, 20068 Programs-Faculty Productivity Faculty productivity will be calculated based on status in program and nature of dissertation service. Example: Professor Smith is considered Core in Program A and Program B. Between 2001-02 and 2005-06, she served as principal advisor on 3 dissertations and participated in 4 others in Program A and served as principal advisor on 1 dissertation and participated in 2 others in Program B. = 69% allocated to Program A. And therefore, 31% allocated to Program B.

9. Columbia University :: Office of the Provost :: Planning and Institutional Research 2006-7 NRC Assessment of Research-Doctoral Programs October 27, 20069 Faculty Questionnaire Who receives the survey? All Core and New faculty members. What’s being collected? Name and contact information. Demographic data. Specific information about faculty’s research field(s). Education and work experience. Doctoral graduates for the past 5 years. Information about scholarship/research activity (with opportunity to upload CV). Section on “relative importance of program characteristics to program quality.” How will the data be used? Information on names used to aid in NRC’s productivity data collection Information on doctoral graduates will be used to check NRC’s effort to gather placement data “Program quality” responses will be used to identify and weigh key study variables in conjunction with “Anchoring” study.

10. Columbia University :: Office of the Provost :: Planning and Institutional Research 2006-7 NRC Assessment of Research-Doctoral Programs October 27, 200610 Anchoring Study Who receives the survey? Random sample of Core faculty members who completed faculty questionnaire who indicated willingness to participate. What’s being collected? The faculty will receive a list of program faculty for a sample of 15 to 20 programs as well as additional program data and information (e.g. faculty diversity, student completion & placement) Respondents will: indicate familiarity with program rate the quality of the programs on a 5-point scale. How will the data be used? Respondent ratings regressed on quantitative variables to derive weights of the key study variables.

11. Columbia University :: Office of the Provost :: Planning and Institutional Research 2006-7 NRC Assessment of Research-Doctoral Programs October 27, 200611 Summary How can programs help with the process? Recognize that there still is uncertainty in the process Think through faculty lists carefully, and Encourage faculty participation! For more information: Website: http://www.columbia.edu/cu/opir/nrc/index.htmlhttp://www.columbia.edu/cu/opir/nrc/index.html Email: nrc_survey@columbia.edunrc_survey@columbia.edu