Download presentation
Presentation is loading. Please wait.
Published byDwayne Shepherd Modified over 7 years ago
1
The Methodology of the NRC Doctoral Program Assessment
Charlotte Kuh National Research Council Our Methodology Guide to the NRC Assessment of Research Doctorate Programs was released on last Thursday. It was sent to the Institutional Coordinators at each of your institutions as an attachment to an , reported on our project website, and reported in the news blog of the Chronicle of Higher Education, and in Inside Higher Education and both articles contained a link. After we sent it out, we learned that about 12% of the Institutional Coordinators were “out of the office”—but most of you should have seen it. The main part of it is a 31 page description of the methodology. It is followed by a 19 page technical appendix that explains the techniques that were used. After encouragement at the 2007 CGS Annual Meeting, the Committee thought that it was important to publish the guide first for two main reasons: 1. The methodology is complex. It relies on statistical techniques to derive weighted averages of measures. Transparency is one of our prime objectives and laying out a complex process in layman’s language is one way to achieve transparency. 2. If you read it carefully, you will know how to “deconstruct” your ranges of rankings—to see what is contributing to them and how these contributory factors compare to other programs in a field. 3. If you are looking to improve your doctoral programs, all the dimensional measures (research activity, student support and outcomes, and diversity) are important in addition to the overall measure. As you educate your institution, this is an important point. Using the tables in the Methodology Guide, I would like to take you through this deconstruction. Of course, the Guide may be so clear that this is unnecessary—but on the off chance that it isn’t, I thought this might be helpful!
2
Committee* (Sociology) Norman Bradburn ( Statistics)
University of Chicago John I. Brauman (NAS) (Chemistry) Stanford University Jonathan Cole (Sociology) Columbia University Eric W. Kaler (Engineering) Stony Brook University Jeremiah P. Ostriker (NAS) Committee Chair (Astrophysics) Princeton University Virginia S. Hinshaw, Vice-Chair* (Biology) University of Hawai’i, Mano’a Paul Holland (Statistics) Educational Testing Service Elton D. Aberle (Agriculture) University of Wisconsin-Madison *Several members of Committee are present or former Deans of graduate schools.
3
Committee (2) Earl Lewis * (history) Emory University
Joan F. Lorden* (Neuroscience) University of North Carolina, Charlotte Carol B. Lynch* (Biology) University of Colorado, Boulder Robert M. Nerem (NAE) (Bioengineering) Georgia Institute of Technology Suzanne Ortega* (Sociology) University of New Mexico Robert J. Spinrad (NAE) (Computer Science) Xerox Corporation resigned November 2007 Catharine R. Stimpson* (English) New York University Richard P. Wheeler* (English) University of Illinois
4
When the ratings come out, what will the university receive to explain the range of rankings for each of its programs? A table comparing the data from its programs to the average values for the fields as a whole. A table showing, for each program, how the first quartile rating was calculated and the ranking that corresponded to it. A table showing, for each program, how the third quartile rating was calculated and the ranking that corresponded to it These tables will be sent to your institutional coordinator BEFORE the rankings are made public. Let’s start with what the universities will receive shortly before the final report is released.
5
When the report is released
All the ranges of rankings for all the programs in each field will be available on the Web in Excel spreadsheets All the data provided by all the programs will be available on the Web, provided the cell sizes aren’t too small. There will be a separate spreadsheet with the ratings and rankings, coefficients, and the mean values of the 20 variables in each field.
6
The Twenty Key Variables
Publications Citations (exc. Humanities) Percent faculty with grants Awards per faculty Percent 1st Yr. Full Support Percent 1st Yr. National Fellowship Percent Completing in 6 yrs. or less (8 yrs. for humanities) Median Time to degree Students with Academic Plans Collects Outcomes data Percent Faculty Minority Percent Faculty Female Percent Students Minority Percent Students Female Percent Students International Percent Interdisciplinary Average GRE-Q Number of PhDs Student Workspace Student Health Insurance Student Activities These are the twenty key variables. The colored ones went into the dimensional measures. The black ones were only included in the overall measure. The blue items went into the dimensional measure of research activity. The red measures went into the measure of student support and outcomes. The green items went into the measure of diversity. The remaining measures were taken into account in the overall measure, but did not enter into the dimensional measures.
7
This table summarizes the information that will be used to calculate the ratings for your program and for all the programs in this field.
8
You submitted raw data and we did two things to it
You submitted raw data and we did two things to it. For publications and awards, we turned it into a percapita measure, using the percent of faculty allocated to the program at your institution. We then took the value of these variables for all the programs in economics and standardized them. That is, we placed them on a scale with mean 0 and variance 1.
9
We also have a range of combined coefficients
We also have a range of combined coefficients. What we mean by this we will discuss shortly. But this table shows you the range of combined coefficients. You can see that some ranges are quite tight—for example the percent faculty with grants, and others have a wider range—cites for example.
10
Some of the twenty variables didn’t turn out to be significantly different from zero, and so they weren’t used in the ranking calculation. Some do figure, in the dimensional measures, which I will talk about shortly.
11
This is how we calculate a ranking
This is how we calculate a ranking. We take the program values, standadize them and allow them to vary (±10% for most of the variables). We then multiply that value by the combined coefficient (again chosen from a distribution of values depending on what raters are chosen) and use that to calculate the contribution of each significant variable to the rating. This calculation is for the top of the first quartile of ratings that we calculate. You can see the contribution of each variable to the rating and the total rating. This rating also corresponds to a ranking, 56 in this case, but let’s look at the calculation of the other end of the range of rankings before we go into this.
12
This is the calculation of the rating that corresponds to the upper end of the range of rankings for the program. When we combine all the 500 ratings for this program and rank order them with all the 500 ratings for all the other programs and look at the bounds on the middle of that range of rankings, we find that this program ranks somewhere between 45 and 56 among economics programs. Now, let’s get into what’s behind these numbers.
13
Sources of Data Institutions and Programs Faculty Existing data
Institutional practices, program characteristics, faculty and student demographics, faculty lists and lists of advanced students in 5 fields Faculty Characteristics (work history, publication identification data, demographics, c.v.’s, importance to quality of program characteristics) Existing data Publications and citations (ISI) Ph.D. post-graduation plans (NSF) Most of these data came from you and your faculty and were combined with data from commercial data vendors (ISI) as well as with data from the National Science Foundation.
14
The Twenty Key Variables
Publications Citations (exc. Humanities) Percent faculty with grants Awards per faculty Percent 1st Yr. Full Support Percent 1st Yr. National Fellowship Percent Completing in 6 yrs. or less (8 yrs. for humanities) Median Time to degree Students with Academic Plans Collects Outcomes data Percent Faculty Minority Percent Faculty Female Percent Students Minority Percent Students Female Percent Students International Percent Interdisciplinary Average GRE-Q Number of PhDs Student Workspace Student Health Insurance Student Activities These are the twenty key variables. The colored ones went into the dimensional measures. The black ones were only included in the overall measure. The blue items went into the dimensional measure of research activity. The red measures went into the measure of student support and outcomes. The green items went into the measure of diversity. The remaining measures were taken into account in the overall measure, but did not enter into the dimensional measures.
15
Two ways of getting at importance
Weights Two ways of getting at importance Directly—asked faculty on the faculty questionnaire to choose most important of the twenty variables Regression based: A sample of faculty rated a sample of programs in each field. These ratings were regressed on the 20 variables. The coefficients were the regression-based weights. Combined the weights obtained in 1) and 2). How did we calculate our coefficients. We started by obtaining weights, and we did this in two different ways: (to slide)
16
Variability in Values of Weights and Data
Took 500 samples of weights, each using half the raters. Data Obtained a distribution of variable values, either from annual values provided on the questionnaires, or from taking ±10% Statistical Calculated the standard error for each regression A single value doesn’t do justice to the variability that is inherent in rankings. Importance weights for the measures will depend on who you ask—so we took 500 samples of raters for both the direct weights and the regression-based rates and combined them to obtain 500 weights for each variable. Data are not the same from year-to year. We either know that explicitly or, when we have an observation for only a single year, we constructed a plus or minus 10% interval around the value of the measure. Finally, our coefficients were derived through a statistical estimation—and every such estimate has a standard error of estimate.
17
Obtaining the Range of Rankings
Calculated 500 ratings for each program and ordered these from highest to lowest across all programs in a field. Constructed a “range of rankings” that showed the middle half of the calculated rankings (the interquartile range). Given these sources of variation, we calculated 500 ratings for each program. Why 500—it’s a big number and should get us close to the true distribution of ratings. [To slide]
18
Overall Rating AND Dimensional Measures
Student Treatment and Outcomes Diversity Scholarly Productivity of Program Faculty The Dimensional Measures add information to the overall rankings The Overall Measure, which we have just been discussing, is not the only measure of doctoral programs that will be presented in the final report. We also have three dimensional measures.
19
Obtaining Dimensional Measures
In each of the 3 dimensions, the weights were re-normalized to add up to one. Then they were applied to the data. This weighted average yielded the measure. Each of these measures also takes into account rater and data variability.
20
Dimensional Rankings for our Sample Program
Overall Measure: Research Activity: Student Support and Outcomes: Diversity of the Academic Environment 64-77
21
One more example from Appendix G
Overall Measure: 7 to 8 Research Activity: 7 to 9 Student support and Outcomes: 1 to 4 Diversity of the Academic Environment: 106 to 109 This is a highly ranked program that treats its students very well. The only problem is—it isn’t diverse at all. Is that a problem? That is a matter for discussion between programs and the administration. I just use this example to point out that there is lots of food for thought to be found in the NRC study, when it appears.
22
When will the Report and Database be Released?
With the publication of the Methodology Guide, we have completed a major milestone. The NRC and its committee feel an enormous obligation to produce data and rankings that are of the highest quality possible. Striving for this objective requires careful review of the data and rankings for over 5000 programs. This has already taken considerable time and is nearing completion. The committee must also finish summarizing the most important findings from the data, as well as discuss the strengths and shortcomings of the methodology for incorporation in the final report. The last step of the process is for the final report to undergo the Academy’s rigorous review. We are working to complete all this work as expeditiously as possible.
23
To Learn More About the Study
Or contact Or
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.