The Methodology of the NRC Doctoral Program Assessment

Slides:



Advertisements
Similar presentations
Roberta Spalter-Roth, Ph.D Director of Research American Sociological Association Enhancing Diversity in Science: Working Together to Develop Common Data,
Advertisements

1 1 Slide © 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole.
Assessing Research-Doctorate Programs: A Methodology Study.
National Research Council Assessment of Research Doctorate Programs Summary of Methodology Michigan State.
1 Academic Rankings of Universities in the OIC Countries April 2007 April 2007.
NRC Assessment of Doctoral Programs Charlotte Kuh
Slides by JOHN LOUCKS St. Edward’s University.
Analysis of Individual Variables Descriptive – –Measures of Central Tendency Mean – Average score of distribution (1 st moment) Median – Middle score (50.
NRC 2010 Survey of Research Doctorates GSNB NRC Survey “A Data-Based Assessment of Research-Doctorate Programs in the United States.
2002 IBNA Student Survey Sarah Scaturro, Paul Campbell, Niagara Falls, Ontario July 17-20, 2003.
DIGEST OF KEY SCIENCE AND ENGINEERING INDICATORS 2008 Presentation Slides National Science Board.
Meryle Weinstein, Emilyn Ruble Whitesell and Amy Ellen Schwartz New York University Improving Education through Accountability and Evaluation: Lessons.
Agronomic Spatial Variability and Resolution What is it? How do we describe it? What does it imply for precision management?
Why Study Economics at Baylor? Steve Green Professor of Economics & Statistics Chair, Department of Economics.
Agronomic Spatial Variability and Resolution What is it? How do we describe it? What does it imply for precision management?
Chapter 11 Descriptive Statistics Gay, Mills, and Airasian
The NRC Doctoral Program Assessment Charlotte Kuh National Research Council.
Columbia University :: Office of the Provost :: Planning and Institutional Research NRC Assessment of Research-Doctoral Programs October 27,
The Guessing Game The entire business of Statistics is dedicated to the purpose of trying to guess the value of some population parameter. In what follows.
Chapter 6 Foundations of Educational Measurement Part 1 Jeffrey Oescher.
Some ACS Data Issues and Statistical Significance (MOEs) Table Release Rules Statistical Filtering & Collapsing Disclosure Review Board Statistical Significance.
11/23/2015Slide 1 Using a combination of tables and plots from SPSS plus spreadsheets from Excel, we will show the linkage between correlation and linear.
National Research Council Assessment of Doctorate Programs: The Methodology Report E. L. Fink & S. Chai University of Maryland.
Chapter 6: Analyzing and Interpreting Quantitative Data
Agronomic Spatial Variability and Resolution What is it? How do we describe it? What does it imply for precision management?
Educational Research Descriptive Statistics Chapter th edition Chapter th edition Gay and Airasian.
An Evaluation of Pipeline Interventions for Minority Scholars An Evaluation of Pipeline Interventions for Minority Scholars Roberta Spalter-Roth, Jean.
One-Variable Statistics. Descriptive statistics that analyze one characteristic of one sample  Where’s the middle?  How spread out is it?  How do different.
Stats Methods at IC Lecture 3: Regression.
Progress 8 and Attainment 8:
GS/PPAL Section N Research Methods and Information Systems
One-Variable Statistics
Who wants to be in the top 1 percent?
DTC Quantitative Methods Bivariate Analysis: t-tests and Analysis of Variance (ANOVA) Thursday 20th February 2014  
Eastern Region and London NE Boroughs Headteacher Associations’
Measures of dispersion
Deans and Chairs Academic Affairs Update
DESCRIPTIVE STATISTICS
APPROACHES TO QUANTITATIVE DATA ANALYSIS
Analyzing and Interpreting Quantitative Data
Week 14 Chapter 16 – Partial Correlation and Multiple Regression and Correlation.
AP Lab Skills Guide Data will fall into three categories:
Descriptive Statistics
Statistics Collecting and analyzing large amounts of numerical data
Science of Psychology AP Psychology
The NRC Study and Neuroscience
Progress 8 and Attainment 8:
Your Institutional Report Step by Step
Introduction to Statistics
Gender, Faculty Salaries and Inequity at UTK
Gender, Faculty Salaries and Inequity at UTK
Data analysis and basic statistics
One-Way Analysis of Variance
Mapping the ACRL Framework and Nursing Professional
Topic 1: Statistical Analysis
© 2017 by McGraw-Hill Education
A DATA BASED ASSESSMENT OF RESEARCH DOCTORATE PROGRAMS
Common Core Math I Unit 1: One-Variable Statistics Boxplots, Interquartile Range, and Outliers; Choosing Appropriate Measures.
Some Approaches to Faculty Assignments
Day 52 – Box-and-Whisker.
Amir Ghaisi Grigorios Fountas Panos Anastasopoulos Fred Mannering
Some Approaches to Faculty Assignments
Descriptive Statistics
Geology Geomath Chapter 7 - Statistics tom.h.wilson
Xbar Chart By Farrokh Alemi Ph.D
These probabilities are the probabilities that individual values in a sample will fall in a 50 gram range, and thus represent the integral of individual.
Chapter 3: Data Description
Basic Biostatistics Measures of central tendency and dispersion
Business Statistics For Contemporary Decision Making 9th Edition
2018 NSSME+: Briefing Book Slides
Presentation transcript:

The Methodology of the NRC Doctoral Program Assessment Charlotte Kuh National Research Council Our Methodology Guide to the NRC Assessment of Research Doctorate Programs was released on last Thursday. It was sent to the Institutional Coordinators at each of your institutions as an attachment to an e-mail, reported on our project website, and reported in the news blog of the Chronicle of Higher Education, and in Inside Higher Education and both articles contained a link. After we sent it out, we learned that about 12% of the Institutional Coordinators were “out of the office”—but most of you should have seen it. The main part of it is a 31 page description of the methodology. It is followed by a 19 page technical appendix that explains the techniques that were used. After encouragement at the 2007 CGS Annual Meeting, the Committee thought that it was important to publish the guide first for two main reasons: 1. The methodology is complex. It relies on statistical techniques to derive weighted averages of measures. Transparency is one of our prime objectives and laying out a complex process in layman’s language is one way to achieve transparency. 2. If you read it carefully, you will know how to “deconstruct” your ranges of rankings—to see what is contributing to them and how these contributory factors compare to other programs in a field. 3. If you are looking to improve your doctoral programs, all the dimensional measures (research activity, student support and outcomes, and diversity) are important in addition to the overall measure. As you educate your institution, this is an important point. Using the tables in the Methodology Guide, I would like to take you through this deconstruction. Of course, the Guide may be so clear that this is unnecessary—but on the off chance that it isn’t, I thought this might be helpful!

Committee* (Sociology) Norman Bradburn ( Statistics) University of Chicago John I. Brauman (NAS) (Chemistry) Stanford University Jonathan Cole (Sociology) Columbia University Eric W. Kaler (Engineering) Stony Brook University Jeremiah P. Ostriker (NAS) Committee Chair (Astrophysics) Princeton University Virginia S. Hinshaw, Vice-Chair* (Biology) University of Hawai’i, Mano’a Paul Holland (Statistics) Educational Testing Service Elton D. Aberle (Agriculture) University of Wisconsin-Madison *Several members of Committee are present or former Deans of graduate schools.

Committee (2) Earl Lewis * (history) Emory University Joan F. Lorden* (Neuroscience) University of North Carolina, Charlotte Carol B. Lynch* (Biology) University of Colorado, Boulder Robert M. Nerem (NAE) (Bioengineering) Georgia Institute of Technology Suzanne Ortega* (Sociology) University of New Mexico Robert J. Spinrad (NAE) (Computer Science) Xerox Corporation resigned November 2007 Catharine R. Stimpson* (English) New York University Richard P. Wheeler* (English) University of Illinois

When the ratings come out, what will the university receive to explain the range of rankings for each of its programs? A table comparing the data from its programs to the average values for the fields as a whole. A table showing, for each program, how the first quartile rating was calculated and the ranking that corresponded to it. A table showing, for each program, how the third quartile rating was calculated and the ranking that corresponded to it These tables will be sent to your institutional coordinator BEFORE the rankings are made public. Let’s start with what the universities will receive shortly before the final report is released.

When the report is released All the ranges of rankings for all the programs in each field will be available on the Web in Excel spreadsheets All the data provided by all the programs will be available on the Web, provided the cell sizes aren’t too small. There will be a separate spreadsheet with the ratings and rankings, coefficients, and the mean values of the 20 variables in each field.

The Twenty Key Variables Publications Citations (exc. Humanities) Percent faculty with grants Awards per faculty Percent 1st Yr. Full Support Percent 1st Yr. National Fellowship Percent Completing in 6 yrs. or less (8 yrs. for humanities) Median Time to degree Students with Academic Plans Collects Outcomes data Percent Faculty Minority Percent Faculty Female Percent Students Minority Percent Students Female Percent Students International Percent Interdisciplinary Average GRE-Q Number of PhDs 2002-2006 Student Workspace Student Health Insurance Student Activities These are the twenty key variables. The colored ones went into the dimensional measures. The black ones were only included in the overall measure. The blue items went into the dimensional measure of research activity. The red measures went into the measure of student support and outcomes. The green items went into the measure of diversity. The remaining measures were taken into account in the overall measure, but did not enter into the dimensional measures.

This table summarizes the information that will be used to calculate the ratings for your program and for all the programs in this field.

You submitted raw data and we did two things to it You submitted raw data and we did two things to it. For publications and awards, we turned it into a percapita measure, using the percent of faculty allocated to the program at your institution. We then took the value of these variables for all the programs in economics and standardized them. That is, we placed them on a scale with mean 0 and variance 1.

We also have a range of combined coefficients We also have a range of combined coefficients. What we mean by this we will discuss shortly. But this table shows you the range of combined coefficients. You can see that some ranges are quite tight—for example the percent faculty with grants, and others have a wider range—cites for example.

Some of the twenty variables didn’t turn out to be significantly different from zero, and so they weren’t used in the ranking calculation. Some do figure, in the dimensional measures, which I will talk about shortly.

This is how we calculate a ranking This is how we calculate a ranking. We take the program values, standadize them and allow them to vary (±10% for most of the variables). We then multiply that value by the combined coefficient (again chosen from a distribution of values depending on what raters are chosen) and use that to calculate the contribution of each significant variable to the rating. This calculation is for the top of the first quartile of ratings that we calculate. You can see the contribution of each variable to the rating and the total rating. This rating also corresponds to a ranking, 56 in this case, but let’s look at the calculation of the other end of the range of rankings before we go into this.

This is the calculation of the rating that corresponds to the upper end of the range of rankings for the program. When we combine all the 500 ratings for this program and rank order them with all the 500 ratings for all the other programs and look at the bounds on the middle of that range of rankings, we find that this program ranks somewhere between 45 and 56 among economics programs. Now, let’s get into what’s behind these numbers.

Sources of Data Institutions and Programs Faculty Existing data Institutional practices, program characteristics, faculty and student demographics, faculty lists and lists of advanced students in 5 fields Faculty Characteristics (work history, publication identification data, demographics, c.v.’s, importance to quality of program characteristics) Existing data Publications and citations (ISI) Ph.D. post-graduation plans (NSF) Most of these data came from you and your faculty and were combined with data from commercial data vendors (ISI) as well as with data from the National Science Foundation.

The Twenty Key Variables Publications Citations (exc. Humanities) Percent faculty with grants Awards per faculty Percent 1st Yr. Full Support Percent 1st Yr. National Fellowship Percent Completing in 6 yrs. or less (8 yrs. for humanities) Median Time to degree Students with Academic Plans Collects Outcomes data Percent Faculty Minority Percent Faculty Female Percent Students Minority Percent Students Female Percent Students International Percent Interdisciplinary Average GRE-Q Number of PhDs 2002-2006 Student Workspace Student Health Insurance Student Activities These are the twenty key variables. The colored ones went into the dimensional measures. The black ones were only included in the overall measure. The blue items went into the dimensional measure of research activity. The red measures went into the measure of student support and outcomes. The green items went into the measure of diversity. The remaining measures were taken into account in the overall measure, but did not enter into the dimensional measures.

Two ways of getting at importance Weights Two ways of getting at importance Directly—asked faculty on the faculty questionnaire to choose most important of the twenty variables Regression based: A sample of faculty rated a sample of programs in each field. These ratings were regressed on the 20 variables. The coefficients were the regression-based weights. Combined the weights obtained in 1) and 2). How did we calculate our coefficients. We started by obtaining weights, and we did this in two different ways: (to slide)

Variability in Values of Weights and Data Took 500 samples of weights, each using half the raters. Data Obtained a distribution of variable values, either from annual values provided on the questionnaires, or from taking ±10% Statistical Calculated the standard error for each regression A single value doesn’t do justice to the variability that is inherent in rankings. Importance weights for the measures will depend on who you ask—so we took 500 samples of raters for both the direct weights and the regression-based rates and combined them to obtain 500 weights for each variable. Data are not the same from year-to year. We either know that explicitly or, when we have an observation for only a single year, we constructed a plus or minus 10% interval around the value of the measure. Finally, our coefficients were derived through a statistical estimation—and every such estimate has a standard error of estimate.

Obtaining the Range of Rankings Calculated 500 ratings for each program and ordered these from highest to lowest across all programs in a field. Constructed a “range of rankings” that showed the middle half of the calculated rankings (the interquartile range). Given these sources of variation, we calculated 500 ratings for each program. Why 500—it’s a big number and should get us close to the true distribution of ratings. [To slide]

Overall Rating AND Dimensional Measures Student Treatment and Outcomes Diversity Scholarly Productivity of Program Faculty The Dimensional Measures add information to the overall rankings The Overall Measure, which we have just been discussing, is not the only measure of doctoral programs that will be presented in the final report. We also have three dimensional measures.

Obtaining Dimensional Measures In each of the 3 dimensions, the weights were re-normalized to add up to one. Then they were applied to the data. This weighted average yielded the measure. Each of these measures also takes into account rater and data variability.

Dimensional Rankings for our Sample Program Overall Measure: 45-56 Research Activity: 21-31 Student Support and Outcomes: 74-87 Diversity of the Academic Environment 64-77

One more example from Appendix G Overall Measure: 7 to 8 Research Activity: 7 to 9 Student support and Outcomes: 1 to 4 Diversity of the Academic Environment: 106 to 109 This is a highly ranked program that treats its students very well. The only problem is—it isn’t diverse at all. Is that a problem? That is a matter for discussion between programs and the administration. I just use this example to point out that there is lots of food for thought to be found in the NRC study, when it appears.

When will the Report and Database be Released? With the publication of the Methodology Guide, we have completed a major milestone. The NRC and its committee feel an enormous obligation to produce data and rankings that are of the highest quality possible.  Striving for this objective requires careful review of the data and rankings for over 5000 programs.  This has already taken considerable time and is nearing completion.  The committee must also finish summarizing the most important findings from the data, as well as discuss the strengths and shortcomings of the methodology for incorporation  in the final report.  The last step of the process is for the final report to undergo the Academy’s rigorous review.  We are working to complete all this work as expeditiously as possible.

To Learn More About the Study http://sites.nationalacademies.org/pga/Resdoc/index.htm Or contact ckuh@nas.edu Or jvoytuk@nas.edu