1. Publishing Productivity of US Academic Scientists: an Empirical Examination through Data Envelopment Analysis Youngsun Baek Research Value Mapping Program School of Public policy Georgia Institute of Technology

2. Contents Introduction Introduction Background Background Difficulties with Performance Indicators in the field of higher educationDifficulties with Performance Indicators in the field of higher education Efficiency and Data Envelopment Analysis (DEA)Efficiency and Data Envelopment Analysis (DEA) Research Design: Publication producer model Research Design: Publication producer model Results Results Most Productive Scientists (Good Apples)Most Productive Scientists (Good Apples) Decomposition of Technical Efficiency (TE) into Pure Technical Efficiency (PTE) and Scale Efficiency (SE)Decomposition of Technical Efficiency (TE) into Pure Technical Efficiency (PTE) and Scale Efficiency (SE) Returns to Scale (RTS) of each scientistReturns to Scale (RTS) of each scientist Disciplinary Efficiency (DE) and Individual Ability (IA)Disciplinary Efficiency (DE) and Individual Ability (IA) 2 nd stage regression (Tobit Model)2 nd stage regression (Tobit Model) Disentanglement of TE into DE and IADisentanglement of TE into DE and IA Conclusion Conclusion

3. 1.Introduction: Which scientist is more productive?  10 articles/year, 2 books/year  3 articles/year, 1 book/year Research funding: $ 1,000,000 # of coworkers: 30 No teaching load Research funding: $ 50,000 # of coworkers: 5 3 classes to teach Outputs Inputs Scientist AScientist B

4. Difficulties with performance indicators in the field of higher education “… In particular, appropriate adjustment needs to be made in order to prevent confusion between output and efficiency. For example, crude measure of research output can be obtained by a straightforward publications or citations count; but, this makes no allowance for the vast differences in resources …”, Johnes & Johnes (1995)

5. Difficulties with performance indicators in the field of higher education Johnes & Johnes (1995) Johnes & Johnes (1995) Confusion between output and efficiencyConfusion between output and efficiency Multiple inputs and multiple outputsMultiple inputs and multiple outputs Difficulty with citation analysisDifficulty with citation analysis Adding together different types of publicationsAdding together different types of publications Hanney & Kogan (1991) Hanney & Kogan (1991) Problem with impact factorsProblem with impact factors Avkiran (2001) & Korhonen (2001) Avkiran (2001) & Korhonen (2001) Absence of market mechanism in the academic field.Absence of market mechanism in the academic field. ⇒ Data Envelopment Analysis (DEA)

6. Efficiency and Data Envelopment Analysis (DEA) X1X1 X2X2 B C D E A A’ O Technical efficiency of A = OA’/OA H F G

7. Methodological Strengths of DEA as a Performance Indicator DEA can handle multiple inputs and multiple outputs.DEA can handle multiple inputs and multiple outputs. It does not require an assumption of a functional form.It does not require an assumption of a functional form. Producers are directly compared against a peer or combination of peers.Producers are directly compared against a peer or combination of peers. Inputs and outputs can have different units.Inputs and outputs can have different units.

8. DEA Application to Performance Evaluation DEA applications to evaluating academic organizations DEA applications to evaluating academic organizations Johnes & Johens (1995) ; U.K. university departments of economicsJohnes & Johens (1995) ; U.K. university departments of economics Korhonen (2001) ; Universities and research centers Korhonen (2001) ; Universities and research centers Thursby & Kemp (2002) ; University licensing activitiesThursby & Kemp (2002) ; University licensing activities Caballero et al. (2003) ; Allocation and management of university financial resourcesCaballero et al. (2003) ; Allocation and management of university financial resources Few studies conducted to evaluate individual researchers through DEA. Few studies conducted to evaluate individual researchers through DEA. ⇒ Focus on each individual scientist (Human Resource Analysis) ⇒ Focus on each individual scientist (Human Resource Analysis)

9. Publication producer model NNFC NNGSC NAG NNSP NNA NNB USCIT DSM CM YGPD G Multi inputs (Production Factors) Producer (Each Academic Scientist) Multi outputs (End Products) Contextual factors * NNFC: Normalized number of faculty coworkers, NNGSC: Normalized number of graduate students coworkers, NAG: Normalized amount of grants, NNSP: Normalized number of submitted proposals, NNA: Normalized number of published articles, NNB: Normalized number of published books, DSM: Disciplinary society membership, CM: Career mobility, YGPD: Year got a ph D. degree, USCIT: US Citizenship, G: Gender

10. Decomposing Technical Efficiency & Nature of Returns to Scale TE CCR model SE PTE BCC model 0.56 0.78 IRS CRS DRS Input Output 68% 10% 22% 0.45

11. Efficiency in each discipline DRS IRS Total Technical Efficiency Pure Technical Efficiency (Managerial Efficiency) Scale efficiency

12. Most Productive Scientists (Good Apples) 9 scientists out of 109 were selected as most efficient scientists 9 scientists out of 109 were selected as most efficient scientists O.28 faculty coworkers/ yearO.28 faculty coworkers/ year O.33 graduate student coworkers/ yearO.33 graduate student coworkers/ year 1.15 proposals/ year1.15 proposals/ year $160,277(present value-2000)/ year$160,277(present value-2000)/ year 3.6 articles/ year3.6 articles/ year 0.6 book / year0.6 book / year 7 scientists out of 9 were male7 scientists out of 9 were male Two thirds of them had US citizenshipTwo thirds of them had US citizenship 6 disciplinary society memberships6 disciplinary society memberships 10 career changes10 career changes 51 years old51 years old

13. The 2 nd stage regression (Tobit Regression Model) Coef. Std. Error Z-StatisticP-Value C0.6270.1205.2300.0000 DSM0.0980.1160.8420.3997 CM-0.197*0.076-2.5870.0097 Ph D in 60s 0.412*0.1093.7890.0002 Ph D in 70s 0.367*0.0814.5120.0000 Ph D in 80s 0.0920.0731.2530.2101 USCIT-0.0540.005-1.0120.3115 G-0.1140.073-1.5680.1168 Dependent variable- Pure technical efficiency (DEA efficiency score) Independent Variables - DSM: a variable for disciplinary society membership, CM: a variable for career mobility, Ph D in 60s, ph D in 70s, and ph D in 80s: dummy variables for year got a ph D degree, USCIT: a dummy variable for citizenship, G: a variable for gender) ** At the 1% significant level R-square 0.383921 Adjusted R-square 0.334136

14. Disentanglement of disciplinary efficiency Source: Thanassoulis (2001)

15. Disentanglement of disciplinary efficiency Inter-Group Efficiency Comparison

16. Disentanglement of disciplinary efficiency Decomposition of total efficiency into Individual and Group Efficiency

17. Disentanglement of disciplinary efficiency Differential Efficiency Distribution

18. Conclusion DEA can suggest an alternative performance indicator of publishing productivity. DEA can suggest an alternative performance indicator of publishing productivity. Contextual factors Contextual factors Existence of vintage effect in publishing productivityExistence of vintage effect in publishing productivity No correlation between publishing productivity and disciplinary society membershipNo correlation between publishing productivity and disciplinary society membership Negative relationship between career mobility and productivityNegative relationship between career mobility and productivity In order to catch up with the best performing scientists, improving publishing process is better way than trying to increase input levels such as coworkers and research grant. In order to catch up with the best performing scientists, improving publishing process is better way than trying to increase input levels such as coworkers and research grant. Project management and effective collaboration strategies are crucial to increase publishing productivity.Project management and effective collaboration strategies are crucial to increase publishing productivity. Performance of each individual depends not only on individual ability but also on the characteristics of the discipline under which the person is working. Performance of each individual depends not only on individual ability but also on the characteristics of the discipline under which the person is working.

19. Thank you!