Program Level Assessment of Outcomes and Impacts of Research of Centers Juan D. Rogers School of Public Policy Georgia Tech Project: Assessment of 15 Nanotechnology Science and Engineering Centers’ (NSECs): Outcomes and Impacts: Their Contribution to NNI Objectives and Goals, NSF

Objectives Publication output and impact of the NSECs Co-authorship patterns change over time Geographical spread or concentration of NSEC research collaboration Evolution of areas of research as reflected in keyword changes Multidisciplinary footprint of the centers The case for centers 11/5/2011AEA, Annaheim CA2

Methodology Outline Qualitative-quantitative concurrent design ◦ Qualitative component  Multiple embedded case studies of mechanisms for center outcomes  Follow up field level impact with quantitative analysis ◦ Quantitative component  Bibliometrics (productivity, citation, co-authors, etc.)  Personnel and funding data analysis  Intellectual property instances (patents, licenses, etc.)  Links with business and commercial applications ◦ Methodologically: Final results are generalizations to theory from qualitative analysis  We offer analysis of combined quant-qual data  Interpretive schemes for quantitative findings on centers 11/5/2011AEA, Annaheim CA3

Data and methods Acquisition of 85 center annual reports from all 15 NSEC centers Extraction and clean up (duplicate removal) of publication lists Extraction of NSEC articles from Web of Science (n=3,500) Look up and extraction of articles citing NSEC articles (n=75,000) Clean up and classification of collaborating organizations Clean up, look up, identification, and matching of NSEC authors in author listings Analyses of different type, multiple tools: ◦ Growth, shares, and overall trends (tabular) ◦ Networks and collaborations (Gephi) ◦ Geographical spread, GIS (ArcGIS) ◦ Keywords and topics (VantagePoint) ◦ Multidisciplinarity and science maps (Pajek) 11/5/2011AEA, Annaheim CA4

NSEC publication activity grows in three waves 11/5/2011AEA, Annaheim CA5 Notes: *Publication data not reported by all NSEC centers; last column reports average annual change for rows with change data. Source: ISI-WoS publication data based on NSEC annual reports by center * * NSEC publications (all centers) ,870 ▪ Annual change 102%66%19%90%3%39%3%-51%34% Citing articles ,164 2,619 4,595 7,415 10,469 15,243 19,149 94,484 ▪ Annual change 715%198%125%75%61%41%46%26%12% Publications

One out of 10 NSEC publications has an industry co-author 11/5/2011AEA, Annaheim CA6 Total unique firms co-authoring articles with NSEC ( ): 146 Total unique firms maintaining other types of collaborations with NSEC (as of 2010): 275** Notes: * Publication data not reported by all NSEC centers; last column reports average annual change for rows with change data. **The type of collaborations are not specified by centers (only number of industry partners was provided). Source: ISI-WoS publication data based on NSEC annual reports by center and lists of industry partners provided by NSEC centers * * NSEC centers with publications NSEC publications (all centers) ,509 NSEC pubs. co-auth. with industry ▪ Annual change 8%23%6%106%49%46%-14%-48% 22% ▪ Share industry co-auth / all pubs.18%10%7%6%7%10%11%9% 10% Unique co-author firms ▪ Annual change 18%-31%78%94%-6%72%-14%-49% 20% Industry collaborations

Comparative Impact of NSEC Papers (Citations Cohort 2001) C_Mean C_Med C_Max N_Mean N_Medi N_Max /5/2011AEA, Annaheim CA7

Comparative Impact of NSEC Papers (Citations Cohort 2002) C_Mean C_Med C_Max N_Mean N_Med N_Max /5/2011AEA, Annaheim CA8 NSEC papers appear to have higher impact measured by citations: median and mean citations grow faster than the cohort with window length

Rank in Cohort of Top 20 NSEC Papers /5/2011AEA, Annaheim CA NSEC papers rank highly in their cohort of Nano papers. Total Cohort 2001: papers. NSEC Cohort 2001: 66 papers Total Cohort 2002: papers. NSEC Cohort 2002: 128 papers

Comparative Impact of NSEC Papers (Citations Cohort 2003) N_mean N_median N_max C_mean C_median C_max /5/2011AEA, Annaheim CA10 The distributions parameters show NSEC high position in the field but the top paper is still not by the centers. N = CN =

Comparative Impact of NSEC Papers (Citations Cohort 2004) Xmean Xmedian Xmax NXmean NXmedian NXmax /5/2011AEA, Annaheim CA N = 48952; CN = 259

Comparative Impact of NSEC Papers (Citations Cohort 2005) N_mean N_median01357 N_max C_mean C_median C_max /5/2011AEA, Annaheim CA N = 55998; CN = 499

Comparative Impact of NSEC Papers (Citations Cohort 2006) N_mean N_median01356 N_max C_mean C_median C_max /5/2011AEA, Annaheim CA N = 62351; CN = 512

Increasing JIF of Target Journals 11/5/2011AEA, Annaheim CA14

Increasing JIF of Target Journals 11/5/2011AEA, Annaheim CA15

Leveraging Support Reflected in Publication Support Acknowledgment 11/5/2011AEA, Annaheim CA16

NSEC co-authorship networks grow and become more widespread 11/5/2011AEA, Annaheim CA17 Co-authorships Notes: Nodes represent authors. Node size represents number of publications for the period. Node colors represent 15 NSEC centers. Line colors are those of the centers that maintain each co-authorship as found in publication databases. Source: ISI-WoS publication data based on NSEC annual reports by center. Co-authorships

11/5/2011AEA, Annaheim CA18 Central and well connected authors Highly collaborative projects Multiple productive and collaborative authors NSEC collaboration networks present diverse patterns of co- authorship ( ) Co-authorships

Wide geographic spread of NSEC research with concentration in specific locations/regions 11/5/2011AEA, Annaheim CA19 Note: Number of NSEC publications from = 3509; number of citing publications = Citing publications, exclude all NSEC publications. Collaboration and places

Co-authoring extends beyond the NSECs to nearly all US states 11/5/2011AEA, Annaheim CA20 Collaboration and places

Clusters of top keywords (21 cluster solution reported) More specialized terms More linked terms Research topics

Top 20 Terms Across 15 NSECs (relative position of 30 most common, v ) 11/5/2011AEA, Annaheim CA22 FOUNDATIONAL FABRICS THIN-FILM SELF-ASSEMBLED MONOLAYER SPECTROSCOPY CARBON NANOTUBE STABLE NANOCRYSTAL OPTICAL PROPERTY MOLECULAR-DYNAMICS GROWTH FIELD-EFFECT TRANSISTOR LITHOGRAPHY GOLD NANOPARTICLE QUANTUM DOT SENSOR WALL CARBON NANOTUBE IN-VITRO SEMICONDUCTOR MOLECULAR-DYNAMICS SIMULATION BIOSENSOR IN-VIVO THIN-FILM-TRANSISTOR GRAPHITE MATURE DNA SCATTERERS TRANSISTOR DIP PEN NANOLITHOGRAPHY SILVER NANOPARTICLE AQUEOUS-SOLUTION ATOMIC FORCE MICROSCOPY WAVE-GUIDE HETEROSTRUCTURE Research topics

Neurosciences Agriculture Chemistry Physics Engineering Environ Sci Matls Sci Clinical Med Computer Sci. Geosciences Ecology General Med Map of Science: Subject Categories within all Nano (2009) Source: Rafols, I. and Meyer, M. (2009) Diversity and Network Coherence as indicators of interdisciplinarity: case studies in bionanoscience. /Scientometrics/, 81(2), in print; Leydesdorff, L. and Rafols, I. (2009) A Global Map of Science Based on the ISI Subject Categories. Journal of the American Society for Information Science and Technology, 60(2), /5/201123AEA, Annaheim CA

Neurosciences Agriculture Biomed Sci Chemistry Physics Engineering Environ Sci Matls Sci Infectious Diseases Clinical Med Computer Sci. Geosciences Ecology General Med NSEC research is multidisciplinary with focus areas in materials science, chemistry and biomedical sciences Map source: Rafols, I., Meyer, M. (2009) Diversity and Network Coherence as indicators of interdisciplinarity: case studies in bionanoscience. Scientometrics, 81(2), in print; Leydesdorff, L., Rafols, I. (2009) A Global Map of Science Based on the ISI Subject Categories. Journal of the American Society for Information Science and Technology, 60(2), Multidisciplinarity Global Map of Science, SCI Subject Categories 11/5/2011AEA, Annaheim CA24

Neurosciences Agricultur e Biomed Sci Chemistry Physics Engineerin g Environ Sci Matls Sci Infectious Diseases Clinical Med Computer Sci. Geoscience s Ecology General Med A range of disciplines beyond focal areas cite NSEC works Multidisciplinarity Global Map of Science, SCI Subject Categories 11/5/2011AEA, Annaheim CA25

Implications for Center Policy NSEC publication growth rate indicates rapid take-off by new centers. NSEC mechanism allows for greater involvement of authors over time in diverse locations. NSEC research involves some foundational as well as some emerging (and some maturing) topical areas. NSECs work in multiple disciplines and their citation influence involves many more disciplines (especially in Biotechnology related fields). 11/5/2011AEA, Annaheim CA26 Conclusions

The Case for Centers Reporting by PIs on crucial role of centers: ◦ Unique incentive to go deeply across disciplinary boundaries  Significantly reduced cross-disciplinary transaction costs ◦ Unique research experience for graduate students  Rich and diverse research infrastructure enables more risk taking  Exposure to greater number of high quality scientific contacts  Exposure to unique industry contacts ◦ Accelerator of promising young researcher development  Center as recruiting tool of top talent  Diverse mentoring opportunities for rapid career development ◦ Unique infrastructure possibilities  Leverage of resources for shared new facilities  Design of new unique instruments and experimental arrangements 11/5/2011AEA, Annaheim CA27

Some Program Challenges Some scientific contributions are difficult to explain to the lay public ◦ Important for long term support of the enterprise The special benefits for development of faculty and graduate students should be scaled up to reach the rest of the university community The pre- and extra- university education efforts are dependent on the center program and are difficult to institutionalize without it. ◦ Their sustainability should be a program concern Similar sustainability issue is raised by specialized infrastructure that may go to waste if centers are discontinued 11/5/2011AEA, Annaheim CA28