ATTEMPTS TO MEASURE and MAXIMIZE SOCIO-ECONOMIC IMPACTS of METROLOGY at NIST Karen H. Brown Deputy Director, National Institute of Standards and Technology NCSL International Annual Workshop & Symposium August 5, 2002

NIST Assets Include: 3,000 employees 1,600 guest researchers $760 million annual budget NIST Laboratories -- National measurement standards Advanced Technology Program -- $570 million current R&D partnerships with industry Manufacturing Extension Partnership centers nationwide to help small manufacturers Baldrige National Quality Award NIST’s mission is to develop and promote measurement, standards, and technology to enhance productivity, facilitate trade, and improve the quality of life. National Institute of Standards & Technology

NIST’s Overarching Goal NIST’s overarching goal is to maximize our impact through: Improvements in productivity. Improvements in productivity. Increased access to markets. Increased access to markets. Expanded public benefit through measurements and standards. Expanded public benefit through measurements and standards.

Outline Measuring Impact Measuring Impact Maximizing Impact Maximizing Impact

Measuring Impact

Metrology Underpins Growth Technological Advances: Vital for commerce and international trade Vital for commerce and international trade Account for ~50% of U.S. economic growth Account for ~50% of U.S. economic growth Drives demand for new measurements and standards Drives demand for new measurements and standards Scientific Knowledge Technologies/ Processes Products/ Services Economic Growth Applied Research Development Production Research Market Penetration Measurement Techniques Metrology Standards Reference Data/Materials Calibrations Reinvestment

Assessing Economic Impact for NIST Planning Purposes Administration and congressional priorities External environment scanning Technology assessment, market and economic analysis Industry interaction-- roadmaps, conferences, workshops Long-Term Strategic Planning Annual Program Planning Cycle Performance Evaluation Program Implementation & Management Strategic planning studies External Assessment Quantitative Output Metrics Customer Satisfaction Data Economic Impact Studies

Measuring Impact Inputs/Outputs “Activity” counts are easiest to compile “Activity” counts are easiest to compile Cost accounting may not match project content Cost accounting may not match project content Results may occur irregularly in research projects Results may occur irregularly in research projects Outputs change over project life cycle Outputs change over project life cycleOutcomes Difficult to measure Difficult to measure Economic Economic Take time to be realized Take time to be realized Can extend over long periods Can extend over long periods

Quantitative Output Measures NIST Laboratories Outputs Measurement and test methods Measurement and test methods Science and engineering databases Science and engineering databases Simulation models Simulation models Interface protocols Interface protocols Test artifacts (SRMs) & services (calibrations) Test artifacts (SRMs) & services (calibrations) Dissemination of Outputs Via peer-reviewed publications Via peer-reviewed publications Via NIST publications Via NIST publications Via the web Via the web Via standards Via standards

Strengths/Weaknesses of Output Measures Strengths Highly reliable quantitative data Highly reliable quantitative data Robust data collection systems Robust data collection systems Data are cumulative; supports trend analysis Data are cumulative; supports trend analysisLimitations No indication of quality or impact No indication of quality or impact Trends require contextual interpretation Trends require contextual interpretation Indicators as a whole do not comprehensively represent NIST output Indicators as a whole do not comprehensively represent NIST output

Economic Impact studies: General features Studies must characterize and analyze Supply chain structure and dynamics Supply chain structure and dynamics Industrial technology trends Industrial technology trends Technical and economic dimensions of technology infrastructure Technical and economic dimensions of technology infrastructure NIST impact pathways NIST impact pathways Impact and cost data Impact and cost data

Impact studies: Expected results High quality technology assessment and characterization of supply chain Detailed specification of impact pathways R&D R&D Production Production Market development Market development Impact studies yield quantitative estimates of: Net Present Value: inflation-adjusted value of net benefits Net Present Value: inflation-adjusted value of net benefits Benefit-to-Cost Ratio: efficiency measure of project’s production of benefits based on a given discount rate Benefit-to-Cost Ratio: efficiency measure of project’s production of benefits based on a given discount rate Internal Rate of Return: efficiency measure of project’s production of net benefits without discount rate Internal Rate of Return: efficiency measure of project’s production of net benefits without discount rate

Impact studies: Key issues Scope of impact Tracing the impact pathway within and across supply chains Tracing the impact pathway within and across supply chains Establishing that the impact was due to NIST outputs Establishing that the impact was due to NIST outputs Boundaries of quantitative and qualitative assessment Boundaries of quantitative and qualitative assessment Data gathering Willingness of industry representatives to share data Willingness of industry representatives to share data Quality/reliability of data provided Quality/reliability of data providedTimeframes How long to wait before initiating a study How long to wait before initiating a study Too long – impact too diffuse Too long – impact too diffuse Too soon – impact may be minimized Too soon – impact may be minimized

Strengths/Weaknesses of Impact Studies Strengths Yield quantitative and qualitative data Yield quantitative and qualitative data Examine long time periods and multiple layers of supply chains Examine long time periods and multiple layers of supply chains Qualified experts use well-developed research methods Qualified experts use well-developed research methodsLimitations Studies are intermittent/results are not cumulative Studies are intermittent/results are not cumulative Uneven availability and quality of industry data Uneven availability and quality of industry data Outcomes specific to each project (not comparable) Outcomes specific to each project (not comparable) Complex/expensive to perform Complex/expensive to perform

Economic Impact Examples

Sulfur Standards NIST production of sulfur SRMs for use by instrument manufacturers, fossil fuel producers, fossil fuel consumers NIST-developed isotope dilution thermal ionization mass method improved accuracy of reference material by factor of 1.25 – 1.75 NIST-developed isotope dilution thermal ionization mass method improved accuracy of reference material by factor of 1.25 – NIST SRMs available to help assess sulfur in kerosene, crude oil, fuel oil, coal, etc. 20+ NIST SRMs available to help assess sulfur in kerosene, crude oil, fuel oil, coal, etc. Significantly improved production efficiency for petroleum refiners and coal companies Significantly improved production efficiency for petroleum refiners and coal companies Surveyed Benefit-to-Cost Ratio: 113:1

Radiopharmaceutical Standards NIST Production of Radionuclide SRMs for use by Radiopharmaceutical Manufacturers Calibration of manufacturer-submitted radionuclide samples Calibration of manufacturer-submitted radionuclide samples NIST calibration error +/- 1% NIST calibration error +/- 1% 25+ NIST SRMs available 25+ NIST SRMs available Samples provided by commercial labs traceable to NIST provide calibration error +/- 5% Samples provided by commercial labs traceable to NIST provide calibration error +/- 5% Surveyed Benefit-to-Cost Ratio: 97:1

Fiber Optic Power Calibrations Laser & Fiberoptic Power and Energy Calibration Services Calibrations & measurements supporting: Calibrations & measurements supporting: Power output of laser light sources Power output of laser light sources Optical fiber power Optical fiber power Frequency response of high-speed detectors Frequency response of high-speed detectors Example NIST Customers Example NIST Customers Test instrument manufacturers Test instrument manufacturers Communications equipment providers Communications equipment providers Semiconductor manufacturers Semiconductor manufacturers Surveyed Benefit-to-Cost Ratio: 11:1 for Primary Calibration Services

Josephson Voltage Standard 1 and 10 volt voltage standards based on the Josephson Effect Enabled high-accuracy voltage sources and voltage measuring devices Enabled high-accuracy voltage sources and voltage measuring devices JVS technology transferred to more than 40 government and private organizations JVS technology transferred to more than 40 government and private organizations More efficient calibration services for instrument manufacturers and users More efficient calibration services for instrument manufacturers and users Surveyed Benefit-to-Cost Ratio: 5:1

Maximizing Impact

Strategic Planning to Meet Changing Demands: NIST 2010 NIST 2010 is designed to: Envision the Institute’s future out to Envision the Institute’s future out to Establish long-term strategic goals, and Establish long-term strategic goals, and Implement a plan for achieving those goals. Implement a plan for achieving those goals.

Major NIST 2010 Activities Refining the NIST mission, vision and core values. Refining the NIST mission, vision and core values. Refining criteria for NIST program selection. Refining criteria for NIST program selection. Developing framework for program management. Developing framework for program management. Developing customer feedback mechanisms. Developing customer feedback mechanisms. External input on future environment for NIST and our stakeholders. External input on future environment for NIST and our stakeholders.

Planning for Maximum Impact Greater impact through improved: Productivity Market access Public benefit NIST Industrial Liaison Office KnowledgeNet Web-based project database NIST 2010 external: S&T environment NIST 2010 internal: NIST organizational opportunities Focus NIST on customer needs Focus NIST on longer- term goals Strategic management

Assessing the S & T Environment: NRC Report on Future R&D Environments Commissioned National Research Council to examine forces and trends over the next 5-10 years pertinent to NIST’s mission. Push Factors,i.e., developments likely to come from within or at the intersection of: Biological Science & Engineering Biological Science & Engineering Materials Science & Technology Materials Science & Technology Computer/Information Science & Technology Computer/Information Science & Technology Contextual Factors: Patterns of public & private investment Patterns of public & private investment Where research is done and by whom Where research is done and by whom Effectiveness of the educational system Effectiveness of the educational system Settings for innovation Settings for innovation Pull Factors that encourage direction of S & T development: Combating terrorism Combating terrorism Environmental concerns Environmental concerns Social/cultural concerns Social/cultural concerns

Assessing the S&T Environment: Strategic Focus Areas (SFAs) Externally Focused Internally Focused Healthcare Healthcare Nanotechnology Nanotechnology Information Technology/ Knowledge Management Information Technology/ Knowledge Management Homeland Security Homeland Security People People Customer Focus Customer Focus IT Infrastructure IT Infrastructure

SFA Process: Where We Are Identify Priority strategic needs within each SFA. Priority strategic needs within each SFA. Resources required for success. Resources required for success.Analyze Balance of priorities and resources. Balance of priorities and resources. Recommend & Plan Develop 3-5 year near-term plan identifying the strategic needs to pursue, investment levels, and mechanisms for funding. Develop 3-5 year near-term plan identifying the strategic needs to pursue, investment levels, and mechanisms for funding.Operationalize Plan within the NIST Laboratories. Plan within the NIST Laboratories. April May June August

Assessing S & T Opportunities: Prospective Economic Studies Planning studies provide quantitative and qualitative assessment of technical infrastructure needs. Methodology Methodology Assess technical infrastructure / market failure Assess technical infrastructure / market failure Survey affected industries to estimate costs owing to market failure Survey affected industries to estimate costs owing to market failure Identify potential NIST roles to address infrastructure failure Identify potential NIST roles to address infrastructure failure

Prospective Study Examples Interoperability Cost Analysis of the U.S. Automotive Supply Chain (1999) Identifies types of costs incurred owing to inadequate standards for data exchange among automotive manufacturers and their suppliers Identifies types of costs incurred owing to inadequate standards for data exchange among automotive manufacturers and their suppliers Estimated auto sector cost due to imperfect interoperability: $1B / year Estimated auto sector cost due to imperfect interoperability: $1B / year Inadequate Infrastructure for Software Testing (2002) Identifies costs to software developers and software users owing to software errors Identifies costs to software developers and software users owing to software errors Focused on two industry sectors: transportation equipment and financial services Focused on two industry sectors: transportation equipment and financial services Estimated costs in 2 sectors due to inadequate software testing: $5B / year Estimated costs in 2 sectors due to inadequate software testing: $5B / year Extrapolating results to entire U.S. economy: $60B / year Extrapolating results to entire U.S. economy: $60B / year

Budgeting for Maximum Impact Strategic Focus Area Findings (Evolving) NIST 2010 Strategic Plan Design Review Offsite FY04 Budget Proposals VCAT Review Lab Council Review

SFAs and FY04 Budget Proposals Nanotechnology Nanoscale Measurement Science Nanoscale Measurement Science Quantum Information Science Quantum Information ScienceHealthcare Health Care Quality Assurance Health Care Quality AssuranceIKM Measurements and Standards for a Knowledge-Based Economy Measurements and Standards for a Knowledge-Based Economy Homeland Security Measurement Infrastructure for Homeland Security Measurement Infrastructure for Homeland Security Protecting the Nation’s Electric Power Grid Protecting the Nation’s Electric Power Grid Support for Emergency First Responders Support for Emergency First Responders

Administration FY04 R&D Priorities Interagency Priorities for R&D Budgets Homeland Security & Antiterrorism R&D Homeland Security & Antiterrorism R&D Networking & Information Technology R&D Networking & Information Technology R&D National Nanotechnology Initiative National Nanotechnology Initiative Molecular-level Understanding of Life Processes Molecular-level Understanding of Life Processes Climate Change Science & Technology Climate Change Science & Technology Education Research Education Research Investment Criteria Relevance – why is this investment important Relevance – why is this investment important Quality – how will funds be allocated to ensure quality R&D Quality – how will funds be allocated to ensure quality R&D Performance – how will this investment be monitored for performance Performance – how will this investment be monitored for performance

Summary: Measuring & Maximizing Impact Focus Technical Work for Maximum Impact Fewer, larger projects aligned to highest impact areas. Fewer, larger projects aligned to highest impact areas. Strategic planning - changing demands on NIST. Strategic planning - changing demands on NIST. Leverage Resources Through External Partnerships Cannot “do it all”. Cannot “do it all”. Utilize unique capabilities outside of NIST. Utilize unique capabilities outside of NIST. Work with others to develop needed reference materials, standards, and measurements. Work with others to develop needed reference materials, standards, and measurements. Tele-calibration - utilize internet to deliver services. Tele-calibration - utilize internet to deliver services. Develop strategic partnerships both nationally and internationally. Develop strategic partnerships both nationally and internationally. Measure Impact – Manage for Results Peer review, true customer feedback. Peer review, true customer feedback. Analyze economic impact data. Analyze economic impact data. Track outputs closely: SRMs, calibrations, technical standards, etc. Track outputs closely: SRMs, calibrations, technical standards, etc.