Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 1 Technology Program Evaluation: Methodologies.

Slides:



Advertisements
Similar presentations
Interview Skills Training
Advertisements

Evaluation Mary Rowlatt MDR Partners. Definition of project evaluation Evaluation focuses on whether the project was effective, achieved its objectives,
Office of Research, Trent University Social Sciences and Humanities Research Council of Canada An Overview of Standard Research Grants Prepared by the.
V Alyssa Rosemartin 1, Lee Marsh 1, Ellen Denny 1, Bruce Wilson USA National Phenology Network, Tucson, AZ; 2 - Oak Ridge National Laboratory, Oak.
Principal Patent Analyst
Software Engineering Institute Carnegie Mellon University Pittsburgh, PA Sponsored by the U.S. Department of Defense © 1998 by Carnegie Mellon.
Falling Short? College Learning and Career Success
Job Analysis and Rewards
EUPIDE - June 12th & 13th 2008 "Doctoral Consulting" A partnership between universities and enterprises toward a better recognition of doctors' skills.
Pittsburgh, PA Software Engineering Institute Carnegie Mellon University Pittsburgh, PA Sponsored by the U.S. Department of Defense.
FACULTY OF Management Sciences Department of Office Management and Technology 1 THE IMPORTANCE OF USING INFORMATION SYSTEMS AS VEHICLES FOR KNOWLEDGE PRODUCTION.
Copyright Course Technology Chapter 9: Project Communications Management.
Lecture Nine Database Planning, Design, and Administration
4 4 By: A. Shukr, M. Alnouri. Many new project managers have trouble looking at the “big picture” and want to focus on too many details. Project managers.
project management office(PMO)
Regulating the engineering profession 1 EC UK Experience in Accreditation of Engineering Programmes Professor Ian Freeston University of Sheffield, UK.
FLCC knows a lot about assessment – J will send examples
© 1998 Concept Five Technologies Enterprise Application Integration Capability Maturity Model.
Website Content, Forms and Dynamic Web Pages. Electronic Portfolios Portfolio: – A collection of work that clearly illustrates effort, progress, knowledge,
What is Business Analysis Planning & Monitoring?
Santiago Canyon College Profile Orange, California Technology Integration Profile Arranged by Maureen Roe.
Database System Development Lifecycle © Pearson Education Limited 1995, 2005.
© VESP International Pty Limited To Contents Slide CLICK to advance slides/ bullet points within slides Integrated Master Planner An Overview.
PDHPE K-6 Using the syllabus for consistency of assessment © 2006 Curriculum K-12 Directorate, NSW Department of Education and Training.
Student Learning Objectives: Considerations for Teachers of Career and Technical Education Courses Name Title Date 1 Copyright © 2014 American Institutes.
Evaluation in the GEF and Training Module on Terminal Evaluations
STRATEGY OF LIFELONG LEARNING AT THE UNIVERSITY OF NIS
Chapter 6 : Software Metrics
GBA IT Project Management Final Project - Establishment of a Project Management Management Office 10 July, 2003.
Creating a Shared Vision Model. What is a Shared Vision Model? A “Shared Vision” model is a collective view of a water resources system developed by managers.
Crossing Methodological Borders to Develop and Implement an Approach for Determining the Value of Energy Efficiency R&D Programs Presented at the American.
1 Analysing the contributions of fellowships to industrial development November 2010 Johannes Dobinger, UNIDO Evaluation Group.
LLP-LdV-TOI-2007-FR-008 VALORIS TIQSS. LLP-LdV-TOI-2007-FR-008 The partners of the project.
National Institute of Standards and Technology Technology Administration U.S. Department of Commerce Accelerating Emerging Technologies to the Marketplace.
BIS3324 Group Assignment – Schedule, guideline & templates (2015-Jan Semester)
HPN: IFSS1 Intelligent Flight Support System (IFSS) A Real-Time Intelligent Decision Support Prototype PRESENTER/COTR Anthony Bruins (X37071) HPN Software.
You’re the Entrepreneur Lesson 11 Slide 11A. What Does That Mean? TermDefinition design engineera person educated as an electrical, mechanical, chemical,
Let Ascension take your business to new heights Tender Manager Scott Warnock Andrew Smillie.
Documentation As a Tool for Training and Communication Sara Spiegel AJL NYMA Cataloging Workshop 2005.
ONTARIO CENTRES OF EXCELLENCE Briefing for the York Boot Camp July 19, 2012.
Pilot Projects on Strengthening Inventory Development and Risk Management-Decision Making for Mercury: A Contribution to the Global Mercury Partnership.
Senior Evaluation Officer GEF Independent Evaluation Office Minsk, Belarus September 2015 Evaluation in the GEF and Training Module on Terminal Evaluations.
Qualifications Update: Human Biology Qualifications Update: Human Biology.
Africa Regional Center (IIRR-ARC) The Writeshop Process Introducing…
ELECTRICAL ENGINEERS KENDALL HIMEL INTRO TO ENGINEERING 4TH.
Consultant Advance Research Team. Outline UNDERSTANDING M&E DATA NEEDS PEOPLE, PARTNERSHIP AND PLANNING 1.Organizational structures with HIV M&E functions.
Evaluate Phase Pertemuan Matakuliah: A0774/Information Technology Capital Budgeting Tahun: 2009.
Competency based learning & performance Ola Badersten.
Course, Curriculum, and Laboratory Improvement (CCLI) Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics PROGRAM.
Industry’s Perspective on Industry-University Intellectual Property External Research Directors Network Industrial Research Institute, Inc. April 17, 2001.
National Institute of Standards and Technology U.S. Department of Commerce Technology Program Evaluation: Methodologies from the Advanced Technology Program.
National Institute of Standards and Technology U.S. Department of Commerce Technology Program Evaluation: Methodologies from the Advanced Technology Program.
National Institute of Standards and Technology U.S. Department of Commerce Technology Program Evaluation: Methodologies from the Advanced Technology Program.
Foundations of Information Systems in Business
Student Growth Model Salt Lake City School District Christine Marriott Assessment and Evaluation Department Salt Lake City School District State.
Impact Evaluation of the Program for the Development of the Industrial Districts in Brazil Impact Evaluation Workshop Multilateral Investment Fund Washington.
Encouraging Faculty Development Where You Can: Using a Review of Assessment Management Software Joanne Liebman Matson | Belinda Blevins-Knabe University.
Organisation Control KPI’s & an industry Review
Monitoring and Evaluation
The Five Secrets of Project Scheduling A PMO Approach
Collaborative Expedition Workshop #71 National Science Foundation
Developing a Student Learning Outcomes Assessment Plan and Report
Identify the Risk of Not Doing BA
Grant Writing Information Session
PRMSP/AlACiMa Project
Kari Kleine Educating for Technology Entrepreneurship Through Scientist-Student Collaboration Kari Kleine SDU Technology.
Measuring the Benefits of the Advanced Technology Program
Training programme MODULE I Project number BE02-KA
Evaluation in the GEF and Training Module on Terminal Evaluations
Portfolio My Portfolio Education Work Learning & Career Skills.
Presentation transcript:

Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 1 Technology Program Evaluation: Methodologies from the Advanced Technology Program Status Reports: Measuring against Mission 1 Lee Bowes Stefanie Cox Technology Innovation Program National Institute of Standards and Technology Collaborative Expedition Workshop #71 National Science Foundation March 18, 2008

Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 2 Table of Contents  Why CPRS?  How does it work?  Aggregate results  Using the data Telling the Story Measuring Outcomes: ATP’s Composite Performance Measurement System (CPRS)  Research & writing process  Sample project

Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 3 ATP’s interim goals were defined as: (1) creating knowledge, (2) disseminating knowledge, and (3) commercializing results. The long-term goal was (4) achieving broad-based national benefits Status Reports were individual assessments of funded research projects with an aim to objectively measure against mission, gathering project data to compare with the goals. Status Report Project Defining the Mission

Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 4 4 Status Reports Overview What are they? –The full story of one project in lay language, 5-10 pages –Description of short-, mid-, long-term outputs, outcomes, impacts How are they written? –Team of technology writers review documents, interview companies, search for accurate information about the technology, company, and industry How are they used? –To understand the company perspective –To put the technology in context of state-of-the-art trends –To consider the context of industry and market trends

Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 5 5 Collecting the Data Status Report Evaluation Process RIGHT CLICK TO INSERT 6 Evaluated research projects' preliminary impact and assessed potential broad-based national benefits. Status reports assessed projects at least three to five years after the end of the project, telling the story of the data. Evaluated projects regardless of outcomes, that is, without an emphasis on successes.

Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 6 6 Example Project Abnormal Situation Management (ASM) Project aimed to develop tools and processes to support a comprehensive, automated decision- support system. Knowledge Creation: 8 patents Knowledge Dissemination: Consortium of 13 companies; 21 publications, 15 presentations Commercialization Progress: 20+ commercial products and services around security, training, architecture, etc. 11 Abnormal situations range from suboptimal performance to production delays to costly fires and explosions.

Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 7 7 Example Project, ASM cont. A Patent Tree for each patent visually demonstrates knowledge spillover. A patent tree is a graphical display of knowledge dissemination from patented technological advances Knowledge Creation: Patent Tree Patent tree (follow-on patents) for No. 6,414,594, labeled by patent number Patent tree for No. 6,587,108, labeled by Assignee Name

Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 8 8 Example Project, ASM cont. Control Room -- BeforeControl Room -- After

Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 9 9 Status Reports: Measuring against Mission Role of the CPRS – Why? A portfolio approach based on objective criteria facilitates cross-project analysis. 2 ATP asked itself how to measure project results objectively. The system must measure quantifiable data and allow multiple paths to success. One project has publications but no commercial progress. Another has extensive collaboration but a dismal outlook due to changes in market forces. Another has products on the market, but no publications or patents, because the company wants to keep IP secret.

Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 10 Rating the Project 4 Components from ATP’s Mission include:  Scientific and technical knowledge creation  Collaboration and knowledge dissemination  Technology commercialization progress and diffusion of benefits Progress Overall Performance + + Knowledge Creation Knowledge Dissemination Commercialization Overall Performance Overall Performance Progress KC KD CP

Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 11 Distribution of CPRS Ratings

Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 12 Sample Projects Technical awards000 Publications106 Presentations0010 Types of Collaborations211 New products/processesNoYes (1)Yes (2) Attract capitalNoYes Employment gains-100%N/A180%+ Business awards004 Outlook (+4=optimistic)-4+4 Examples from the Information Technology Group Optical tape storage technology with up to 180 tracks for rapid data storage, retrieval and transfer. Tools for debugging applications built from component software to increase reliability. New Process for assuring the security of software components, enabling technology for e-commerce. Overall Performance Score 0 Star 2 Stars 4 Stars LOTS Technology Cigital, Inc. L3 Communications

Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 13 Aggregate Data Profile Comparison of Star Ratings

Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 14 Aggregate Data, cont. How Star Rating Averages Differ among Project Attributes Profile Comparison of Star Ratings

Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 15 Commercial Activity Is Highly Associated with Collaboration Aggregate Data, cont. Profile Comparison of Star Ratings

Status Reports: Measuring against Mission National Institute of Standards and Technology U.S. Department of Commerce 16 Conclusion Status reports are stories that:  Convey the outputs, products, and long-term impacts of project development  Gather and document outcome data for cross-project analysis Discussion and questions