1. Lessons From Seven Years of Study of the Research Environment Presented at the American Evaluation Association/Canadian Evaluation Society Joint Conference Toronto, Canada October 28, 2005 Gretchen B. Jordan Sandia National Laboratories gbjorda@sandia.gov Work presented here was completed for the U.S. DOE Office of Science by Sandia National Laboratories, Albuquerque, New Mexico, USA under Contract DE-AC04-94AL8500. Sandia is operated by Sandia Corporation, a subsidiary of Lockheed Martin Corporation. Opinions expressed are solely those of the author.

2. G. Jordan 10/17/2005 2 Outline Motivation, Overview of DOE project Conceptual framework Overview of research environment survey Lessons learned from survey Future research

3. G. Jordan 10/17/2005 3 Motivation for assessing organizational effectiveness Desire to define strategies to improve research effectiveness –Concerns the research environment is deteriorating –Add to very slim study to date of management of science –Organize thinking about differences in R&D, organizations, and circumstances –Examine multiple levels and linkages among levels, such as differences in R&D within a portfolio of projects And have a reasonable response to public demand for demonstrating accomplishments –Legislative and administrative requirements (GPRA, PART) –Need for a leading indicator and one to balance the emphasis on outcome assessment

4. G. Jordan 10/17/2005 4 DOE project overview Research by Sandia labs in collaboration with Dr. Jerald Hage and the Center for Innovation, University of Maryland Sponsored by U. S. Department of Energy (DOE) Office of Basic Energy Sciences to define innovative measures of operational and scientific performance. Concentration has been on – Understanding and developing theory relating the research environment to broader management of R&D – Tools to assess key factors in the research environment that foster excellence and impact in order to improve these.

5. G. Jordan 10/17/2005 5 Evolution of the project 19 focus groups (DOE, industrial, university) and extensive literature review Defined attributes and organized within the Competing Values Framework (Cameron, Quinn, et al), then extending and validating that framework for R&D. Developed, tested and refined a survey to capture employee perceptions of their research environment –To link to nature of work –To analyze and present data to encourage action plans Have used the survey in conjunction with case studies to determine impact of specific management interventions Beginning to link survey findings with data on performance, solving challenges associated with this Beginning to develop management and measurement models for multiple levels (project, R&D organization, sector)

6. Conceptual Framework

7. G. Jordan 10/17/2005 7 Assessing the research/work environment focuses on an important part of S&T evaluation Within an external environment economic technical political/legal social/ demographic funds people and their characteristics knowledge base/ competencies Resources Meet Customer “Products & Services” That Needs basic or applied research or development phase of innovation S&T Organizational Effectiveness Purpose/ Mission Achieved new knowledge trained students public goods economic advantage informed public debate

8. G. Jordan 10/17/2005 8 Internal Processes Perspective Flexible, minimum possible bureaucracy to maintain required controls but not impede innovation The culture appropriately encourages risk taking, exploration, internal and external scanning, cooperation and collaboration Strategic planning decisions, with progress measurement, balance continuity with disinvestment Organizational Mission & Financial Perspective Existing/ new customers value our innovativeness enough to invest in it Vision: Effectively innovate and better harness innovation for sustainable service to the mission S & T Learning & Growth There is a stable yet dynamic S & T base to support mission (ideas, people, facilities, knowledge/skills ) Innovation Cycle (readiness levels) New and improved products and product platforms developed with functionality required Technologies mature toward customer needs Concepts proved and progress to development for key technologies Customer Perspective Serve needs/solve problems for existing and new customers Our framework describes tensions inherent in management interventions to improve innovativeness DRAFT 04/20/05 Informs management -Strategy Map for Improving New Product Innovation Organizational Learning & Growth Barriers and bottlenecks to innovation are identified and coordinated initiatives are in place to remove them.

9. G. Jordan 10/17/2005 9 Conceptual framework behind the survey Contingency theory of organizational effectiveness says performance is highest when an organization’s structure matches its strategy (for a given set of external circumstances) Not all R&D has the same strategy (e.g. amount of risk or scope, reasons for taking on larger amounts) so there are tensions inherent in management These can be captured with two dimensions of strategy and two related dimensions of structure Dimensions of strategic choices -Where to be on continuum from incremental to radical -Where to be on continuum of narrow to broad/ systemic scope Related are structural choices and tensions - Research autonomy vs. coordination - Specialized vs. complex teams - Organizational autonomy vs. inter-organizational ties

10. G. Jordan 10/17/2005 10 Two dimensions result in four Research Profiles Quality Outcome Profiles build on and names are borrowed from the Competing Values Framework, Cameron and Quinn, 1999 and DeGraff, 2002 Incremental Advance Specialized Task Intra Organizational Broad Scope of Focus Large, Coordinated Programs Narrow Scope Advance Small, Autonomous Projects Radical Advance Complex Task Inter Organizational How you manage depends on your profile organic or hierarchical structure inter organizational ties or not Be Sustainable Be New Be First Be Better

11. G. Jordan 10/17/2005 11 Related to research strategy and structure profiles are management practices in four areas Desired Strategy and Outcomes RTD Strategy Profiles Radical Advance Incremental Advance Broad Scope Narrow Scope RTD Structure/Management Profiles Structure and Work Environment Be Sustainable Be FirstBe Better Be New Complex, External Specialized, Internal Small, Research Autonomy Large, Research Coordination Time to Explore Intellectual Integration Encourage Change Values Individuals Teamwork HR Development Rich in Resources Technical Management Low burden Systems Vision and Strategies Plan & Execute Strategic Relationships

12. Overview of the Survey

13. G. Jordan 10/17/2005 13 Key attributes of the research environment were determined through … Information from 19 focus groups of scientists and managers at three DOE laboratories, one industry lab, and one university “What do you need in your research environment to do excellent work?” “What attracted you to the lab and what keeps you here?” Study of current literature for links between excellence, quality outputs and the work environment -Included R&D management, innovation, organizational development/innovation, evaluation literatures Developed and tested survey questions √PNNL EHSD Division in 1999, Ford Research Lab in 2000 √SNL – 3 Centers in 1998, 17 Centers in 2001 and 2003 √SNL and NOAA case studies in 2003-2004 √NMSU in 2005

14. G. Jordan 10/17/2005 14 DOE Research Environment Survey is a diagnostic tool looking at 36 attributes of management practices Clearly Define Goals & Strategies Research Vision & Strategies Sufficient, Stable Funding Investing in Future Capabilities Plan and Execute Well Project Planning & Execution Project-Level Measures of Success Laboratory-Wide Measures of Success Build Strategic Relationships Relationship with Sponsors Champion Foundational Research Reputation for Excellence Organizational Effectiveness Provide Capital, Knowledge Resources Equipment & Physical Work Environment Research Competencies/Knowledge Base Salaries & Benefits Ensure Good Technical Management Informed, Decisive Management Rewards & Recognition Internal Resource Allocation Insist on Efficient, Low Burden Systems Laboratory Services Laboratory Systems & Processes Competitiveness/Overhead Rates Value the Individual Respect for People Optimal Use of Skills Management Integrity Build Teams and Teamwork Teamwork & Collaboration Internal Communication Value-Added Management Commit to Employee Growth Technical Career Advancement Educational & Professional Development Quality of Staff Encourage Exploration, Risk Taking Time to Think & Explore Pursuit of New Ideas Autonomy in Decision-Making Integrate Ideas, Internally & Externally Internal Cross-Fertilization of Technical Ideas External Collaborations & Interactions Integrate Ideas & R&D Portfolio Encourage Change & Critical Thinking Sense of Challenge & Enthusiasm Commitment to Critical Thinking ID New Projects and Opportunities Innovativeness Support Systems Setting Goals Human Resource Development

15. G. Jordan 10/17/2005 15 Survey components 20-25 minute (often web-based) survey includes Rate status of 36 attributes (and preferred for some of these) Overall rating the environment and trend Questions of specific interest to that organization Questions on characteristics of their projects Limited demographics (org. unit, source of funds, job classification, career stage, years at lab) Two Open-ended questions: major barriers to progress and who can do something about it; Other Longer survey includes sub parts on some attributes Shorter survey asks only 18 of the 36 attributes

16. G. Jordan 10/17/2005 16 Question format Each overall question is itself complex and defined in a statement that combines notions from sub-parts from previous long surveys Response is “what percent of the time is this true during past year” (ratio scale) which permit benchmarks Not all attributes should or could be true 100 percent of the time there are trade offs it will differ depending on nature of work and circumstances 2. People have time to think creatively and explore. (I have time to do my research, think creatively, and explore new approaches, all in a normal work week. I don’t work on too many projects at once and am free from excessive organizational obligations.)  0 to 20%  21 to 40%  41 to 60%  61 TO 80%  81 to 100%  NA

17. G. Jordan 10/17/2005 17 Researchers can suggest areas for improvement by providing time preferred 2. People have time to think creatively and explore. (I have time to do my research, think creatively, and explore new approaches, all in a normal work week. I don’t work on too many projects at once and am free from excessive organizational obligations.)  0 to 20%  21 to 40%  41 to 60%  61 TO 80%  81 to 100%  NA 2b. For your profile, for what percent of the time should people have time to think creatively and explore?  0 to 20%  21 to 40%  41 to 60%  61 TO 80%  81 to 100%  NA 0% - 20% 21% - 40% 41% - 60% 61% - 80% 81% - 100% 3% 18% 23% 38% 18% 24% 22% 19% 11% 0% 20% 40% 60% 80% 100% Q2. Have Time to Think Creatively Now Preferred Managers can assess the appropriateness of requests for change, knowing the nature of the work and current circumstances

18. G. Jordan 10/17/2005 18 Analysis by demographic groups helps tailor management actions (e.g. by Job Classification) Significantly Better Significantly Worse *Significant at.05 Data shown here are notional. One job classification or level may have issues another does not (Level 3 in this case) The most unsatisfied do not reveal their job classification Managers (Level 1) often rate the lab higher than staff

19. G. Jordan 10/17/2005 19 Looking at differences across organizations provides benchmarks, stimulates action Very Poor to Fair Staying the Same Getting Worse Improving Good to Outstanding Key: Values represent the percentage of responses Department 1 Department 2 Department 4 Department 3 Department 5 Department 7 Department 8 Department 9 Department 6 Overall Satisfaction? ( Good or Better) Getting Better, Same, Worse? ( Getting Better) Data shown here are notional.

20. G. Jordan 10/17/2005 20 Analyzing differences across time can be useful, especially when tied to management or external changes in that period ANOVA Table 2001-2003 Data shown here are notional.

21. A Few Lessons Learned and Continuing Research

22. G. Jordan 10/17/2005 22 Areas of Agreement Among 40 Research Organizations (2200 staff in three different laboratories) Highest Favorable RatingsLowest Favorable Ratings Quality of staff (37) Respect for people (26) Equipment & physical environment (25) Sense of challenge & enthusiasm (23) Autonomy (18) Identifying new projects/opportunities (28) Rewards & recognition (27) Internal research funds allocation (26) Laboratory-wide measures of success (16) Reducing overhead rate/burden (15) Drivers of Satisfaction (in top ten) Drivers of View on Trend (in top ten) Research vision & strategies (21) Invests in future capabilities (19) Sense of challenge & enthusiasm (19) Identification of new opportunities (17) Project level measures of success (17) Research vision & strategies (27) Investment in future capabilities (28) Identification of new opportunities (20) Decisive, Informed management (19) Champion long term research (18) Reward and recognize merit (18) What is important to R&D workers? Note: Does not include data from 2003 forward

23. G. Jordan 10/17/2005 23 Nature of the work: basic science vs. technological tasks Complexity of labor: 6+ departments vs. less Size: small (< $1M) vs. large projects For example, as expected we observe a steady decline in group means as move from small/less complex to large/complex Small/complex rate external collaboration lower than the other groups Large/complex rate investment in future capabilities lower Survey combined with case study project data shows significant differences across project characteristics

24. G. Jordan 10/17/2005 24 DRAFT 07/25/2004 Advance scientific understanding, capabilities Data Coverage, Reliability, and Predictability For Ecosystems, Oceans, Climate and Weather, Commerce, Infrastructure Sustained data quality in changing conditions Sound, state of the art science Accessible and Cost-effective Data Products and Services Able to assimilate large amounts of data and diverse sources, technologies Seen as good use of taxpayer dollars Responsive to user needs, dynamic conditions Develop new R&D products, services Select a relevant set of R&D projects Robust flow of knowledge among researchers and users Transfer, calibrate, and maintain products, services Managerial Excellence R&D Products and Services Provide Value to Users Good operating satellite service for the nation and global community Secure funds and handle responsibly  # publications, citations  Peer review on amount of learning  # products by type, milestones  Technical progress (sum of multiple attributes of performance)  # products transferred  Level of quality sustained  Funds in by type  $ cost share, leverage  Customer satisfaction  Cost avoidance indicator  Project mix in portfolio  Grants awarded  Cycle time by type  Quality, innovativeness  Leadership positions  Workshops, web hits A motivated, prepared, well-equipped, and networked workforce Excellent Climate for Research  Existing DOE survey Quality People, Equipment, Computers  IT capabilities Good Business Practices  To be determined Improve existing R&D products, services Execute R&D projects well Customer Perspective Managerial/ Operational Perspective Mission Perspective Example Balanced Scorecard and Strategy Map Organizational Learning Perspective + = + includes health of research environment Includes indicators for different Research Profiles

25. G. Jordan 10/17/2005 25 Areas of current and future research Continue to define and validate theory for evaluating the research environment within various contexts, moving to portfolio level Continue to refine research environment survey –Improve nature of work questions –Add data on external influences, both technical and non –Add comparisons across multiple organizations, fields –Develop a menu of sub part questions that drill down in some areas such as mechanisms for cross fertilization of ideas Analyze performance data to link specific performance with presence of specific attributes –Peer review findings –Real time progress measures linked to use of that progress Continue to build a data base of ratings of attributes by nature of work, level of performance of the group, and external circumstances

26. G. Jordan 10/17/2005 26 Selected References Jordan, Gretchen, “What is Important to R&D Workers”, Research Technology Management, Vol. 48 No. 3, May-June 2005. Jordan, Gretchen, Jerry Hage, Jonathon Mote and Bradford Hepler, “An Exploration of Differences Among R&D Projects and Implications for R&D Managers,” R&D Management Journal, forthcoming November 2005. Jordan, Gretchen, “Factors Influencing Advances in Science and Technology: Variation Due to Diversity in Research Profiles,” Chapter in Innovation, Science, and Macro Institutional Change: Knowledge Dynamics, Jerald Hage, Marius Meeus, Editors, Oxford University Press, forthcoming in 2006. Jordan, Gretchen, L. Devon Streit, and J. Stephen Binkley, “Assessing and Improving the Effectiveness of National Research Laboratories,” IEEE Transactions in Engineering Management, 50, no.2 (2003): 228-235. Jordan, G.B. and L.D. Streit. “Recognizing the Competing Values in Science and Technology Organizations: Implications for Evaluation,” in Learning From Science and Technology Policy Evaluation, Shapira, Philip and Kuhlman, Stefan, Eds., Edward Elgar, Cheltenham, UK and Northampton, Mass. 2003. Please contact me if you have questions, suggestions, or opportunities to collaborate. Email gbjorda@sandia.gov, phone 202-314-3040.gbjorda@sandia.gov