Record Keeping Studies – Love ‘Em or Leave ‘Em National Science Foundation Division of Science Resources Statistics International Conference on Establishment Surveys Montreal, Quebec, CA June 19, 2007 Jeri Metzger Mulrow, National Science Foundation Stanley Freedman & Robert Rutchik, Energy Information Administration National Science Foundation Division of Science Resources Statistics

Outline of Today’s Talk National Science Foundation Division of Science Resources Statistics Background and impetus for study What we hoped to learn The record keeping study design Results and some conclusions Lessons learned

NSF Survey of Industrial R&D Background Collects information on research and development (R&D) Long running, annual survey (1953) Company level (not establishment level) Intended to cover all for-profit, public or private, non-farm companies with 5 or more employees operating in the U.S. Self administered paper form Response rates ~ 80% National Science Foundation Division of Science Resources Statistics

National Science Foundation Division of Science Resources Statistics Industr y Expert Panel Data User Needs Workshops Record Keeping Study Process & Methods Evaluation s Methodological Work – Response Studies, Cognitive Interviews, Respondent Debriefings Redesign Efforts Literature Review

OMB Guidance – Pre-Survey Design Visits for Establishment Surveys Is the information available? How closely aligned are the records with the requested data? Are the record keeping systems compatible with the collection instruments? How difficult is it to provide the data? National Science Foundation Division of Science Resources Statistics

NSF Questions of Interest Where are the records kept? Who has access to them? How many record keeping systems exist in the organization? Which ones contain the data of interest? How knowledgeable is the person about the data? How often are the records updated? What specific data items are kept? National Science Foundation Division of Science Resources Statistics

National Science Foundation Division of Science Resources Statistics Overview of Recordkeeping Study National Science Foundation Division of Science Resources Statistics Preparations (October June 2005) Background interviews and project design Question and protocol development Phase 1: (July-September 2005) Broad understanding of firms’ R&D definitions, organization and availability of records Phase 2: (June-August 2006) Detailed review of financial recordkeeping practices Phase 3: (TBD) Recordkeeping practices on specific priority areas

National Science Foundation Division of Science Resources Statistics Preparations October June 2005 Eight (8) pre-interviews with R&D or innovation experts –Business schools –Accounting firms –Government organizations Developed protocol Developed recruitment strategy Tested with five (5) companies Revised strategy and protocol

National Science Foundation Division of Science Resources Statistics Phase 1 July - September company visits Sample of companies –Used public lists –Unknown if respondents to R&D survey –Covered a variety of industries –Included 10 small companies (Small Business Innovation Research) Did not mention the R&D survey Broad understanding of how the company performed R&D How they defined R&D General availability of records

National Science Foundation Division of Science Resources Statistics Phase 2 June – August 2006 Evaluated the results from Phase 1 Revised the approach –Focused more on large companies –Focused on specific information and records –Requested to speak to people familiar with the financial records of the company rather than the R&D 9 company visits Did not mention the R&D survey

National Science Foundation Division of Science Resources Statistics Industry by Company Size (to date)

National Science Foundation Division of Science Resources Statistics Job Function by Company Size (to date)

National Science Foundation Division of Science Resources Statistics What happened? Pre-interviews provided good insights Difficult to gain access to companies Companies did not provide access to records However --- Extremely valuable to talk with companies Gained useful knowledge

National Science Foundation Division of Science Resources Statistics Did we answer our questions? Where are the records kept? –Different records kept in different places Who has access to them? –Different people have access to different types of information How many record keeping systems exist in the organization? –Financial/accounting often separate from HR separate from licenses and patents Which one contains the data of interest? –All of them National Science Foundation Division of Science Resources Statistics

National Science Foundation Division of Science Resources Statistics Did we answer our questions? (2) How knowledgeable is the person about the data? –We did not speak to the people who created the data –People in one part of the organization did not know about how things were done in other parts of the organization How often are the records updated? –? What specific data items are kept? –Varies across companies National Science Foundation Division of Science Resources Statistics

What did we learn about R&D? Most companies do not have a formal definition of R&D Informal definitions of R&D are not consistent within or across companies The line between research (R) and development (D) is not clear or consistent R&D is done in a variety of ways –In a central lab –At each business line –Anywhere in between National Science Foundation Division of Science Resources Statistics

What else did we learn? R&D details are tracked and managed at the project level Companies can provide a total cost for R&D BUT the components vary Companies know who is paying for the R&D – themselves or others Companies can separate domestic from international spending Many collaborate with universities to do R&D National Science Foundation Division of Science Resources Statistics

National Science Foundation Division of Science Resources Statistics Our conclusions Provide clear definitions of R&D on the survey Tell them what to include or exclude in the R&D totals Involve more than one person to answer the survey High level R&D data are available but many of the details aren’t kept in the record keeping systems

National Science Foundation Division of Science Resources Statistics Lessons Learned Company visits are extremely valuable Flexibility is good, but still need a framework Talk to more than one person in the organization, if possible The more you know, the more you learn Record keeping is just one piece of the puzzle

Love ‘Em or Leave ‘Em? National Science Foundation Division of Science Resources Statistics