Diane K. Willimack U.S. Census Bureau Any views expressed on statistical, methodological, or technical issues are those of the authors and not necessarily.

Slides:



Advertisements
Similar presentations
Statistics NZs experience in using Administrative Data in an Integrated Programme of Economic Vince Galvin General Manager Strategy & Communications.
Advertisements

1. 2 Considering the Establishment Survey Response Process in the Context of the Administrative Sciences Diane K. Willimack U.S. Census Bureau.
The Response Process Model as a Tool for Evaluating Business Surveys Deirdre Giesen Statistics Netherlands Montreal, June 20th 2007 ICES III.
Sources of Measurement Errors in Business Surveys
Reasons for Evaluating Training Companies are investing millions of dollars in training programs to help gain a competitive advantage. To justify the costs.
Copyright 2010, The World Bank Group. All Rights Reserved. Statistical organization Principles and models Part 1 Strengthening Statistics Produced in Collaboration.
© The McGraw-Hill Companies, Inc., 2002 McGraw-Hill/Irwin Operational Budgeting Chapter 22.
Enhancing Data Quality of Distributive Trade Statistics Workshop for African countries on the Implementation of International Recommendations for Distributive.
Integrating a Gender Perspective into Time Use Statistics.
© Statistisches Bundesamt, Institute for Research and Development in Federal Statistics Statistisches Bundesamt Business Surveys – Testing Strategies in.
9.401 Auditing Chapter 1 Introduction. Definition of Auditing The accumulation and evaluation The accumulation and evaluation Of evidence about information.
Chapter 3: System design. System design Creating system components Three primary components – designing data structure and content – create software –
MBE Research Module 5. Week Prepare for Kick-off Meeting Assign teams Team forming Review and execute consulting contract Interview and.
Consumer Expenditure Survey Redesign Jennifer Edgar Bureau of Labor Statistics COPAFS Quarterly Meeting March 4, 2011.
Introduction to SAP R/3.
JOB ANALYSIS AND HUMAN RESOURCE PLANNING
© 2008 by Prentice Hall4-1 Human Resource Management Chapter 4 JOB ANALYSIS, STRATEGIC PLANNING, AND HUMAN RESOURCE PLANNING.
1 1 Web and Paper Questionnaires seen from the Business Respondent’s Perspective Gustav Haraldsen, Statistics Norway Jacqui Jones, UK Office of National.
Use of administrative data in statistics - challenges and opportunities ICES III End Panel Discussion Montreal, June 2007 Heli Jeskanen-Sundström Statistics.
Developing Business Data Collection and measuring Response Burden European Conference on Quality in Official Statistics Helsinki 4-6 May 2010 Johanna Leivo,
Collaborating for Quality: A Cross-Discipline Approach to Questionnaire Content Evaluation in Business Surveys Diane K. Willimack Peter Gibson U.S. Census.
1. 2 Considering the Establishment Survey Response Process in the Context of the Administrative Sciences Diane K. Willimack U.S. Census Bureau.
1 WORLD TOURISM ORGANIZATION (UNWTO) MEASURING TOURISM EXPENDITURE: A UNWTO PROPOSAL SESRIC-UNWTO WORKSHOP ON TOURISM STATISTICS AND THE ELABORATION OF.
Regional GDP Workshop. Purpose of the Project October Regional GDP Workshop Regional GDP Scope Annual Current price (nominal) GDP By region.
The Future of Statistical Data Collection? Challenges and Opportunities Johan Erikson (Statistics Sweden) Gustav Haraldsen (Statistics Norway) Ger Snijkers.
12th Meeting of the Group of Experts on Business Registers
Version: 2.0 © University of Tasmania All rights reserved. CRICOS Provider Code: 00586B Accounting & Financial Decision Making BFA103BFA103.
HFT 2401 Chapter 1 Introduction to Accounting. Accounting A Means to an End  Provides answers to questions  How much cash do we have  What was our.
HR Planning Introduction & Revision. HRM Is the effective use of an organization's human resources to improve its performance.
S7: Audit Planning. Session Objectives To explain the need for planning To explain the need for planning To outline the essential elements of planning.
Assessing Quality for Integration Based Data M. Denk, W. Grossmann Institute for Scientific Computing.
Needs Assessment EDTC General Definition The process of comparing a desired goal state with existing conditions Data is fundamental to all decision.
Audit Planning. Session Objectives To explain the need for planning To outline the essential elements of planning process To finalise the audit approach.
HOW TO WRITE RESEARCH PROPOSAL BY DR. NIK MAHERAN NIK MUHAMMAD.
2  Mission Statement.  Company’s overall purpose and direction, products, services and values.  Goals.  That accomplish the mission. E.g. 5 year plan.
Coverage and Completeness of Demographic Yearbook Data Demographic and Social Statistics Branch United Nations Statistics Division Expert Group Meeting.
American Community Survey ACS Content Review Webinar State Data Centers and Census Information Centers James Treat, ACSO Division Chief December 4, 2013.
Systems Analysis and Design 8 th Edition Chapter 2 Analyzing the Business Case.
Copyright 2010, The World Bank Group. All Rights Reserved. 1 GOVERNMENT FINANCE STATISTICS INTRODUCTION TO GOVERNMENT FINANCE STATISTICS Part 2 This lecture.
Module 5b: Measuring Household ICT Ms Sheridan Roberts, Consultant Information Society Statistics Tuesday 10 March 2009.
Agile Method Paper Report 資工 4A 余修丞. 2 Agile methods rapidly replacing traditional methods at Nokia: A survey of opinions on agile transformation.
Copyright © 2010 Pearson Education, Inc. Publishing as Prentice Hall 1-1 Organizational Theory, Design, and Change Sixth Edition Gareth R. Jones Chapter.
Introduction to Management
Use of Administrative Data Seminar on Developing a Programme on Integrated Statistics in support of the Implementation of the SNA for CARICOM countries.
New sources – administrative registers Genovefa RUŽIĆ.
The 2006 American Community Survey (ACS) Content Test Questions on International Migration: Improving Data on the U.S. Foreign-Born Dean H. Judson For.
Census of Governments and Related Programs Governments Division Economic Programs Directorate U.S. Census Bureau Presented at McCormick SRI: Going Deep.
12/10/15.  It is a Cross Life Cycle Activity (CLCA) that may be performed at any stage ◦ In fact, some part of it (e.g. risk analysis and management)
Household Economic Resources Discussant Comments UN EXPERT GROUP MEETING 9 September 2008 Garth Bode, Australian Bureau of Statistics.
Feasibility Study.
Q Rome Italy, July Towards Producing and Using Response Burden Data for Establishment Surveys at Statistics Sweden Johan Erikson, Dan Hedlin,
United Nations Oslo City Group on Energy Statistics OG7, Helsinki, Finland October 2012 ESCM Chapter 8: Data Quality and Meta Data 1.
1 Statistical business registers as a prerequisite for integrated economic statistics. By Olav Ljones Deputy Director General Statistics Norway
1 Cognitive Aspects Associated with Sample Selection Conducted by Respondents in Establishment Surveys La Toya Barnett Thomas Rebecca L. Morrison Grace.
1 Overview of Economic Statistics in Africa UNECA Andry Andriantseheno Regional Workshop on Basic Economic Statistics Addis-Ababa October 2007.
Accounting Information Systems An Introduction
HFT 2401 Chapter 1 Introduction to Accounting. Accounting – A Means to an End  Provides answers to questions  How much cash do we have  What was our.
Census quality evaluation: Considerations from an international perspective Bernard Baffour and Paolo Valente UNECE Statistical Division Joint UNECE/Eurostat.
Job Analysis and Evaluation. Definition Job - Consists of a group of tasks that must be performed for an organization to achieve its goals Position -
1 1 Reflections about the Impact Business Questionnaires have on Perceived Response Burden and Survey Quality Gustav Haraldsen Statistics Norway
Workshop on MDG, Bangkok, Jan.2009 MDG 3.2: Share of women in wage employment in the non-agricultural sector National and global data.
1. 2 Issues in the Design and Testing of Business Survey Questionnaires: Diane K. Willimack U.S. Census Bureau Economic Census The International.
McGraw-Hill/Irwin Chapter 1 The Nature and Purpose of Accounting Copyright © The McGraw-Hill Companies. All Rights Reserved.
Administrative Data and Official Statistics Administrative Data and Official Statistics Principles and good practices Quality in Statistics: Administrative.
1 A stylized satellite account for human capital Gang Liu Statistics Norway Presentation at 4th World KLEMS Conference, Madrid, May 23-24,
Copyright © 2012 Pearson Education, Inc. Publishing as Prentice Hall 15-1 # Copyright © 2015 Pearson Education, Inc. The Role of Accountants and Accounting.
MANAGEMENT AND COST ACCOUNTING
MANAGEMENT AND COST ACCOUNTING
Chapter 1 Introduction Prof. G. B. Bhosale
Presentation transcript:

Diane K. Willimack U.S. Census Bureau Any views expressed on statistical, methodological, or technical issues are those of the authors and not necessarily those of the U.S. Census Bureau. The Response Process and Burden in Business Surveys: Implications for Official Statistics Business Survey

Outline  Business vs. household surveys  Business context and –  Survey response process  Response burden  Consequences for error sources  Focus on measurement error  Implications for Official Statistics 2

Businesses vs. Households  What are businesses?  Economic units  Made up of one or more establishments  Locations where business is conducted or industrial operations performed (FCSM, 1988)  Goals –  To produce goods and services  To remain open and viable over tim  To remain open and viable over time 3

Businesses vs. Households  How are business surveys different?  Population  Data  Survey design  Response process 4

Household Surveys vs. Business Surveys Household Surveys Business Surveys Qs are aboutSelf Business or Organization Types of data Autobiographical, Attitudes, Behaviors Information abt the organization and/or its employees or clients; Financial; Quantities Data sourcesMemory Records & R’s knowledge of them Respondent Identity Self or ProxyInformant 5

Business Survey Topics  Business characteristics  Production measured by volume or value  Number of employees, their qualifications, field of work, hours worked, payroll and benefits  Expenditures and investments  Volume sold and financial results (e.g., figures from balance sheets and income statements)  Consumption of energy and other resources  Transactions between businesses  Business trend evaluations  Evaluations of working conditions 6

Household Surveys Business Surveys R selection; R rule Single R or Selected proxy Multiple informants Mode Interviewer administered Self-administered Authority Voluntary or Mandatory Response Process Cognitive Cognitive and Organizational Household Surveys vs. Business Surveys continued 7

8 Complex Response Process in Business Surveys 1)Record formation & encoding 2)Organizing response tasks a)Respondent selection b)Scheduling c)Prioritizing & motivation 3)Comprehension 4)Retrieval 5)Judgment 6)Communication 7)Releasing the data Sudman et al. (2000) / Willimack & Nichols (2010); Bavdaz (2007, 2010) Cognitive Response Process (Tourangeau, 1984) Organizational Steps

What is Response Burden? Traditional Approach:  Burden is the time taken to complete the survey Actual burden = Costs Perceived Response Burden:  Respondents’ perceptions of their survey experience “Respondents rarely equate burden with the time it takes to respond to a survey; instead they perceive burden in relation to mode, who is conducting the survey, and whether the produced statistics are useful to businesses/society.” (Hedlin et al. 2005) 9 R = Number of people involved in completing the questionnaire T = Time taken by people involved C = Salary costs of these people

Causes of Perceived Response Burden (PRB) 10

Respondent(s) Authority Capacity Motivation 11 (Dale et al., 2007) Origins and Flows of Response Burden in Business Surveys

Respondent(s) Authority Capacity Motivation Origins and Flows of Response Burden in Business Surveys Management considerations 4 Burden starts here – with stakeholders’ and data users’ needs.

13 Official Statistics and Stakeholders’ Needs 1. Theoretical world: Conceptualization Research Needs: Questions, Issues, Problems, Decisions Empirical world: Physical Observations Attributes Data Concepts Measurements Question(s) Questionnaire Response

14 1. Theoretical world: Conceptualization Research Needs: Questions, Issues, Problems, Decisions Empirical world: Physical Observations Attributes Data Concepts Measurements Question(s) Questionnaire Response Specification errors Operationalization errors Measurement & Non-response errors Processing errors Official Statistics in the Context of Total Survey Error Through survey development and design, burden transfers from stakeholders & data users to the survey organization.

Origins and Flows of Response Burden in Business Surveys Respondent(s) Authority Capacity Motivation Through survey development and design, burden transfers from stakeholders & data users to the survey organization. So far, nothing particularly unique to business surveys, right? …. …… but we’re just getting to the good part ….. …..the business survey response process ….. …..the business survey response process ….. 3 Management considerations 4

16 Response Process in Business Surveys 1)Record formation & encoding 2)Organizing response tasks a)Respondent selection b)Scheduling c)Prioritizing & motivation 3)Comprehension 4)Retrieval 5)Judgment 6)Communication 7)Releasing the data Sudman et al. (2000) / Willimack & Nichols (2010); Bavdaz (2007, 2010)

17 Response Process: Organizationally Determined 1)Record formation & encoding 2)Organizing response tasks a)Respondent selection b)Scheduling c)Prioritizing & motivation 3)Comprehension 4)Retrieval 5)Judgment 6)Communication 7)Releasing the data

18 Business Reasons for Records Sudman et al. (2000) / Willimack & Nichols (2010)  External requirements:  Legal purposes  Regulatory purposes  Internal business  Manage resources  To remain open and viable  To produce goods and services  Recorded data ≠ requested data  Data are “owned” by the business

19 Response Process: Organizationally Determined Bavdaz (2007, 2010) 1)Record formation & encoding 2)Organizing response tasks a)Respondent selection b)Scheduling c)Prioritizing & motivation 3)Comprehension 4)Retrieval 5)Judgment 6)Communication 7)Releasing the data

Performing Response Tasks  Respondent selection, scheduling and prioritizing the Work to provide a survey response 20 Benefits Costs  Surveys represent a cost without associated benefit – “non-productive cost”

21 The “most knowledgeable” respondent – The person who Knows the records and their location The meaning and purpose of the data in the records – Determined in the context of record formation Response Process: Respondent Selection Who is the Best Respondent? Edwards & Cantor (1991)

22 Division of labor – Compartmentalize the work for efficiency Organizational structures reflect the distribution of the work – Distributed knowledge – Data are located where they are needed and used Response Process: Respondent Selection Who is the “Most Knowledgeable” Respondent? Benefits Costs

23

24 Technical Core –Those whose job is the production of goods & services –And the associated data Boundary Spanners –Interface with the outside world and across internal units Response Process: Respondent Selection Who is the Best Respondent? Tomaskovic-Devey, et al. (1994)

25 Boundary Spanners  protect the Technical Core

26 Response Process Who is the Best Respondent? Willimack & Nichols (2010); Bavdaz (2007, 2010) Benefits Costs Routine –Boundary spanners  Respondent selection is under the control of the business Response Process Who is the Best Respondent? Willimack & Nichols (2010); Bavdaz (2007, 2010)

27 Response Process: Respondent Selection Who is the Best Respondent? Willimack & Nichols (2010); Tuttle (2009); Bavdaz (2007, 2010) Distributed data requires multiple respondents and data providers Respondent manages the collection of data from others –Gathers and compiles data  Respondent = Survey Coordinator

28 SURVEY COORDINATOR obtains data from others s/faxes actual survey question(s) including instructions Phone/ ; does not include actual survey questions & instructions SURVEY COORDINATOR completes the questionnaire Respondent Data Provider Understands contents of records Expectations of the Survey Coordinator Lorenc (2006, 2007), Willimack (2007), Bavdaz (2007), Tuttle & Willimack (2012) Knows who to contact!

29 Response Process: Personal Cognitive Steps 1)Record formation & encoding 2)Organizing response tasks a)Respondent selection b)Scheduling c)Prioritizing & motivation 3)Comprehension 4)Retrieval 5)Judgment 6)Communication 7)Releasing the data

Retrieval  Data sources  Personal knowledge from memory  Records  Knowledge of records  Retrieval incorporates –  Cognitive step – retrieval of knowledge of data sources  Access to records  Physical act of retrieving data  Extracting information from files  Consulting multiple sources  Compiling information 30

Judgment  Judging the adequacy of the response  Does the Respondent’s answer retrieved from records or memory match the data being requested?  The Respondent doesn’t have the requested data in records. How does the Respondent estimate that figure?  The Respondent thinks, “My answer is good enough.” 31 Business Survey

32 Retrieval + Judgment: Requested Data ≠ Recorded Data Bavdaz (2010)

33 Mismatch Problems Adapted from Bavdaz (2010)

34 Mismatch Problems Adapted from Bavdaz (2010)

Business Survey Response Process Summary Management Considerations  Survey response tasks = COST with no associated production  Respondent selection  Scheduling, prioritizing & motivation  Data release process Respondent Requirements  Taking the role as survey coordinator  Knowledge of available information  Handling differences between recorded data and requested data 35

36 1 Respondent(s) Authority Capacity Motivation Evidence of Response Burden in Business Surveys Management considerations 4 2 3

37 Perceived Response Burden 9,6 29,3 39,9 15,5 5,8 Very easy Quite easy Neither easy nor difficult Quite difficult Very difficult Sources of PRB E-technical E-Usability Other Layout No of questions Wait for information Terms Calculations Need of help Mismatch Response categories (2) Survey Design  Perceived Response Burden (PRB) Source: Norwegian Structural Statistics, 2010 n = 16,572

(3) Time (Actual) Cost of Response Burden 38 Source: Statistics Canada, Survey of Regulatory Compliance Costs, 2005 & 2008 Average reporting costs, per business, increase was the size of the business increases (based on employment). Average reporting costs, per employee, decrease as the size of the business increases (based on employment).

39 (3) Conditions Associated with the Business  Perceived Response Burden (PRB) PRB- index Size of0 – 9 employees0,11 establishment10 – 19 employees0,13 20 – 49 employees0,06 50 or more employees0,10 DocumentationNone0,15 problems1-0,14 2-0,23 3-0,45 No of10,15 Respondents2 or more-0,01 Value of PRB-Index PRB

40 (4) Respondent Characteristics  Perceived Response Burden PRB- index Available timeVery easy to find0,68 Quite easy to find0,41 Neither easy nor difficult0,06 Quite difficult-0,09 Very difficult-0,37 Perception ofVery useful0,22 usefulnessQuite useful0,30 Neither useful nor useless0,18 Quite useless0,09 Very useless-0,11 ExperienceNo0,04 Yes0,11 Value of PRB-Index PRB

41 How Does Perceived Response Burden Affect Response Quality?

Mismatches & Measurement Error 42

Respondent(s) Authority Capacity Motivation Example: Official Statistics on Research & Development 43 R&D EXPENSES R&D PERFORMED Management considerations

R&D: Requested vs Recorded Data 44 Some Attributes of R&D Stakeholder : PERFORMED R&D PERFORMED by “This” Company Accountant: EXPENSES R&D EXPENSES TypeBasic, Applied, DevelopmentHuh? Tech Support & Maintenance No Yes, if done by same employees doing R&D GeographyYesHuh? Outsourced No (some other company did the R&D for this company) Yes (my company paid for this R&D for the benefit of my company) Reimbursed Yes (this company did the R&D, even though someone else paid for it No (someone else paid my company to do the R&D, so netted out of our costs) TimingSpecific reference period Differs depending on accounting purpose, e.g, for taxes, shareholders, other

Respondent(s) Authority Capacity Motivation Example: Official Statistics on Research & Development 45 R&D EXPENSES Management considerations Survey organisation R&D PERFORMED

Respondent(s) Authority Capacity Motivation 46 Management considerations Official Statistics Implications for Official Statistics Survey organisation Business Records Role of the National Statistical Institute is to: Bridge the gap between Stakeholder requirements & data in business records. Translate Stakeholder requirements into speci- fications for survey design. Use survey design tools & features to aid business Rs, despite variations across company Records/sources. Aid business R (informant) in applying survey specs to data in records, to reduce measurement error. Bear burden for both Stakeholders & Rs. Survey Design Implications for Official Statisticians

Origins and Flows of Response Burden in Business Surveys 3 Management considerations Respondent(s) Authority Capacity Motivation 4 Mismatches – Burden flows back to survey organization, and the data are reviewed and edited.

Respondent(s) Authority Capacity Motivation Origins and Flows of Response Burden in Business Surveys 3 Management considerations Through data collection and processing, burden flows from the Survey Organization back to stakeholders and data users for any (remaining) differences between “final” data set and the underlying concepts associated with the research question.

49 1. Theoretical world: Conceptualization Research Needs: Questions, Issues, Problems, Decisions Empirical world: Physical Observations Attributes Data Concepts Measurements Question(s) Questionnaire Response Specification errors Operationalization errors Measurement & Non-response errors Processing errors Official Statistics in the Context of Total Survey Error Data Mismatch Implications for Official Statistics: “Final” data retain survey errors. Consequences of survey error for the underlying concepts associated with the research question?

Respondent(s) Authority Capacity Motivation Enter BIG DATA 50

Respondent(s) Authority Capacity Motivation 51 Origins and Flows of Response Burden in Business Surveys The “correct” and least burdensome data for a business are data found in records.

Respondent(s) Authority Capacity Motivation 52 Origins and Flows of Response Burden in Business Surveys The “correct” and least burdensome data for a business are data found in records. BURDEN Data Processing

Respondent(s) Authority Capacity Motivation 53 The “correct” and least burdensome data for a business are data found in records. BURDEN Data Processing Implications for Official Statisticians Role of the National Statistical Institute is to: Bridge the gap between Stakeholder requirements & data in business records. Use IT and statistical methodologies to translate data from recorded data to requested data, despite variations across company Records/sources. Bears burden for Stakeholders. Survey Design Edits

Concluding Thoughts The role of Official Statisticians is to bridge the conceptual and definitional gaps between Stakeholder needs and the Respondent’s response process via survey design, data collection and processing. 54

Concluding Remarks What is unique about Business Surveys with respect to Burden?  Survey response is a non-productive cost to the business.  Complex response process  Organizationally determined  R selection controlled by the business, not the survey organization  Mismatch between data in business records and data needed to support Official Statistics 55

Concluding Challenge Implications of the Business Survey Response Process and Burden for Official Statistics:  Data obtained directly from or derived based on Business Records retain Burden, Error, Costs.  Burden flows from the Stakeholder to the Survey Organization to the Respondent, and, ultimately, back to the Stakeholder….  Who has responsibility for associated Errors and Costs??? 56

Thank You! Diane K. Willimack 57