GS/PPAL 6200 3.00 Research Methods and Information Systems February 14 & 15, 2017 Professor Brenda Spotton Visano Office: 130 McLaughlin Voice Mail: (416) 736-2100 ext. 20470 E-mail: spotton@yorku.ca

Agenda Other Data Sources: where to find them, how to assess their usability Articles to be discussed: Bevan, S., Baumgartner, F. R., Johnson, E. W., & McCarthy, J. D. (2013). Understanding selection bias, time-lags and measurement bias in secondary data sources: Putting the Encyclopedia of Associations database in broader context. Social science research, 42(6), 1750-1764. Available at http://www.unc.edu/~fbaum/articles/SSR-2013-BevanEtAl.pdf   Oulasvirta, L. (2014). The reluctance of a developed country to choose International Public Sector Accounting Standards of the IFAC. A critical case study. Critical Perspectives on Accounting, 25(3), 272-285. (Available via York University Libraries)

Other Sources of Data Focus: Data not collected by researcher’s survey Includes: personal documents, mass media outputs, institutional documents, data collected by other researchers or by statistical agencies (i.e., “secondary” data)

Scott’s Criteria Choosing other data sources: Are the data/documents authentic? Is the origin known/confirmed? Are the data/documents credible? Do they represent what they say they represent? Are the data/documents representative? If they are supposed to be emblematic, are they? Is the meaning of the data/documents clear?

Catalogues of Secondary Data Data Catalogues and Resources York University Libraries → Research & Learn → Research Guides → Finding Types of Information: Data & Statistics Finding existing data for a specific concept → Research Data Centre → <odesi> Odesi: A web-based data exploration, extraction and analysis tool. It provides researchers the ability to search for survey questions (variables) across hundreds of datasets, including Statistics Canada master and public-use microdata files. Need help? Contact Walter Giesbrecht, Data librarian at Scott Library datalib@yorku.ca

Integrity of Secondary Data and Appropriateness for Use Understanding the Source Who produced the data/document? What is the public reputation of the producer? What are the primary operating objectives of the producer of the data? Why were the data produced? What was the original purpose for which the data were collected? Who benefits and how?

Integrity of Secondary Data and Appropriateness for Use (cont’d) Understanding the Data Reported What were the operational definitions of key concepts? What was the intended population coverage (who, when)? What was the rule to decide inclusion/exclusion? Is the filter for deciding inclusion/exclusion consistent over time? Who interprets for whom and who represents whom?

Secondary Data Issues Secondary Data Issues Criteria for Evaluating Research Secondary Data Issues Reliability: are the results stable over time and consistent over researcher? Measurement or Construct Validity: does the indicator reflect well the concept we intended to measure? Internal Validity: is there a direct causal relationship between dependent and independent variables? External Validity: Can we generalize the results? Bias from changes in official definitions and policies over time Selection biases due to filtering inclusions/ exclusions and reporting lags Measurement bias from interview effects, question order, respondent interpretations and coding effects

Resolutions Understand sources and structure of potential biases Is there a second set of data of similar coverage with which you could compare data accuracy? Are biases consistent over sampling? How will the biases affect results? Either correct for bias or note the bias in your write up and explain possible effect on results.

Questions for Reviewing Bevan et al. Overview Identify the main sections of the paper What motivates the study? What are the authors’ research questions? What is (are) the authors’ conclusion(s)? Content Details What are their examples of Selection bias? Sampling bias? Measurement bias? What are the sources? What are their resolutions?

Questions for Reviewing Oulasvirta Overview Identify the main sections of the paper What motivates the study? What is the author’s research question? What is the author’s research method? What is (are) the authors’ conclusion(s)? Content Details Is accounting an objective factual exercise or an interpretive exercise? What are the barriers identified to adopting the international accounting standards (in Finland)?

Technical Notes for Oulasvirta Cash basis accounting = report income and expenses as payment received/made Accrual basis accounting = consistent treatment of economic events; accounts for outstanding commitments; more accurate measure of net income For more information: IMF Technical Notes: Implementing Accrual Accounting in the Public Sector by J. Cavanagh, S. Flynn, and D. Moretti (2016) https://www.imf.org/external/pubs/ft/tnm/2016/tnm1606.pdf