1 ABOLaureen Maines 2008 Choosing Participants for Financial Accounting Experiments October 10, 2008 ABO Doctoral Consortium

222 ABOLaureen Maines 2008 Objectives Identify participants in financial accounting experiments Identify characteristics of participants Discuss factors to consider when choosing participants Provide advice for finding participants

333 ABOLaureen Maines 2008 Financial Reporting Process Standard Setters PreparersFinancial Reports AnalystsStock Market Reaction AuditorsInvestors RegulatorsCreditors

444 ABOLaureen Maines 2008 Research Process Research Question (Theory) Experimental Design and Task Participants (Theory)

555 ABOLaureen Maines 2008 Variables Influencing Judgments and Decisions Decision-Maker Characteristics Task Characteristics Environmental Characteristics (Sarah Bonner, Judgment and Decision Making in Accounting, Prentice-Hall, 2007)

666 ABOLaureen Maines 2008 Participant (Decision-Maker) Characteristics External / Easily observable or measured Gender, Age, Professional Certifications, Work Experience Internal / Need to be measured via test or possibly manipulated Abilities Knowledge Preferences

777 ABOLaureen Maines 2008 Participant Characteristics - Abilities Abilities Verbal abilities – reading comprehension Reasoning abilities - interpret data, reason from analogy, reason analytically Spatial abilities – identify and comprehend patterns Measurement Typically through standardized college admission tests – SAT and GMAT Manipulation Difficult in adults

888 ABOLaureen Maines 2008 Participant Characteristics - Knowledge Knowledge Semantic General world knowledge, domain-specific knowledge, sub-specialty knowledge Episodic knowledge/beliefs based on personal experience Incentives, reactions of other parties, empirical regularities Measurement Knowledge-specific tests (e.g., CPA exam questions) Post-experimental questions about beliefs Manipulation Possible, but need to be careful

999 ABOLaureen Maines 2008 Participant Characteristics - Preferences Preferences (Utility) Money (Risk Aversion) Effort Honesty, ethical behavior, welfare of others Measurement Validated tests from economics, psychology, sociology Manipulation Induced preferences for money (Berg et al. QJE 1986)

10 ABOLaureen Maines 2008 Role of Participant Characteristics Independent Variable Decision-maker characteristic directly affects dependent variable McDaniel et al. (TAR 2002) Moderating Variable Decision-maker characteristic changes the relation between a task or environmental variable and the dependent variable Elliott (TAR 2006) Neither of above – just choosing participants for your task Think in terms of a moderating role

11 ABOLaureen Maines 2008 Participant Characteristics as Independent Variable (McDaniel et al.) XY Experience (Accounting) (Business) Judgment (Reporting Issue Identified – recurring vs. nonrecurring) Causal relation

12 ABOLaureen Maines 2008 Participant Characteristics as Independent Variable Not typical in financial accounting research Typically requires measuring participant characteristics – external/internal May be hard to avoid confounding External characteristics (e.g., years of experience) typically represent a number of internal characteristics (e.g., knowledge, ability) For example, knowledge and ability are rarely independent (Link 2 in Libby and Luft AOS 1993) A problem if you want to establish causality for internal characteristics (need manipulation + random assignment, not measurement)

13 ABOLaureen Maines 2008 Participant Characteristics as Moderating Variables (Elliott 2006) Y U GAAP – ProForma Reconciliation (Yes/No) Decision (Investment Decision) Knowledge X

14 ABOLaureen Maines 2008 Participant Characteristics as Moderating Variables Use this approach even if you arent planning on a moderating variable approach Does theory suggest that judgments and decisions will depend on characteristics (abilities, knowledge, preferences) of participants? Will the JDM process or outcomes change? Does the level of relevant characteristics vary among relevant decision-makers or participant groups?

15 ABOLaureen Maines 2008 Participant Characteristics as Moderating Variables If answers to prior questions are yes, how do I proceed? Embed theory in paper and test multiple groups that should differ based on theory – true moderating study Examine only one group What group is most important in terms of the research question? External validity limitations

16 ABOLaureen Maines 2008 Participant Characteristics – Professionals versus Students Some generalizations Basic reasoning abilities typically dont change much after adulthood College students likely have similar abilities to most adults Be careful – there can be cross- sectional variation - top people in any field have survived tournaments and are likely to have higher ability

17 ABOLaureen Maines 2008 Participant Characteristics – Professionals versus Students Some generalizations Knowledge is a big factor in accounting experiments Semantic knowledge – world knowledge and sub-specialty knowledge increases with experience Episodic Knowledge / Beliefs also change with experience – personal experiences with incentives, behavior of other people

18 ABOLaureen Maines 2008 Participant Characteristics – Professionals versus Students Some generalizations Preferences Reaction to economic incentives is basic human reaction Experimental research based on economic theory typically can use abstract tasks and college students

19 ABOLaureen Maines 2008 Issues – Working with Professional Participants Negative externalities of using professionals when not needed: limited resource (Libby, Bloomfield & Nelson AOS 2002) Places high demands on researcher – good research question requiring professionals, good design, good instrument Personal experiences of professionals lead to expectations that will create problems if not met Need to run materials by professionals and pilot test

20 ABOLaureen Maines 2008 Issues – Working with Professional Participants Selection and non-response bias may be stronger with professionals than with students (Peecher and Solomon 2001) End up using one firm – how general are your results? Professionals are busy people which leads to non- response Need to get them interested and committed Work around their schedule

21 ABOLaureen Maines 2008 Source of Participants Preparers Alumni directory – controllers, CFOs Financial Executive Institute Institute of Management Accountants

22 ABOLaureen Maines 2008 Source of Participants Investors - Nonprofessional College Students Think carefully about research question and task Insights on using MBA students as proxies for investors – Elliott et al. (TAR 2007) National Association of Investors Corporation – 400,000 members See Hodge (AH 2003)

23 ABOLaureen Maines 2008 Source of Participants Financial Analysts (sell and buy side) Alumni Directory Individual investment banks, financial management firms, pensions funds, etc. Association for Investment Management and Research Nelsons Directory of Investment Managers

24 ABOLaureen Maines 2008 Source of Participants Standard setters and regulators Good luck!

25 ABOLaureen Maines 2008 Conclusions Appropriate choice of participants requires a good research question, a good design and task, and careful consideration of how personal characteristics affect JDM How does theory suggest different levels of a characteristic would change JDM?

26 ABOLaureen Maines 2008 Conclusions Be enthusiastic and able to explain why people should care about your research Practice describing your research to your accountant and non-accountant friends BE PERSISTENT !!! (but not annoying!)

27 ABOLaureen Maines 2008 References Berg, J.E., Daley, L.A., Dickhaut, J.W., and J. OBrien Controlling preferences for lotteries on units of experimental exchange. Quarterly Journal of Economics 101 (May): Elliott, W. B Are investors influenced by pro forma emphasis and reconciliations in earnings announcements? The Accounting Review 81 (January): Elliott, W. B., F. Hodge, J. Kennedy, and M. Pronk Are MBA students a good proxy for nonprofessional investors? The Accounting Review (January): Hodge, F. D Investors perceptions of earnings quality, auditor independence, and the usefulness of audited financial statements. Accounting Horizons 17(Supplement): Libby, R., Bloomfield, R., and M. Nelson Experimental research in financial accounting. Accounting, Organizations and Society 27 (November): Libby, R. and J. Luft Determinants of judgment performance in accounting settings: Ability, knowledge, motivation, and environment. Accounting Organizations and Society 18 (July): McDaniel, L.S., Martin, R. D., and Maines, L.A. (2002). Evaluating financial reporting quality: The effects of financial expertise vs. financial literacy. The Accounting Review 77 (Supplement): Peecher, M. and I. Solomon. (2001). Theory and experimentation in studies of audit judgments and decisions: Avoiding common research traps. International Journal of Auditing (November):