Field Experiments in Economics David Reiley, University of Arizona and Yahoo! Research

Is Economics Like Astronomy? “If I hadn’t become an astronomer, I would have wanted to be an economist.” No controlled experiments like in other “hard sciences.” Herschel’s Garden: billions of different stars to observe, at different stages of development

How do we answer scientific questions? Astronomy questions: How do we know what the Sun is made of? Can we predict the fate of a star like the Sun? Economics questions: How do we know what price a monopolist will set? Can we predict what will happen when a monopolist’s unit cost increases?

How do economists learn from empirical observation? My claim: almost everything we learn from data can be considered to be an experiment of some kind. Laboratory experiments Natural experiments Field experiments Are there any non-experimental modes of observation in economics?

Examples of laboratory experiments Double-auction market institution yields prices predicted by intersection of supply and demand. Ultimatum game yields behavior different than predicted by subgame perfection. First-price sealed-bid auction yields higher revenues than English auction, with independent private values drawn from a uniform distribution. Increasing the number of bidders in a sealed-bid auction causes the average bid to increase.

Examples of natural experiments How does labor demand vary with wages? (Use state variation in minimum-wage laws). Does legalization of abortion lead to less crime 15 years later? (Use state-by-state variation.) How does time in line (temptation) affect purchase of grocery impulse items? Do orchestra audition committees discriminate against women? (Use variation in whether the orchestra uses a screen to audition blindly.)

Examples of field experiments Does one auction yield higher revenues than another? Does one fundraising mechanism yield higher revenues than another? Do employers discriminate against job applicants with African-American names? How effective are computer techniques at promoting reading in American public schools? How does the demand for a mail-order retail good change as we change the price from $44 to $49 to $54?

How do field experiments compare to lab experiments? Benefits: Real goods instead of induced values People making decisions very similar to those they would normally make. Better dialogue with mainstream empirical economists. Costs: Can’t control as much (demand & supply curves) Can’t even observe as much (welfare & efficiency)

How do field experiments compare to natural experiments/empirical economics? Benefits: Generate exactly the variation you want in your data, when the real world might not have cooperated. Random assignment eliminates sample-selection and omitted-variable bias. Costs: Field experiments might be too expensive. Field experiments might be ethically infeasible.

Problems with Traditional Empirical Economics Usually can’t test theory (with rare exceptions). Instead, assumes theory to be true, and makes measurements based on these assumptions: What’s the elasticity of labor supply? What does this imply for the optimal income tax structure? What is the discount rate? What does this imply for optimal design of Social Security and 401(k) rules? What’s the distribution of values in a timber auction? How do we determine the optimal minimum bid? How much did social welfare increase with the introduction of the minivan?

Dangers of Traditional Experimental Economics By defining “experimental economics” as a self-contained field, we can lose touch with the rest of the profession. No field of “experimental physics,” instead have both experimental & theoretical “nuclear physics” By taking theoretical models too literally, we can lose touch with the real world: Does a real threshold-public-good fundraiser burn the money if it fails to reach the threshold? Do we want to know whether one auction format raises more revenue under laboratory conditions, or under real-world conditions?