1. Amy Finkelstein January 12, 2006
Most projects fail… and other things I’ve learned from (trying to do) empirical work
Amy Finkelstein
January 12, 2006

2. Purpose Graduate school is well-structured to teach you:
Economics
i.e. What are the interesting and important questions?
Technical skills
i.e. How to answer them
But what about the process of doing (or trying to do) research?
This is a semi-structured list of some tips I wish someone had told me….

3. Disclaimer This presentation is not yet rated.
The opinions expressed here are not officially endorsed by the MIT faculty, the American Economics Association, or anyone else
Viewer discretion is advised.
Applied micro

4. The bottom line 1. It is hard to do good research
for everyone
2. Research is not a solo process
Talk about your ideas with people early and often
3. Most “projects” fail
The key is to triage efficiently
This is not easy. They call it “re-search”. No one finds it easy.
By projects I mean ideas – the key is to keep them from getting too far along before you triage them.

5. Two key steps early in the process
Coming up with ideas
2. Triaging your ideas

6. Step 1: Coming up with ideas
All (good) research starts with a question that is interesting
Can you explain to others why it’s interesting and exciting?
Your peers and professors
Non economists
Your family
Non-economist friends (if any)
Are you interested in this question?
If you are not interested and excited in your project, how can you possibly expect anyone else to be?
And you will certainly not enjoying working on it for many years!

7. All topics need good motivations
Papers should be motivated by broad, interesting, and/ or important issues, although the specific problem you end of working on may be narrow and technical
This will keep you (and others) motivated and interested
Write down the motivation so you remember it
and look back at it from time to time
Keep adding to it
Example 1: Development
Broad question: How to reduce corruption in developing countries?
Specific project: How do we get good measures of corruption?
Example 2: Public Finance
Broad question: should the government stay out of insurance markets?
Narrow, technical paper: how do we design empirical tests of adverse selection?
If what you are doing can no longer be related to a big picture, it suggests you have gotten off track…

8. There’s a problem when…
Your answer to “what are you working on” starts with:
“My paper is an extension of Famous Economist (2003b)”
“I found a really cool source of identification / natural experiment”
“My project is really hard because it involved
The use of a really cool new econometric technique
Really messy data
Really complicated programming
A lot of time
Second bullet: that doesn’t mean it doesn’t involve all these things (and in fact sometimes these are correlated with being a good paper) but come on – that’s not why you’re interested in the question (I hope!)
Good source of identification – the everest phenomenon.

9. Always be on the frontier
Avoid corner solutions
Interest / importance of question
The more interesting / novel the question the less you have to nail it completely. If you are the 10th paper on X, you’d better have the cleanest approach by far.
People / fields have different tastes for where to be
Corner solutions never good
How convincingly can you answer the question?

10. How to come up with ideas??
Chicago adage
“vote early, vote often”
Think about ideas early, often and always
Write all your ideas down!
Especially why you care
Sources of ideas:
Classes – what are the important unanswered questions?
Seminars – what does the seminar make me think about?
In general do not go to the literature for ideas
Broad survey articles can stimulate ideas
JEL, JEP, Handbook Chapters
Read the newspaper with an eye towards economic questions
Look at the real world, not just the economics literature
Read non-economics non-fiction
Biography, history
Talk to people – economists and non economists
Ideas come at random times
Be sure to write them down
Keep at it
Nothing is worse than staring at a blank wall with no thoughts on what to do….
Seminar example: Bertrand and Mullainthan on CEOs rewarded for luck (corporate finance) Wolfers on are governors rewarded for luck (pol economy)
Baicker and Staiger: how does hopspital getting $$ help recipients– falsification test gone awry or really interesting alternative theory – spillovers and why we would care about them.
Someone else’s paper can be your instruemnt – David Matsa and union spillovers (Holmes)
Newspaper examples:
If you only look at the literature you’ll never find a new question.
If drug prices are low, firms won’t innovate. (is this true? Turns out no info. And why does it matter? Static vs dynamic policy effects. E.g. govt regulating low prices…)
2. Friedman article two days ago: high oil prices support autocratic regimes? (is that true? What determines regime shifts? Are natural resources good for a country in the long run?).
I often excuse myself and leave myself a voice mail message if during dinner something hits me.
Keep at it – there’s no other way.

11. 2. How to triage your ideas
Most projects fail
Being able to figure out the lethal problem quickly is the key to having time to work on your (ultimately successful) projects
The first line of defense – talk to people
Especially your peers
Is this interesting / sensible? What are your reactions / thoughts / suggestions?
Do not self-censor your thoughts
Do not wait long before talking to them
You should be constantly tossing out your latest goofy thought
Don’t wait to talk to faculty either
We expect you to bring us many non-workable ideas
We are on your side!
Gruber line: 10 bad ideas a day.
Don’t work on something for 3 weeks before running it by a faculty member.

12. Important triage steps
Do I have the facts right?
If the effect is there would it be detectable?
Would it be interesting if I found nothing?
I almost wrote my dissertation on pet health insurance in Sweden….
Talk to your friends
Need example of first one… (again, often talking to people). Often want to think about a program a certain way bc that’s what you want to study. But is that really what it did?
Second one. Three exmapls
Swamped by other events. one stop shopping for firm registration (and indo financial crisis) – even with DD just going to be really hard.
swamped by noise
Incidence of paid vacation – MLK in Arizona. Arizona tiny. At most 1/200 (0.5%) of income – which is so noisy.
3. Sample size. Mandated employer hi – hawaiin workers in 1974 NHIS data – 5 peopke
Third bullet – really hard but ideal.
Fourth bullet – projects take a life of their own so best to pick something you are innately interested in
Another example: tech change became about GE effect of hi on health spenidng.

13. Do not immediately give up on an idea because
You found a paper on econ-lit that seems to be on a similar topic
The first time you tried to explain it to someone they weren’t that interested
It’s not obvious how to go about answering the question
You don’t think there’s any data to answer it
Econ lit – many papers sound similar but are different
If you are excited about it rethink the motivation and try again
If it were obvious and interesting it would have been done
Again, if it were obvious where the data were, more likely to have been done; besides that’s what faculty are for

14. Data triage What are the ideal data I would need for this project?
What kinds of data exist?
Look on ICPSR, the NBER data website, google
Talk to people!
Is there anything I can do quickly to get a rough sense of whether the effect would be there?
Maybe not with the ideal data
Step #1 with the data: PLOT YOUR DATA
And then show the plot to your friends
For what kind of papers exist – ask faculty.
Last bullet: Regressions can be misleading (outliers; misspecifications etc). Can I see anything in the figure. Gives me a sense of how to run regression and a wawrning if I find something in the regression that I don’t see in the data – or vice cerse. (if looking for a RD at age 55, plot first).

15. What if the data just don’t exist?
Collect your own data!!!
An enormous number of the exciting / interesting / important empirical papers involve new data collection
If it’s interesting and easy to do, more likely it’s been done already
It’s barriers to entry that make monopolists
I can’t recommend this strongly enough.
Certainly my favorite projects of my own
-- vaccines – “sure – what data are you going to use”?
Substitute brawn for brains!!
(cambodian data entry)
Friends and families – brother consulting (over rosh hashannah).
Internet resources creatively – anybirthday.com; campaign contribution organization put on website campaign contributions by firms (common cause) – had to query one by one not a “data set” per se (seema)
Cold calling – people like to help students!!
companies, toll authorities etc.

16. Where to look for your own data?
Archival work
People printed out data before computers
Your friends and families may unwittingly be sitting on great data
Think creatively about what they might have access to
Use internet resources creatively
Design a survey
Don’t be afraid to make cold calls to organizations that might have data
What do you have to lose (aside from your time…)?
I can’t recommend this strongly enough.
Certainly my favorite projects of my own
-- vaccines – “sure – what data are you going to use”?
Substitute brawn for brains!!
(cambodian data entry)
Friends and families – brother consulting (over rosh hashannah).
Internet resources creatively – anybirthday.com; campaign contribution organization put on website campaign contributions by firms (common cause) – had to query one by one not a “data set” per se (seema)
Cold calling – people like to help students!!
companies, toll authorities etc.

17. The bottom line re-visited.
It is hard to do good research
for everyone
and that means everyone
Research is not a solo process
Form weekly working groups with your friends to talk about your latest thoughts
Force everyone to talk about at least one idea, no matter how lame they think it is
Most projects fail
The key is to triage efficiently…..
…. and to keep at it!!
Serious point: I used to think sometimes when I had no ideas / was staring at a blank wallthat this was prima facie evidence that I shouldn’t be doing this. If I don’t have anything to work on why do I want to do research? PEOPLE get depressed & unahppy. “not going to happen to me”.
Some of you will know what I’m talking about. Some of you maybe thinking “Everyone but me” but don’t be so sure…