1 Randomized Control Trials for Business Solutions: Putting Them to Work for You Jon Zinman Dartmouth College October 6, 2006

2 Research Team Members Dean Karlan, Yale University Sendhil Mullainathan, Harvard University

3 Overview of Talk 1.Examples of product spaces where academics can help CUs improve profits and member satisfaction (using randomized-control trials) –Focus on mortgages, small business lending Where $ is! Biggest opportunity for comparative advantage. Little academic and consultant attention compared to, e.g., saving, financial planning, credit card use 2. Methodology: –How we do it –Operational requirements

4 Example 1: Mortgage Menus Too much choice depresses consumer demand –Psychology studies –Bertrand-Karlan-Mullainathan-Shafir-Zinman (2005) showed this in an experiment with an actual consumer finance company But what’s the “right” menu of choices? –“Right” depends on our partner institutions’ objectives, but for now let’s say: Profitable Sustainable (long-run member satisfaction)

5 Example 1: Mortgage Menus Here is a hard choice: –Choice A: 7-1 ARM, no points at 7.5% vs. –Choice B: Fixed rate, no points, at 8%. However, if you add choice C, Choice C: Fixed rate, 2 points, at 8.05% B looks much more attractive and becomes easy choice

6 Outline of a Mortgage Menus Experiment Experiment: 1) Identify small set of promising menus 2) Run horserace in course of actual ops 3) Analyze results and identify best menu(s)

7 Example 2: Refinancing Evidence suggests that mortgagees procrastinate or don’t pay attention, therefore don’t finance at right time Business implication. Help members refi at right time and: –Profit (new clients, member retention, more frequent refis) –Increase client satisfaction

8 Outline of a Refi Experiment Marketing for attention. Small cues, messages drive clients choices Our prior work shows this. Showing a female photo on direct mail increased consumer loan takeup among dormant male clients by about same amount as a 50% drop in the interest rate Tap social networks (potential comparative advantage for CUs) So test a set of promising cues/messages via direct mail, phone

9 Outline of Another Refi Experiment Experiment with automatic refis. –Help members choose refi triggers in advance comparative advantages here for CUs: trust, personal touch –Offer binding and non-binding commitments –Test takeup rate, profitability, ex-post member satisfaction for a few automatic refi options

10 Example 3: Home equity loan/line utilization Seems low given: –Simultaneous holdings with relatively expensive credit card debt –# of homeowners who have substantial card debt and no HELOC or HEloan

11 Outline of Home Equity Experiment Use psychological principles to drive use of equity lines (instead of more expensive forms of borrowing) Marketing with: –Salient rate comparisons –Cues (picture of family, home improvements) –Make mental accounting work for HELOCs (“put your interest savings back into your house, kids’ education, etc.”)

12 Example 4: Pricing Risk on Small Business Loans How do you know where to draw the line? Some lenders leave profitable deals on table. Example from Karlan-Zinman (2006): –Experiment on high-risk consumer loans –Added step to Lender’s normal underwriting process: applicants who were rejected but close to the bar got a loan 25% or 50% of the time (randomly assigned) –These loans ended up being profitable –Also measurably positive impacts on clients Another method we use for identifying profitability frontier: credit scoring for small businesses

13 What we Mean by “Experiment”: Randomized-Control Trials (RCTs) What they are: –Identical members or prospective clients get different offers –Offers are randomly assigned (“drawn out of a hat”) within a reasonable, predefined range range is guided by business considerations

14 Why RCTs? RCTs are “gold standard” method for identifying causality in social, medical science; biz applications Highly practical/profitable biz proposition –Our work with smaller firms –Larger firms have integrated RCTs into routine marketing and merchandizing operations, face-to-face contact Credit card companies Amazon Food/drug retailers H&R Block

15 RCTs: Step-by-Step 1.Identify product space and key questions 2.Design experiment to answer key questions 3.Implement design into operations Piloting Full launch 4.Monitor implementation for compliance with design 5.Analyze data from experiment –Data sharing: consent and disclosure issues 6. Apply results of experiment

16 RCT Step #1: Identify Area(s) for Experimentation Process: focused brainstorming Players: researchers, senior mgmt. Outcomes: –choice of product spaces –outlines of experimental design like ones I gave earlier

17 RCT Step #2. Design experiment Process: take design outline, meld it to operational realities of firm. Players: research team, senior mgmt, field/area managers, front-line personnel, IT personnel Examples: –Ensure data sharing feasible: consent, disclosures required? –Designing direct mailers –Coordinating experimental direct mailers with routine marketing efforts –Training loan officers to offer a new automatic refi product. Say some clients will get direct mail or face-to-face offer of this new product, other won’t. Then loan officers need to know: Content of new product(s) Presentation Who gets the offer? (compliance with randomization) –Usually involves a software add-on to MIS that loan officer would reference for each client, and automatically prompt a new product offer, or regular product offer Staffing requirements: –Research team: we bring a project manager –CU: may want dedicated project manager (depends on scale)

18 RCT Step #3: Implement design into operations 3a. Pilot experiment Small scale Work out kinks: –Design tweaks –Operational tweaks (compliance, clunkiness) –Fix/eliminate backfires (member complaints, staff burdens) 3b. Full launch

19 RCT Step #4: Monitor Compliance with Design Audits/site visits for any face-to-face components Statistical tracking

20 RCT Step #5: Analyze Data from Experiment Researchers generate a report –Upfront issue: make sure can hand researchers the requisite data needed for analysis What consent, disclosures are needed, if any?

21 Step #6: Apply Results of Experiment What have we learned, and what should the CU do next? –Discussions between researchers, mgmt –Lead to recommendations from researchers about how to apply the results: Specific product, pricing, marketing decisions Strategy and positioning Opportunities for additional experiments

22 Take-Aways and Next Steps RCTs work: standard- and best-practice for answering key strategic questions Research team has some ideas for high- impact projects We are looking for CU partners So let me know if you have interest in RCTs & would like to explore opportunities