1. Response Uplift Modelling
Optimising prompt amounts through behavioural analysis
07/11/2016
Andrew Lockett, Wood for Trees
Matt Champion, British Red Cross

2. Introductions
Andrew Lockett Consultant Analyst, Wood for Trees
Matt Champion BI Manager British Red Cross

3. What we’ll cover Background Stage 1 - Preliminary Analysis
Stage 2 - Initial Testing
Stage 3 - Scaling the Tests
Stage 4 - Response Uplift Modelling (RUM)
Stage 5 - Implementation
Questions

4. Background

5. Background
For many years the British Red Cross prompted our core file of cash supporters at the same amounts in each warm appeal
These prompt amounts were wide-ranging and linked to specific ‘shopping list’ items
We were aware that the prompt values included in our appeals could be influencing the number of donations we received and their value, but had not tested this in a structured and robust way
In collaboration with Wood for Trees we undertook a comprehensive rethink of our prompting strategy
The aim was to develop an approach that would increase the income received from warm cash appeals but without reducing the overall volume of responders
This has been achieved through using supporter information to tailor prompt amounts to individuals, recognising that not everyone responds in the same way to the same asks - one size does not fit all!

6. Initial Bivariate Analysis Multivariate Model - RUM
Timeline
Up to 2013
2014
2015
2016-
One-size
fits all strategy
Initial Bivariate Analysis
Small-scale tests
Scaling the Tests
Multivariate Model - RUM
Implementation
Everyone receives same set of prompts
Split supporters into groups based on last gift bands
Tests carried out on small volumes to manage risk
Tests carried out on large volumes to provide clear results
Built statistical models to tailor supporter prompting further
Roll-out with findings from test phases 1 2 3 4 5 6
£5, £18, £25, £50, £150
£5, £18, £25, £50, £150
T1: £5, £8, £10, £15
T1: £5, £8, £10, £15
T2: £8, £10, £15, £20
T2: £8, £10, £15, £20
T3: £10, £15, £20, £25
T3: £10, £15, £20, £25
£5, £18, £25, £50, £150
<= £5
<= £25
£5, £18, £25, £50, £150
<= £10
£5, £18, £25, £50, £150
Roll-Out
One-Size Fits All
T1: £5, £10, £15, £20
T1: £5, £10, £15, £20
Last Donation
Last Donation
T2: £10, £15, £20, £25
T2: £10, £15, £20, £25
Recency
T3: £15, £20, £25, £30
T3: £15, £20, £25, £30
Frequency
£5, £18, £25, £50, £150
£5, £18, £25, £50, £150
Value
T1: £15, £20, £25, £30
T1: £15, £20, £25, £30
Demographics
T2: £20, £25, £30, £35
T2: £20, £25, £30, £35
Recruitment
T3: £25, £30, £35, £40
T3: £25, £30, £35, £40

7. Preliminary Analysis
- reviewing past tests

8. Preliminary Analysis (2014)
Prompt amounts through 2013/2014 were fixed at £5/£18/£25/£50/£150
Average Donation was £12 – but this does not tell us very much
Looking at the Response Value distribution is much more informative
Greatest number of responses at £5 lowest prompt.
Next most common was £10 – despite this not being a prompted value!
Similar number gave at £20 compared to the prompted £18.
Almost none giving at the highest £50 & £150 prompted amounts.

9. Preliminary Analysis (2014)
Prompt testing had been carried out in the past, however the results had not been statistically significant.
Groups
Prompts
RR%
Avg Gift (inc GA)
Control
£5, £18, £25, £50, £150
10.3%
£11.48
Test fixed prompt
£10
10.4%
£11.46
Results look same!
Although the overall response rate and average gift did not change, when we reviewed the results more closely though we could see that supporter behaviour was changing:
More at £10
Some supporters in the test group who gave at £10 would have responded at a higher level had they received the control prompts and some would have responded at a lower level – clear need to personalise prompts

10. Preliminary Analysis (2014)
Appeal Response Value
To start personalising prompts we first needed to understand the key predictors of response value.
Analysis into historic giving patterns showed that a supporter’s Last Gift is a strong predictor of their Next Gift.
This helps set appropriate prompt amounts for initial testing.
Increased vs last donation
Last Donation Value
Reduced vs last donation
Greatest numbers giving at same level as last donation

11. - finding the right prompt levels
Initial Testing
- finding the right prompt levels

12. Group 1: Last Gift <= £5
Initial Testing (2014)
Preliminary bivariate analysis identified last gift as the strongest predictor of a supporter’s next gift
We used this to split the test prompts for our core cash supporters (last gift <= £25) into 3 groups:
3 sets of alternative ‘test’ prompt asks were devised for each of these groups
The initial analysis had suggested we would require large test volumes to see statistically significant results
However, recognising the inherent risk with any new approach, we initially tested on limited volumes
Group 1: Last Gift <= £5
Group 2: Last Gift £ £10
Group 3: Last Gift £ £25
C: £5, £18, £25, £50, £150, My Choice
T1: £5, £8, £10, £15, My Choice
T1: £5, £10, £15, £20, My Choice
T1: £15, £20, £25, £30
T2: £8, £10, £15, £20, My Choice
T2: £10, £15, £20, £25, My Choice
T2: £20, £25, £30, £35
T3: £10, £15, £20, £25, My Choice
T3: £15, £20, £25, £30, My Choice
T3: £25, £30, £35, £40

13. Initial Testing (2014)
Here is an example of one of the outputs from the initial small-scale testing:
Last Gift £5.01-£10
Unprompted £5
Unprompted £10
P1 £15
P2 £20+
Unprompted £5
P1 £10
P2 £15
P3 £20+
P1 £5
P2 £10
P3 £15
P1 £5
Unprompted £10

14. Initial Testing (2014)
Overall the test results showed small changes in behaviour
Generally the impact on income was positive, largely through narrowing the jump from £5 -> £18:
We investigated the poor performance of the bottom group:
For supporters recruited through a Cold Mailing and with a last gift between £10.01-£19.99, response rates were considerably lower in the test groups than control and we decided this group would be better off receiving a lower set of prompts
Last Gift Value
Control (Inc Per ‘000)
Tests (Inc Per ‘000)
Diff (Inc Per ‘000)
<=£5
£496
£516
+£20
£5.01-£10
£600
£624
+£24
£10.01-£25
£1,371
£1,289
-£82
+4%
+4%
-6%
Door Drop
£10.01-£19.99
£20-£25
Control
8.2%
7.1% T1
8.8%
6.4% T2
8.0% T3
7.7%
6.1%
Cold Mailing
£10.01-£19.99
£20-£25
Control
8.3%
6.6% T1
6.1%
6.9% T2
5.9%
6.3% T3
6.0%
Worse than control

15. - checking significance & getting more data
Scaling the Tests
- checking significance & getting more data

16. Scaling The Tests (2015)
The incremental income and responses in the tables below have been calculated as if all of the available volume had been sent the control or test packs, in order to show the anticipated impact on roll-out
Initially we opted to implement the ‘T1’ lower prompts, which deliver a significant boost to income and generate incremental responses
Group
Control or Test
Response Rate
AG (£)
Diff To Control
<= £5
Control
10.56%
£6.90 T1
10.72%
£6.95
+£2,145 T2
9.54%
£7.88
+£3,012 T3
10.17%
£8.03
+£11,122
£ £10 /
£ £19.99
8.23%
£12.69
8.77%
£12.29
+£3,987
8.19%
£13.31
+£5,513
7.56%
£14.35
+£4,872
£ £25
£20 - £25
7.18%
£23.06
7.48%
£23.83
+£12,533
6.93%
£24.89
+£6,877
6.58%
£26.88
+£11,189
Tests
Incremental
Gross Inc
Responses T1
+£18,664
+1,154 T2
+£15,402
-1,566 T3
+£27,183
-1,886
T3,T1,T1
+£27,642
+462

17. Scaling The Tests (2015)
Part of the reason we opted to roll-out with the lower ‘T1’ prompts is that until we had developed the RUM models there was no mechanism to prevent supporters being continually prompted higher
For example, successfully prompting a supporter in the £0- £5 group up to a £10 gift would move them into the £ £10 group for the next appeal. Assuming a role out on test 3 to each group, this would then move them to a starting prompt of £15, at which point they may stop responding and eventually lapse if the prompt level they received wasn’t adjusted.
A modelled approach to prompting could mitigate against this by taking the recency of the donors last gift into account when deciding at what level to prompt them.
£25
Group 3
T3: £25, £30, £35, £40
£20
£15
Group 2
T3: £15, £20, £25, £30
Group 3
T1: £15, £20, £25, £30
£10
Group 1
T3: £10, £15, £20, £25
Group 2
T1: £5, £10, £15, £20 £5
Group 1
T1: £5, £8, £10, £15

18. Response Uplift Modelling
- going beyond last donation

19. Response Uplift Modelling - Concept (2016)
Personalising prompts based on Last Donation was a good start & generated strong gains on the old ‘one size fits all’ approach
… but can we do go further? By identifying some ‘sub groups’ of supporters who give more if prompted higher (without a negative impact on response rate).
… and conversely finding groups of supporters who would be put off by higher prompts and where lowering prompts makes sense.

20. Response Uplift Modelling - Concept (2016)
Similar to the concept of ‘Price Sensitivity’ – not always about asking for more.
A higher Prompt Amount may boost donation value, but likely at the cost of some response.
Some groups of supporters will have a ‘higher sensitivity’ ie. an increase in prompt would seriously affect response rate.
Group 1 – high sensitivity
Group 2 – low sensitivity
Of course the reality is not so simple.
There is no clear ‘price’ for a donation - supporters will give what they feel comfortable with.
But we can use the concept to find sub-groups who respond favourably to a higher ask
…and importantly those that would be put off by a higher ask
High Prompt = Higher Donation Average
Little impact on response
Increased prompt
Big reduction in response
Low Prompt
Low Response
Response Rate for Given Prompt Amount
High Response

21. Response Uplift Modelling - Method (2016)
Test data:
With similar supporters prompted at differing amounts. From 2015 tests we had 3 groups based on last donation amount each being prompted at 3 different levels (T1/T2/T3)
Predictive Factors:
We created a data set with ~100 potential predictive factors. Covering recruitment method, demographics, past giving behaviour, other relationships.
Analysis method:
looks for factors that are predictive of differences in ‘Income per Supporter Mailed’ across the 3 different prompt levels.
But also consider the Response Rate & Average Value (to ensure Response Rate does not suffer).
Information Value statistic is used to pull out the most significant factors.
Top factors reviewed to see if make intuitive sense and used to create model.
Decision Tree:
End result is a decision tree allocating supporters to an end node with a suggested prompt level.

22. Response Uplift Modelling - Outputs (2016)
Example of the model that was built for the £0-£5 last gift group. The tree has a two way split at each branch, with up to 6 levels and 13 end nodes. Each end node has a suggested prompt level.
£0-5 Last Donation
= Node 9
0.87
= Node 10
0.89
= Node 11
1.09
= Node 15
1.70
= Node 16
2.19
= Node 17
= Node 20
1.29
= Node 30
1.23
= Node 35
1.20
= Node 36
0.98
= Node 37
1.24
= Node 38
0.63
= Node 40
0.85
Splitting Rule
Node with T1 prompt
Node with T3 prompt
Node 16: Benefits from £10 prompt. High value, given recently, highest don <£10 but average >£7.50.
Node 38: Benefits from £5 prompt. received several mailings, low average, >10 months since last given.

23. Implementation

24. Implementation - Testing (2016)
In 2016 we carried out testing across 2 appeals to assess the impact of the RUM prompting approach against the lower ‘T1’ prompts we had rolled out with already
Overall the testing showed modest incremental gains on top of the big gains already made from moving away from the old fixed prompts:
Control or Test
RR %
Avg Resp Amt
Inc Per Mailed Supp
Incremental Value
Control = T1
8.4%
£11.07
£0.926
Test = RUM
8.1%
£11.69
£0.952
£8,883
+3%

25. Implementation - Results (2016)
Following the large-scale testing in 2015 we rolled out with the ‘T1’ prompts for each last gift band as an interim strategy while the models were still being built.
We then tested the more sophisticated RUM approach vs the ‘T1’ prompts
On top of the £18.6k additional income we can attribute to using a supporter’s last donation to determine the prompts they receive, a further £9.2k is generated by tailoring supporter prompts further using RUM
This equates to around £27.7k in incremental income per appeal plus an extra 565 responses
One-Size Fits All
Last Donation
RUM
<= £5
<= £19.99
<= £25
+ £18,664
+1,154 responses
Per Appeal
£18,664 + £9,153 = + £27,817
1,154 – 589 = +565 responses
Per Appeal

26. Implementation - Practicalities (2016)
The models are now being tested in a variety of campaigns to see how they perform over a period of time and at different times of year
The results may lead to some further testing to fine tune the strategy, but once enough evidence has been collated the new prompts can be rolled out
How much additional resource does it take to setup tests for each appeal?:
Step 1: Create model scoring script in SQL
Step 2: Setup diagrams in SAS as normal and export campaign data into SQL
Step 3: Run model script to generate model node and prompt amounts for low-value supporters
1-2 extra days resource needed per appeal
Step 4: Create a table storing the model node and prompt amounts for future analysis
Step 5: Test and QA that prompts have been assigned accurately
Step 6: Generate export file for mailing house, including prompts to be lasered

27. Summary

28. Summary
This testing programme has shown that by reacting to supporter behaviour and making full and innovative use of the data we can provide a more personalised experience with our prompting
By taking a structured approach we have delivered significant incremental improvements while managing risk and evidencing the value of changes to the prompting strategy in a robust way
Overall the new prompt strategy has met our success criteria by generating an incremental £27.7k in income and 565 in incremental responses per appeal
Our plan is to continue testing the RUM strategy over a variety of appeals, tweaking the models if necessary before final roll-out
As a next step we plan to expand the project scope to include supporters with a higher value last gift. We believe this could generate further incremental value as well as ensuring supporters receiving consistent prompting

29. Key Takeaways
Testing: Getting prompts right requires commitment to a series of tests over time.
Average Donation: Don’t simply look at the average. This misses a wealth of detailed behaviour. Instead look at response value distribution in relation to prompted amounts.
Supporters give what they want to. Prompting will only have a minor influence – hence we are looking for marginal gains from tailoring prompts.
Personalise prompts based on previous donations. Can use Last, Average or Maximum donation but be aware of pro/cons of each.
Test different prompt levels to find optimum trade off between response & value. Remember prompting higher is not always best!
More sophisticated prompts (eg. RUM) can identify supporters that do better with a higher or lower prompt than their last donation would suggest. This can bring further gains.

30. Any Questions?