1. Hierarchical Recommender System for Improving NPS.
CCI, UNC-Charlotte
Hierarchical Recommender System for Improving NPS.
presented by
Zbigniew W. Ras
Research sponsored by:

3. CUSTOMERS CONSULTING COMPANY in Charlotte What we have to do Services
Build
Recommender System
for each client
(34 clients) helping to
increase its revenue
Client 1
Client 2
Client 3
Client 4
shops
shops
shops
Build Personalized
Recommender System
for each shop helping to
increase its revenue
Services
(heavy equipment repair)
Parts
CUSTOMERS

4. NPS of a company is correlated with its revenue growth
Net Promoter Score (NPS) –
today’s standard for measuring customer loyalty
Promoter - {9,10}
Passive – {8,7}
Detractor – {1,2,3…,6}

5. NPS rating for all clients

6. NPS rating for all shops

7. What is our goal?
Build recommender system giving possibly the simplest
and the best advice to every client/shop on the changes
they need to make in order to improve their NPS ratings.

9. Personal Benchmarks Initial Dataset – provided by Consulting Company
About 100,000 records representing answers to a questionnaire collected from over 35,000 randomly chosen customers in
Customers use services provided by 34 clients
A questionnaire consists of:
Information about the customer
Customer’s name, location, phone number…
Information about the service
Client’s name, invoice amount, service type…
Information on customers’ feeling about the service
was the job completed correctly
are you satisfied with the job
likelihood to refer to friends …
Personal
Benchmarks

10. Randomly chosen customers are asked to complete Questionnaire –
BenchmarkAllOverallSatisfaction 35
BenchmarkAllLikelihoodtobeRepeatCustomer 34
BenchmarkAllDealerCommunication 33
BenchmarkServiceRepairCompletedCorrectly 32
BenchmarkReferralBehavior 31
BenchmarkServiceFinalInvoiceMatchedExpectations
BenchmarkEaseofContact 30
BenchmarkAllDoesCustomerhaveFutureNeeds 28
BenchmarkServiceTechPromisedinExpectedTimeframe 26
BenchmarkServiceRepairCompletedWhenPromised
BenchmarkServiceTimelinessofInvoice 25
BenchmarkServiceAppointmentAvailability 24
BenchmarkServiceTechEquippedtodoJob 23
BenchmarkAllContactStatusofFutureNeeds 22
BenchmarkServiceTechArrivedWhenPromised 21
BenchmarkAllHasIssueBeenResolved 19
BenchmarkAllContactStatusofIssue 17
BenchmarkServiceTechnicianCommunication 6
BenchmarkServiceContactPreference 3
BenchmarkCBCallansweredpromptly 1
BenchmarkServiceReceivedQuoteforRepair
BenchmarkCBAutoattendantansweredbycorrectdepartment
BenchmarkServiceCallAnsweredQuickly
BenchmarkAllMarketingPermission
Randomly chosen customers are
asked to complete Questionnaire –
It has questions concerning
personal data + 30 benchmarks
To compute NPS we calculate average score
of all benchmarks for all customers. Knowing the
number of promoters and detractors we know NPS.

12. Classical Approach Data
Data Preprocessing (including reduction of benchmarks)
Knowledge Extraction
(Classical Tools - WEKA +
Our Software for Extracting Action Rules & Their Triggers)
Recommender Systems

13. Classification Selection of Best Classification Algorithm
PART
RBF
BayesNet
NaiveBayes
KNN
RandomForest
J48
Accuracy 2 3 4 1 5 7 6
Time Taken
Total score 10 8 11

14. Two Options Initial Clients Datasets Recommender Systems1
Initial Clients Datasets Recommender Systems
local approach
global approach
Using Information about 34 Clients to Enlarge these Datasets
global approach
More Powerful Recommender Systems 2

15. FIRST APPROACH for Dataset Enlargement
Semantic Distance between Clients
J48 classifier extracted from dataset D1 of Client-1
J48 classifier extracted from dataset D2 of Client-2
Dataset for Client-2
attr1
attr2
attr3
attr4
Dataset for Client-1
attr2
attr1
attr3
attr4
Vector describing customer 1
Vector describing customer 2
Vector describe ing customer 3
Vector describing customer 1
Vector describing customer 2
Vector describe ing customer 3
More similar these two classification trees, more close semantically the clients are

16. Semantic distance based dendrogram
for Service

17. Recommender System Engine based on Semantic Similarity of Clients
Start with n1.
If NPS(n1) has to be improved, “move to n3”
If NPS(n2) > NPS(n1), then
Table3 := Table1  Table 2 is assigned to n3.
Classifier from Table3 is extracted and its
F-score is computed.
If F-score3 > F-score1,
then “move to n5”, otherwise STOP \ n3 n2 n1
Table 5, Classifier5
Table3
Classifier3
F-score3
NPS(n1)
Table1
Classifier1
F-score1
NPS(n2)
Table2
Classifier2
F-score2
For each client (node) we need to identify the best ancestor to be used for action rules discovery

22. Extracting Meta-Actions /sentiment analysis/:
We use Stanford Typed Dependencies Manual & Stanford Parser to generate grammatical relations
Extract meta-actions from relevant comments that are associated with values of attributes used in action rules;
Find smallest sets of meta-actions triggering these action rules;
Identify all customers supporting these action rules (needed to compute the increase in revenue);
Search for smallest groups of meta-actions that trigger maximal increase in revenue (we present them in descending order for same-size groups of meta- actions).
effect(m) = num(distinct records in dataset involved with m) × conf(r*)
Effect of a set of meta-action indicates the number of customers who are expected to become promoters when these meta-actions are executed.
Organize those groups of meta-actions hierarchically.

23. r1 = (A, a1  a2)  (B, b1->b2)  (C, c1  c2) 
(detractor  promoter)
Sup(r1) – number of customers supporting action rule r1 in a shop which NPS has to be improved
Number of detractors for a given shop having the properties (A,a1), (B,b1), (C,c1) - we target them
Before we start searching for triggers (meta-actions) supporting r1, this number is enlarged by adding customers
semantically similar and geographically close to our shop who also support rule r1.
Op1, Op2, Op3, Op4, Op5 – comments/sentiment about the service provided by this enlarged set of customers.
Comments about the service from promoters who satisfy the description (A,a2), (B,b2), (C,c2) are also listed.
Op1 - positive opinion related to A
Op2 - negative opinion related to C and B
Op3 – positive opinion related to B
Op4 – negative opinion related to A
Op5 – negative opinion related to C
Customers sentiment associated with rule r1 –
{Op1, Op2, Op3, Op4, Op5}
Trigger for A -> extracted from Op1 and not(Op4)
Trigger for B -> extracted from Op3 and not(Op2)
Trigger for C -> extracted from not(Op2) and not(Op5)
We search for minimum set of triggers covering A, B, C

24. Examples of Comments with Sentiment Orientation
staff+++++,best manager=112
staff-----,bad experience because diagnosis=108
staff+++++,good technician=96
staff+++++,nice guy=87
staff+++++,excellent mechanic=85
staff+++++,great guy=83
staff+++++,excellent technician=79
staff+++++,good guy=74
staff+++++,wonderful dealer=71
staff+++++,good team=66
staff+++++,honest guy=60
staff+++++,pleased with manager=40
staff-----,not available technician=34
staff-----,wrong diagnosis=16
staff+++++,best mechanic=12
staff+++++,knowledgeable mechanic=6
invoice+++++,outstanding bill=141
invoice+++++,they billed properly=65
invoice+++++,fine invoice=61
invoice-----,refused pay bills=29
invoice+++++,happy with bill=12
price-----,not fair pricing=108
price+++++,good price=108
price+++++,fair pricing=87
price-----,aggressive pricing=66
price-----,unreasonable charge fee=37
price-----,not satisfied with price=35
price+++++,better pricing=27
price-----,expensive amount charged=12
price+++++,the charged fairly=12

25. How to construct optimal sets of meta-actions (triggers activating changes in customer NPS)
Threshold for adding triggers = 6
Seeds
Sup=160
We add single trigger
one by one following
their order with respect
to support
{TR5}
{TR1,TR4}
Sup=180
Sup=165
Sup=163
{TR5,TR1}
{TR5, TR2}
TR5, TR3}
Triggers activating changes in customer NPS
Sup=184
Sup=189
Sup=185
{TR5,TR1,TR2}
{TR5,TR1,TR3}
{TR5,TR1,TR4} 1 2 3 4
TR5
{TR1,TR4}
{TR5,TR1,TR3}
{TR5,TR1}

26. Search for the best meta-actions
Client-2 is selected to be our target.
Client{2,4,6,1} by HAMIS
NPS
0.77 (73 detractors)  0.852
Action rules extracted
28,126
Meta-actions extracted 15
Number of meta-nodes
447
(*But we only focus on the top 10 in each size category of meta-nodes.)
Maximum effect in meta-nodes
45.83 (out of 73 detractors)

27. Search for the best sets of meta-actions for Client-2
idx
Meta-actions
effect
ensure service done correctly
2.00 1
improve price competitiveness
7.62 2
properly set invoice expectations(slightly high)
5.0 3
sufficient staff
2.0 7
competitive price
11.13 8
10.66 9
9.99 10
keep proactive communication
8.62
idx
Meta-actions
effect 22
ensure service done correctly
competitive price
improve price competitiveness
sufficient staff
15.07 23
properly set invoice expectations(slightly high)
15.00 24
reasonable invoice
13.13 25
11.66 46
19.41

28. Search for the best meta-actions
Why HAMIS helps?
1. 2) Extra meta-actions extracted from datasets extended by HAMIS help improve effect further.
Set of meta-actions
Expected effect
competitive price
ensure service done correctly
improve price competitiveness
sufficient staff
9.87
decrease dealer response time*
11.13
keep proactive communication
nice technician
reasonable invoice
7.59
knowledgeable technician*
13.3

33. QUESTIONS?