Global Consumer Insights Automating business insights through machine learning & artificial intelligence March 6, 2018

General Mills

Purpose of this talk 1. Share our experience and maybe you can find ways to apply it in your space 2. Challenge us as practitioners to figure out how to make automation a reality in business 3. When people think about jobs in this space, don’t only think of Start-ups and tech firms

How do we help over 1,000 people in marketing with Data Analysis & Analytics?

Our original problem DATA EVERYWHERE OTHER PEOPLES DATA SILO DATA OWNERS Each of these prevent analytics and business intelligence to be built across data

So we organized in Marketing Formed a dedicated team ANALYTICS & DATA SCIENCE DATA STEWARDS DATA VISUALIZERS Ensure quality of our data and maintain connections Main point for training and inquiries Develop easy way to use the data Engineer new data sources and access Purpose built analytics

And we partnered with IT FROM TO

And we made an impact Availability Enable access to data people could not access before Documentation to educate Connections Data speaks to each other Align the data to the way marketers think New Insights Purpose built visuals to answer common business questions Analytics to answer advanced questions New possibilities on what we can answer Reduce time to prep analysis Reduce time to answer or enable ability to answer questions

For example…Way to track new products Built a visual to track New Product launches 50+ charts, 120 metrics, all important 15-20 minutes for a user to see what is going on with each new item Data warrants watching it weekly A marketer can have 5-8 new items at any given time

But our marketers are busy Time Priority 8am Finish what they did not yesterday 9am Prepping for a retailer conversation 10am Got an urgent request from leadership 11am Still scrambling to get an answer 12 Talk to media buying group about an ad in flight 12:55 Try to grab something to eat 1 Meet with Finance 2 Work with product development on a future flavor 3 Discuss what to do about plant restrictions 4 Finalize an answer to the urgent request from leadership 5 Run and grab the kids 9pm…. Answer all the emails from the day Jane Marketing Manager on Nature Valley Repeat

Time to think differently How do we identify what products have something unique going on? Quickly guide the marketer to what is unique about that product. Increase speed to action.

Wanted to identify the stuff the marketer should look at Types of Anomalies: Point: Individual data point can be considered anomalous with respect to the rest of the data. Collective / Contextual: instance is anomalous in a specific context (but not otherwise), then it is termed as a contextual (conditional) outlier. If group of points, then termed collective outlier. Global: Anomalous data points are defined by measuring the global deviation of a given data point with respect to its neighbors, globally. Local: Anomalous data points are defined by measuring the local deviation of a given data point with respect to its neighbors. Business Transactions: Anomalous data points related to some business transaction often measured via a KPI ($’s or Volume) over time (think time series). Reference Data: Anomalous data points related to the metadata or reference data about some entity.

How about machine leaning? Approach Example Algorithms Considerations Rule-Based zscore, frequency, hard-coded rules Must be able to define abnormal Must define rules Does not adapt to change Supervised Classification, Logistic Regression Need data labeled as abnormal No need to define rules Adapts to change Unsupervised Clustering (KNN, DBSCAN), Isolation Forest, Deep Learning Autoencoder No need to have labeled data

Algorithms worked We had 2 products that were meant to be limited edition: They were similar but 2 different flavors. PRODUCT A PRODUCT B

What happened with these products? PRODUCT A PRODUCT B GOOD, not AMAZING Let Limited Edition run and discontinue A CLEAR WINNER Let’s keep it

Our algorithms alerted us Consumers were reacting differently Right away we could tell a difference in sentiment between Product B and Product A Product B “I hope it will become a permanent product” Product A “Is there any way I can exchange it”

So our marketers should have said let’s keep Product B very early on Start having discussions with retailers about keeping it Alert manufacturing to keep sourcing this Decide on a strategy to keep this as a permanent product But we had a problem

We deployed the algorithms in our visualizations 1. A business user logs into the visual and sees what launches are anomalous 2. They see what dates where there is something worth their attention 3. They can then see what business metrics are being flagged for their attention

But…it requires a business user to investigate They have to proactively go out and look You are competing with all their other priorities And this is not simple enough

Our big aha… We are competing for the business user time We have to bring automation of insights to where they already are As much as it pains me to say, an email alert on what they need to know is what my user base needs

Key Takeaways It’s a journey Treat business user experience like a consumer experience We are never done

Global Consumer Insights Thank You