How to use your data science team: Becoming a data-driven organization Yael Garten Director of Data Science, LinkedIn March 29, 2016 http://www.linkedin.com/in/yaelgarten, @yaelgarten

Data is the present: every industry, organization, product, decision data driven, or at least data informed.

Good vs Great? Data is Expected, Discussed, & Accessible For decision making e.g. Determining strategy, goal setting, impact estimates of initiatives Discussed Talked about e.g. product strategy reviews, design discussions, in the hallway Accessible When questions come up, easy to get hands on data? e.g. democratized data, self-serve tools Culture Process Tools

How do we get started? (whether from scratch or to transform existing organization) Culture Process Tools Invest in the data Hire the right “data scientists”

Data Scientist – which kind? Producing for: Deliverables: Skills: human machine Find & communicate data insights Machine learning models To have a mental framework of a definition: Splits roughly into two types: Is your data scientist producing analytics for human or computer? Who is the consumer or decision maker? We’ll focus on analytics. So, what is the culture process and tools needed? such as: Product recommendations Opportunity identification Analyses Metrics Recommendation systems Classifiers Predictive models

This talk will be a success if we: Review the steps of data driven product innovation Understand what is needed to best enable fostering of (or transformation into) a data-driven organization: culture, process, and tools.

To create a data driven organization Culture: Create a culture which expects decisions are informed by data & which treats data as a first class citizen Process: Consciously map how you use data (in each phase of the product lifecycle, in making exec decisions) Tools: invest in your data ecosystem (data quality, pipeline, and access tools)

Data driven product innovation Product Lifecycle Ideation Design and Specs Development Test & Iterate Release Data driven product innovation framework: Use data to measure, understand, and improve the product: Actionable insights lead to product feature Success Metric Definition Tracking Instrumentation specification Experimentation Setup and Analysis Post-Launch Analysis Build Measure Track Ship Tweak If data scientist is not involved until this stage, it may be too late.

Build Measure Track Ship Tweak Actionable insights lead to product feature Success Metric Definition Tracking Instrumentation spec Experimentation Setup and Analysis Post-Launch Analysis Build Measure Track Ship Tweak Well-connected. Get relevance right. Few connections. Give them inventory. Opportunity sizing: how big or important is the problem? Use data to predict successful product initiatives: Show news articles Suggest new connections Suggest following active content creators

Invest in developing the right success metric. Actionable insights lead to product feature Success Metric Definition Tracking Instrumentation spec Experimentation Setup and Analysis Post-Launch Analysis Build Measure Track Ship Tweak Invest in developing the right success metric. Hypothesis: Following active sources leads to improved user experience with LinkedIn Feed Success Metric – progression of definition: Total clicks on Follow # clicks / #impressions of Follow suggestions % Feed Inventory created by new followees Downstream sustained engagement with items created by these followees What is engaging? # of clicks? Time spent? # Shares? Combo? is creating content we think you’ll find interesting

Build Measure Track Ship Tweak Actionable insights lead to product feature Success Metric Definition Tracking Instrumentation spec Experimentation Setup and Analysis Post-Launch Analysis Need accurate reliable standardized data logging to enable metric computation. ~Metric = Downstream engagement with items created by these followees Must enable attributing future clicks on feed items to that campaign as a source for the Follow. FeedActivityClick { memberID = 77777 actor = 55555 } FollowSources followCampaign666 followeeID = 55555 Lets say we go with some measure like the last. So we know what we want to measure. Are we logging the data in a way that we can compute this metric?

Design: How long to run experiment, on whom? Actionable insights lead to product feature Success Metric Definition Tracking Instrumentation spec Experimentation Setup and Analysis Post-Launch Analysis Build Measure Track Ship Tweak Rigorously set up, then identify whether the feature increased the success metric. is creating content we think you’ll find interesting Design: How long to run experiment, on whom? Implement: properly set up & randomize to ensure no bias Analyze: Go or no-go? monitor success metric, ideally automated on company-wide platform for holistic view of impacts

Iterate. How can we revise? How can we tweak to optimize? Actionable insights lead to product feature Success Metric Definition Tracking Instrumentation spec Experimentation Setup and Analysis Post-Launch Analysis Build Measure Track Ship Tweak Iterate. How can we revise? How can we tweak to optimize? is creating content we think you’ll find interesting Reporting, monitoring, ad hoc analysis Long term measures of engagement/success Analysis to inform revision of design

Invest in foundations. Culture Process Tools

To create a data driven organization Culture: Create a culture which expects decisions are informed by data & which treats data as a first class citizen Process: Consciously map how you use data (in each phase of the product lifecycle, in making exec decisions) Tools: invest in your data ecosystem (data quality, pipeline, and access tools)

Data Democratization: self serve data exploration platform Build Measure Track Ship Tweak Actionable insights lead to product feature Success Metric Definition Tracking Instrumentation spec Experimentation Setup and Analysis Post-Launch Analysis Data scientist as a partner, not a service – give context and ownership Strong bias to actionable impactful insights, speed of iteration & feedback Data Foundations: governed datasets, consistent shared datasets and metrics Data Democratization: self serve data exploration platform Enable innovation: environment supports speedy ad-hoc analysis Culture Process Tools

Democratize data – self served data exploration platform Actionable insights lead to product feature Success Metric Definition Tracking Instrumentation spec Experimentation Setup and Analysis Post-Launch Analysis Build Measure Track Ship Tweak Democratize data – self served data exploration platform Enable people in your organization (execs, product managers, designers) to have data at their fingertips – to ask and answer questions

s Invest in creating the right metrics Actionable insights lead to product feature Success Metric Definition Tracking Instrumentation spec Experimentation Setup and Analysis Post-Launch Analysis Build Measure Track Ship Tweak s Culture Invest in creating the right metrics user centric mindset; optimize for user value not team success all stakeholders agree upon success metrics prior to launching the feature test Platform of shared tiered metrics visible to entire company a metrics pipeline that enables easy implementation of metrics (and not manual one-offs) Process Tools Invest in created more sophisticated measure of value created for the user or customer. If Profile team increases Profile edits (better matching), but it decreases connections made (which might impact growth, new user acquisition) what does that mean? Example - Total signups vs quality signups At LinkedIn we have true norths. And we have tiers. Total Follows vs follow that drives to more liquidity as seen by long term engagement, click through of content generated by those followees

s Build Measure Track Ship Tweak Actionable insights lead to product feature Success Metric Definition Tracking Instrumentation spec Experimentation Setup and Analysis Post-Launch Analysis s

Data as a first class citizen Data tracking bugs as a launch blocker Actionable insights lead to product feature Success Metric Definition Tracking Instrumentation spec Experimentation Setup and Analysis Post-Launch Analysis Build Measure Track Ship Tweak Culture Data as a first class citizen Data tracking bugs as a launch blocker Proactive joint definition of data requirements and contract (schemas) by producers and consumers Data Model Review Committee Data tracking spec tool for source of truth Automated testing of data tracking Data Quality monitoring tool to ensure data contract is met Process Tools

s Belief in proactive controlled experiment as decision making tool Actionable insights lead to product feature Success Metric Definition Tracking Instrumentation spec Experimentation Setup and Analysis Post-Launch Analysis Build Measure Track Ship Tweak s Culture Belief in proactive controlled experiment as decision making tool Efficient and principled lifecycle, from inception to decision Before: Review experiment design & implementation After: stakeholders discuss impacts and implications of tradeoffs Company-wide experimentation platform, with tiered key metrics Automated metric reporting & analysis capability Process Tools

s Build Measure Track Ship Tweak Actionable insights lead to product feature Success Metric Definition Tracking Instrumentation spec Experimentation Setup and Analysis Post-Launch Analysis s

s Iteration and innovation Actionable insights lead to product feature Success Metric Definition Tracking Instrumentation spec Experimentation Setup and Analysis Post-Launch Analysis Build Measure Track Ship Tweak s Culture Iteration and innovation Metrics meeting: weekly to understand performance and product value Combine qualitative user feedback with quantitative results Long term hold-out groups for deep understanding of impacts Process Tools

This talk will be a success if we: Review the steps of data driven product innovation Understand what is needed to best enable fostering of (or transformation into) a data-driven organization: culture, process, and tools.

To create a data driven organization Culture: Create a culture which expects decisions are informed by data & which treats data as a first class citizen Process: Consciously map how you use data (in each phase of the product lifecycle, in making exec decisions) Tools: invest in your data ecosystem (data quality, pipeline, and access tools)

Thank you. http://www.linkedin.com/in/yaelgarten @yaelgarten