1. How to use your data science team: Becoming a data-driven organization
Yael Garten
Director of Data Science, LinkedIn
March 29, 2016

2. Data is the present: every industry, organization, product, decision
data driven, or at least data informed.

3. Good vs Great? Data is Expected, Discussed, & Accessible
For decision making e.g. Determining strategy, goal setting, impact estimates of initiatives
Discussed
Talked about e.g. product strategy reviews, design discussions, in the hallway
Accessible
When questions come up, easy to get hands on data? e.g. democratized data, self-serve tools
Culture
Process
Tools

4. How do we get started? (whether from scratch or to transform existing organization)
Culture
Process
Tools
Invest in the data
Hire the right “data scientists”

5. Data Scientist – which kind?
Producing for:
Deliverables:
Skills:
human machine
Find & communicate data insights
Machine learning models
To have a mental framework of a definition: Splits roughly into two types: Is your data scientist producing analytics for human or computer? Who is the consumer or decision maker?
We’ll focus on analytics. So, what is the culture process and tools needed?
such as:
Product recommendations
Opportunity identification
Analyses
Metrics
Recommendation systems
Classifiers
Predictive models

6. This talk will be a success if we:
Review the steps of data driven product innovation
Understand what is needed to best enable fostering of (or transformation into) a data-driven organization: culture, process, and tools.

7. To create a data driven organization
Culture: Create a culture which expects decisions are informed by data & which treats data as a first class citizen
Process: Consciously map how you use data (in each phase of the product lifecycle, in making exec decisions)
Tools: invest in your data ecosystem (data quality, pipeline, and access tools)

8. Data driven product innovation
Product Lifecycle
Ideation
Design and Specs
Development
Test & Iterate
Release
Data driven product innovation framework: Use data to measure, understand, and improve the product:
Actionable insights lead to product feature
Success Metric Definition
Tracking Instrumentation specification
Experimentation Setup and Analysis
Post-Launch Analysis
Build Measure Track Ship Tweak
If data scientist is not involved until this stage, it may be too late.

9. Build Measure Track Ship Tweak
Actionable insights lead to product feature
Success Metric Definition
Tracking Instrumentation spec
Experimentation Setup and Analysis
Post-Launch Analysis
Build Measure Track Ship Tweak
Well-connected. Get relevance right.
Few connections. Give them inventory.
Opportunity sizing: how big or important is the problem?
Use data to predict successful product initiatives:
Show news articles
Suggest new connections
Suggest following active content creators

10. Invest in developing the right success metric.
Actionable insights lead to product feature
Success Metric Definition
Tracking Instrumentation spec
Experimentation Setup and Analysis
Post-Launch Analysis
Build Measure Track Ship Tweak
Invest in developing the right success metric.
Hypothesis: Following active sources leads to improved user experience with LinkedIn Feed
Success Metric – progression of definition:
Total clicks on Follow
# clicks / #impressions of Follow suggestions
% Feed Inventory created by new followees
Downstream sustained engagement with items created by these followees
What is engaging? # of clicks? Time spent? # Shares? Combo?
is creating content we think you’ll find interesting

11. Build Measure Track Ship Tweak
Actionable insights lead to product feature
Success Metric Definition
Tracking Instrumentation spec
Experimentation Setup and Analysis
Post-Launch Analysis
Need accurate reliable standardized data logging to enable metric computation.
~Metric = Downstream engagement with items created by these followees
Must enable attributing future clicks on feed items to that campaign as a source for the Follow.
FeedActivityClick {
memberID = 77777
actor = 55555 }
FollowSources
followCampaign666
followeeID = 55555
Lets say we go with some measure like the last. So we know what we want to measure. Are we logging the data in a way that we can compute this metric?

12. Design: How long to run experiment, on whom?
Actionable insights lead to product feature
Success Metric Definition
Tracking Instrumentation spec
Experimentation Setup and Analysis
Post-Launch Analysis
Build Measure Track Ship Tweak
Rigorously set up, then identify whether the feature increased the success metric.
is creating content we think you’ll find interesting
Design: How long to run experiment, on whom?
Implement: properly set up & randomize to ensure no bias
Analyze: Go or no-go? monitor success metric, ideally automated on company-wide platform for holistic view of impacts

13. Iterate. How can we revise? How can we tweak to optimize?
Actionable insights lead to product feature
Success Metric Definition
Tracking Instrumentation spec
Experimentation Setup and Analysis
Post-Launch Analysis
Build Measure Track Ship Tweak
Iterate. How can we revise? How can we tweak to optimize?
is creating content we think you’ll find interesting
Reporting, monitoring, ad hoc analysis
Long term measures of engagement/success
Analysis to inform revision of design

14. Invest in foundations.
Culture
Process
Tools

15. To create a data driven organization
Culture: Create a culture which expects decisions are informed by data & which treats data as a first class citizen
Process: Consciously map how you use data (in each phase of the product lifecycle, in making exec decisions)
Tools: invest in your data ecosystem (data quality, pipeline, and access tools)

16. Data Democratization: self serve data exploration platform
Build Measure Track Ship Tweak
Actionable insights lead to product feature
Success Metric Definition
Tracking Instrumentation spec
Experimentation Setup and Analysis
Post-Launch Analysis
Data scientist as a partner, not a service – give context and ownership
Strong bias to actionable impactful insights, speed of iteration & feedback
Data Foundations: governed datasets, consistent shared datasets and metrics
Data Democratization: self serve data exploration platform
Enable innovation: environment supports speedy ad-hoc analysis
Culture
Process
Tools

17. Democratize data – self served data exploration platform
Actionable insights lead to product feature
Success Metric Definition
Tracking Instrumentation spec
Experimentation Setup and Analysis
Post-Launch Analysis
Build Measure Track Ship Tweak
Democratize data – self served data exploration platform
Enable people in your organization (execs, product managers, designers) to have data at their fingertips – to ask and answer questions

18. s Invest in creating the right metrics
Actionable insights lead to product feature
Success Metric Definition
Tracking Instrumentation spec
Experimentation Setup and Analysis
Post-Launch Analysis
Build Measure Track Ship Tweak s
Culture
Invest in creating the right metrics
user centric mindset; optimize for user value not team success
all stakeholders agree upon success metrics prior to launching the feature test
Platform of shared tiered metrics visible to entire company
a metrics pipeline that enables easy implementation of metrics (and not manual one-offs)
Process
Tools
Invest in created more sophisticated measure of value created for the user or customer. If Profile team increases Profile edits (better matching), but it decreases connections made (which might impact growth, new user acquisition) what does that mean?
Example - Total signups vs quality signups
At LinkedIn we have true norths. And we have tiers.
Total Follows vs follow that drives to more liquidity as seen by long term engagement, click through of content generated by those followees

19. s Build Measure Track Ship Tweak
Actionable insights lead to product feature
Success Metric Definition
Tracking Instrumentation spec
Experimentation Setup and Analysis
Post-Launch Analysis s

20. Data as a first class citizen Data tracking bugs as a launch blocker
Actionable insights lead to product feature
Success Metric Definition
Tracking Instrumentation spec
Experimentation Setup and Analysis
Post-Launch Analysis
Build Measure Track Ship Tweak
Culture
Data as a first class citizen
Data tracking bugs as a launch blocker
Proactive joint definition of data requirements and contract (schemas) by producers and consumers
Data Model Review Committee
Data tracking spec tool for source of truth Automated testing of data tracking
Data Quality monitoring tool to ensure data contract is met
Process
Tools

21. s Belief in proactive controlled experiment as decision making tool
Actionable insights lead to product feature
Success Metric Definition
Tracking Instrumentation spec
Experimentation Setup and Analysis
Post-Launch Analysis
Build Measure Track Ship Tweak s
Culture
Belief in proactive controlled experiment as decision making tool
Efficient and principled lifecycle, from inception to decision
Before: Review experiment design & implementation
After: stakeholders discuss impacts and implications of tradeoffs
Company-wide experimentation platform, with tiered key metrics
Automated metric reporting & analysis capability
Process
Tools

22. s Build Measure Track Ship Tweak
Actionable insights lead to product feature
Success Metric Definition
Tracking Instrumentation spec
Experimentation Setup and Analysis
Post-Launch Analysis s

23. s Iteration and innovation
Actionable insights lead to product feature
Success Metric Definition
Tracking Instrumentation spec
Experimentation Setup and Analysis
Post-Launch Analysis
Build Measure Track Ship Tweak s
Culture
Iteration and innovation
Metrics meeting: weekly to understand performance and product value
Combine qualitative user feedback with quantitative results
Long term hold-out groups for deep understanding of impacts
Process
Tools

24. This talk will be a success if we:
Review the steps of data driven product innovation
Understand what is needed to best enable fostering of (or transformation into) a data-driven organization: culture, process, and tools.

25. To create a data driven organization
Culture: Create a culture which expects decisions are informed by data & which treats data as a first class citizen
Process: Consciously map how you use data (in each phase of the product lifecycle, in making exec decisions)
Tools: invest in your data ecosystem (data quality, pipeline, and access tools)

26. Thank you.
@yaelgarten