1. DataOps
DataOps
DevOps for Analytics

2. Hi, I’m Steph! An MVP & data science consultant.

3. DevOps Dev Test Ops What is DevOps?
DevOps is bridging the gap between different teams, culturally and technologically to reduce the risk and cost of software development in an organisation.
Culturally, it means things like ownership of changes, “left-shifting” responsibility for testing, lower bureaucracy
Technologically, it means things like unit testing, immutable architecture, continuous integration, continuous deployment, real-time application monitoring and alerting

4. DBs
Test
Ops
What’s DLM?
Database Lifecycle Management is the work Microsoft & Redgate are doing to facilitate unit testing, CI, CD, and pipeline management for databases
Is it DevOps?
It depends! Yes, people need to do some of this database stuff for applications so DLM needs to be usable for a DevOps shop, but they should also be doing it for data warehouses and other data repositories. It’s vital but given that it’s not just about development it should be considered separately.

5. BI Test Ops What do people (sort of) use to make decisions?
Facts / data / information. These are retrieved from source systems, integrated, transformed, and finally reported on. So many steps involved and so many point of failure.
What to do about it?
Test, test, test! But how? Unit test frameworks, metadata driven, scripted builds. These enable you to be more RIGHT – incorporate them and do so in an automated fashion and your bringing Ops and Testins closer. This is DataOps

6. Test Ops Analytics Analytics
AKA data science, statistics, data mining, machine learning
The problem of time
We saw in BI that we need to be RIGHT – but we’re only ever aiming for RIGHT at the point of presentation. Analytics is aiming to be RIGHT over people and time. It’s about making predictions that as accurate today as they are tomorrow. That’s a really hard thing since when you start changing things, you can’t tell what would have happened if you hadn’t, unless you experiment. You need to be able to account for changing behaviour over time, and changes to data capture etc. This means your modelling has to be updateable and refinements easy to deploy, and at the same time remaining robust.
How to solve?
Coded analysis, regular retraining, and a validation and deployment pipeline are all needed to facilitate on-going analytics. Analytics has to be automated to be able to cope.

7. Data
Test
Ops
DataOps
What do DBs, BI, and Analytics have in common? They’re all about the data
DataOps is all about bringing rigour and speed of delivery to how we store, process, use, and present data. It’s a vital movement that will enable BI departments to keep pace with the “modern world” and helps them scale to meet the increasing demand for data.
So, yeah, #DataOps is totally a thing!

8. DataOps is moving data roles and ops closer

9. Current People Tools Process Quantified value Planning Continuous
improvement
Process
Code
review
Coping with
change
Productivity
tools
Tools
Satisfaction /
fitness
metrics
Continuous
integration
People
Teach
simplicity
Continuous
learning
Automation
Quick
to build
Coherent
communication
Face to face /
virtual

10. Ideal People Tools Process Quantified value Planning Continuous
improvement
Code
review
Coping with
change
Continuous
learning
People
Productivity
tools
Tools
Satisfaction /
fitness
metrics
Face to face /
virtual
Continuous
integration
Teach
simplicity
Automation
Quick
to build
Coherent
communication

11. To improve is to change; to be perfect is to change often.
Winston Churchill

12. Current
Get & Tidy
Transform
Viz
Model
Transform
@hadleywickham

13. Ideal
Viz
Model
Transform
Get & Tidy
Transform
@hadleywickham

14. Process Value / idea Prototype Dev Test Deliver Prioritise Release
Ticket
Prioritise
Dev
Test
Release

15. It’s never about how you start – it’s always about how you finish
The Rock

16. Good reading The Phoenix Project The Art of Agile
Agile Data Warehouse Design
DataOps.info
DataOps Manifesto

17. Collaboration Azure Visual Studio Online Trello GitHub
White board & post-its
Slack

18. Data Cost Tooling Learning SSDT Free Medium-High Medium Redgate DLM
Anchor Modeling
Low
High
MSBuild
Medium-high
CosmosDB
Github.com/stephlocke/MeDriAnchor

19. ETL Cost Tooling Learning BIML Free or High High SQL Free Medium
Powershell
Azure Functions
Low
SSIS
Free or Medium
Github.com/stephlocke/MeDriAnchor

20. Reporting Cost Tooling Learning Excel Free/Low Low PowerBI Medium
Mediuim
SSRS
Free-Medium
Fairly low R
Free
High
Medium-high
Other ?

21. Cubes Cost Tooling Learning Tabular Free or Medium Medium Original
Free or High
Low
High
Github.com/stephlocke/MeDriAnchor

22. Data Science Cost Tooling Learning R Free High Python Docker
Free/Medium
Medium
Microsoft ML
Free / High
Azure ML
Free / Low
Low
H2O
Spark

23. @stefflocke @lockedata steph@itsalocke.com
Follow up!
@stefflocke @lockedata