Azure ML in SSIS An introduction to Azure Machine Learning Through the eyes of an SSIS developer David Söderlund – SolidQ Nordic LinkedIn:

Sponsors

Using and improving Azure ML in SSIS  Introduction to Azure Machine Learning  Predictive analysis  Fully managed in Azure  Pay as you go  Free trials available with limited functionality  Interacting with Azure ML Predictive experiments through SSIS  Custom component for batch execution  Azure Feature Pack for blob storage interaction |SQL Saturday #433, Gothenburg3 |

Machine Learning / Data mining  Consider machine learning if  There is a pattern you would like to find  There is no apparent mathematical solution  There is enough data  Senior data scientists get involved in larger projects but everyone should know the essentials.  We will learn how to implement machine learning models in our daily ETL! |SQL Saturday #433, Gothenburg4 |

Machine Learning / Data mining  What is learning?  Reinforcement learning by looking at examples, this is why we need data  Look at examples with correct labeling and try to figure out the target function  Come up with hypothesis function  Validate |SQL Saturday #433, Gothenburg5 |

Typical Machine Learning project roles  Domain expert  The people who know the data for the problem  Developer  The people who can transform data to conform to the models and to the project’s needs  Data scientist  Runs the experimental process  The name comes from applying the scientific method to data driven decision making |SQL Saturday #433, Gothenburg6 |

Azure ML compared to SSAS DM  Getting started, developing models and trying out ideas is not a huge project in itself.  To try out a model requires data at hand and a web browser  You don’t have to know everything to get some insights  SSAS is better for big projects with more complex models.  Better visualization tools in visual studio |SQL Saturday #433, Gothenburg7 |

Azure ML short overview (demo soon)  The final product is a model  The model can be used to predict values on data that it hasn’t seen before  Everything is an experiment  Not every experiment is successful  You don’t need to know everything to run it |SQL Saturday #433, Gothenburg8 |

Azure ML short overview (demo soon)  Training experiments  Looks like an SSIS package  When it works we can “compile” the model and produce a predictive experiment  Predictive experiment  Uses a “trained model” and does not retrain every run  The inputs are run through the model and become outputs |SQL Saturday #433, Gothenburg9 |

Demo? DEMO |SQL Saturday #433, Gothenburg10 |

Executing predictive scoring  Web service API  Request – Response Service (RRS)  Row by agonizing row  Useful for streaming data  Batch Execution Service (BES)  Can only run off of files in azure blob storage  Therefor we must make sure our data is in that format before running the batch  Results are also stored in this format and a relative location is returned in the execution status  We can use this to retrieve the results! |SQL Saturday #433, Gothenburg11 |

Batch execution service from SSIS  Make sure the data is in blob storage  Run the batch execution by pointing to the correct container and file  Read the data from blob storage with the container/file that execution returned |SQL Saturday #433, Gothenburg12 |

SSIS requirements for development  SSIS Azure Feature Pack  A custom component for the HTTP-request  Connection manager to our source data  Connection manager to azure blob storage  Connection information to our Azure ML experiment  Variables to help us keep track of where the experiment output is |SQL Saturday #433, Gothenburg13 |

SSIS requirements for development |SQL Saturday #433, Gothenburg14 |

SSIS requirements for development |SQL Saturday #433, Gothenburg15 |

Demo? DEMO |SQL Saturday #433, Gothenburg16 |

What about updating and improving?  Experiments can have multiple web service outputs, so make an output for the score model module.  Run the training experiment through it’s API and catch both the new model object and its score.  In SSIS compare the old and the new then update the predictive experiment with the new model, if it’s better |SQL Saturday #433, Gothenburg17 |

Automate!  Figure out how the components work  Parameterize the locations and keys that change with each experiment  Auto-generate packages from metadata with BIML |SQL Saturday #433, Gothenburg18 |

Useful links to replicate setup  Azure trial account (for storage and studio) Azure trial account (for storage and studio)  SSIS Azure Feature pack SSIS Azure Feature pack  Chris Price (BluewaterSQL) ExecureAzureMLBatch Component Chris Price (BluewaterSQL) ExecureAzureMLBatch Component  Some assembly required… get it?  How to retrain models programmatically by Raymond Laghaeian (not part of demo) How to retrain models programmatically by Raymond Laghaeian  Automate your batch package with BIML Automate your batch package with BIML |SQL Saturday #433, Gothenburg19 |