Azure Automation and Logic Apps: Your SQL Agent in the cloud

Azure Automation and Logic Apps Your SQL Agent in the cloud Intelligent Cloud Conference 30/5-2018 Christian Winther Kristensen Consulting Manager & Principal Architect, Kapacity cwk@kapacity.dk

Event Sponsors Expo Sponsors Expo Light Sponsors

Azure Logic Apps How many have used it?

What is Azure Logic Apps? Automate business processes in the cloud and on-prem Systems integration MS Flow for developers in Azure Can be triggered by events or schedules Has many connectors (200+), e.g. SQL Server, Sharepoint and e-mail

Why Logic Apps as orchestrator? Simple and user-friendly GUI Low cost SSIS in Azure is too expensive + not using SSIS for data transform No VM with SQL Server No Azure SQL DB Managed Instance with SQL Agent Needs other event triggers than date & time You do not want to make everything in runbooks (PowerShell) and/or ADF Start jobs and pipelines from Logic Apps

A lot of template flows

Hybrid scenarios? Use on-prem data gateway and supported connectors https://docs.microsoft.com/en- us/azure/logic-apps/logic-apps- gateway-connection

Logic Apps Demo!

Lessons learned… Biggest limitation is that actions as a general rule has a 2 minute duration before timeout Possible solution: Use PowerShell runbook instead - invoked from Logic Apps ”Automation create job” action

Trigger by event Also can be triggered by a change in a SQL table New row Changed column value Hint: you can use a stored procedure to change a value  trigger a Logic App flow

Trigger by SQL Stored Procedure example CREATE TABLE [etl].[LogicApp]( [RV] [timestamp] NOT NULL, [DWModifiedDate] [datetime] NOT NULL ) ON [PRIMARY] GO CREATE PROCEDURE [etl].[Trigger_LogicAppETLflow] AS BEGIN SET NOCOUNT ON UPDATE [etl].[LogicApp] SET [DWModifiedDate] = GETDATE() END

Azure Automation Who uses azure automation?

Azure Automation delivers a cloud-based automation and configuration service that provides consistent management across your Azure and non-Azure environments. consists of process automation, update management, and configuration features. provides complete control during deployment, operations, and decommissioning of workloads and resources.

Main Features Process Automation  Runbooks PowerShell / PowerShell Workflow Graphical – PowerShell/Workflow Declarative Automation  PowerShell DSC (Desired State Configuration) Use Automation as a pull server Central location to view the state of all of your managed nodes Secure global asset store Certificates Credentials Variables Connections Modules Schedules Run on hybrid or in Azure

PowerShell Runbooks PowerShell runbooks are based on Windows PowerShell. You directly edit the code of the runbook using the text editor in the Azure portal. You can also use any offline text editor and import the runbook into Azure Automation.

PowerShell Runbooks vs. Workflows Advantages Implement all complex logic with PowerShell code without the additional complexities of PowerShell Workflow. Runbook starts faster than PowerShell Workflow runbooks since it doesn't need to be compiled before running. Limitations Must be familiar with PowerShell scripting. Can't use parallel processing to perform multiple actions in parallel. Can't use checkpoints to resume runbook in case of error. PowerShell Workflow runbooks and Graphical runbooks can only be included as child runbooks by using the Start-AzureAutomationRunbook cmdlet which creates a new job. 3 hour time limit for a runbook – long running workflows will restart at latest checkpoint

PowerShell for scaling SQL DB param( [parameter(Mandatory=$False)] [string] $SqlServer = "kapacity-cwk.database.windows.net", [int] $SqlServerPort = 1433, [string] $Database = "master", [string] $SqlCredentialAsset = "cwk" ) $SqlCredential = Get-AutomationPSCredential -Name $SqlCredentialAsset if ($SqlCredential -eq $null) { throw "Could not retrieve '$SqlCredentialAsset' credential asset. Check that you created this first in the Automation service." } # Get the username and password from the SQL Credential $SqlUsername = $SqlCredential.UserName $SqlPass = $SqlCredential.GetNetworkCredential().Password # Define the connection to the SQL Database $Conn = New-Object System.Data.SqlClient.SqlConnection("Server=tcp:$SqlServer,$SqlServerPort;Database=$Database;User ID=$SqlUsername;Password=$SqlPass;Trusted_Connection=False;Encrypt=True;Connection Timeout=30;") # Open the SQL connection $Conn.Open() # Define the SQL command to run. $Cmd=new-object system.Data.SqlClient.SqlCommand $Cmd.Connection = $Conn $Cmd.CommandText = "ALTER DATABASE [AdventureWorks] MODIFY (SERVICE_OBJECTIVE = 'S0') WAITFOR DELAY '00:02' " $Cmd.CommandTimeout=1200 # Execute the SQL command $Cmd.ExecuteNonQuery() # Close the SQL connection $Conn.Close()

Create Runbook Demo!

Automation Jobs A job is a run of a Runbook Jobs can be invoked from a schedule (or called from Logic Apps automation action)

Schedule Runbook

Processing Azure Analysis Services Install SQL Server powershell module

Processing Azure Analysis Services Remember to create a Azure AD service account user with admin access to AAS. # Providing the Server Details $ServerName = "asazure://northeurope.asazure.windows.net/cwkaasdemo" $DatabaseName = "awlt" #Getting the credential which we stored earlier. $AzureCred = Get-AutomationPSCredential -Name "svc_bi_automation" Add-AzureRmAccount -Credential $AzureCred | Out-Null #Full Processing the DB Invoke-ProcessASDatabase -databasename $DatabaseName -server $ServerName -RefreshType "Full" -Credential $AzureCred

Create Job Demo!

Combining Logic Apps with Azure Automation

Azure Automation actions Azure Automation connector and 3 actions (preview) Use the Create job action

Azure Automation Create job action Fill out subscription info, runbook name Wait for job = Yes This action can run for longer than 2 minutes – depending on runbook code timeout settings Eventual runbook parameters

Logic Apps calling Runbook Demo!

Logic apps run

More questions ? https://docs.microsoft.com/en-us/azure/logic-apps/ https://docs.microsoft.com/en-us/azure/automation/automation- intro https://docs.microsoft.com/en-us/azure/automation/automation- runbook-types