Wagner Silveira Microsoft MVP - Azure Creating a Processing Pipeline with Azure Function and AIS

About me Principal Integration Architect Microsoft Azure MVP MCSE Cloud Platform About me wsilveiranz wagner.silveira@notetoself.tech https://notetoself.tech

Agenda Case Study Initial Solution Updated Solution Key Components A year later – what changed? Lessons Learned

Case Study

The solution requirements Onboarding of external EDI messages Should allow for different transports Message need to be Validated, translated, enriched, routed Control on retries and notification over failed messages Auditing of initial message Visibility of where a single message was in the process

Logical Design

Initial Solution

Reality Check Big Payloads EDIFACT schema not available Operational costs End to end Monitoring

Updated Solution

How was this different from the initial solution? Azure Functions EDIFACT support via .NET Package Claim check pattern Dedicated instances Azure Storage Payload storage (claim check pattern) Application Insights End to End with custom events Single technology for monitoring and notification

Key Components

Key Components Claim check pattern App Insights Custom Events

The claim check pattern Publishing function separates content from metadata then: Store content on a safe repository Publish metadata + storage pointer + Reprocessing count to message engine Subscribing processes receive metadata and retrieve content when required The claim check pattern

App Insights custom events End to end traceability using a Correlation ID Allow for exceptions to be captured the same way Leverage Functions support for App Insights Implemented as a reusable component App Insights

Hands on Time?

Key Management Leveraged MSI support where available SQL Server Storage Key vault Kept keys in key vault where MSI where not supported directly Service Bus

Exception Management & Retries on Functions Catch blocks using a notify and throw pattern Leverage Function SB binding peek-lock pattern Adjusted Maximum Delivery Count on SB Messages that failed where delivered to DLQ

DLQ Management Logic Apps polling subscriptions DLQ every 6 hours Each subscription DLQ could have it’s own logic Reprocessing count Resubmission logic Notification logic Email notification Error blob storage

One year later…

New technology candidates Integration Services Engine Azure Durable Functions Event Grid

What about new features Azure Functions Premium Improved support for Key Vault Improved support for MSI Virtual Network support + Service Endpoints

Lessons learned

Review the fine print

Operational cost is a design consideration

Make the best of each technology

Think about the big picture

In Summary Weight your technology options Think about limitations, operational costs Understand how your solution fit in the big picture Understand the components roadmap

Thank you for attending!