Building highly scalable enterprise systems (on Azure) Ronald Harmsen In this practical session I will show you how to design and build a highly scalable and distributed system. Combining the core architectural principles of CQRS, Event Sourcing & Actor Systems to create the design and then building it with .NET core, AKKA.NET & Docker. That’s a lot of technology right? Well, they fit very nicely together and even make sense when combined. This architecture has been used successfully in some very large scale enterprise systems I’ve been working on. Of course I’ll guide you through all steps from understanding the design to deployment into Azure. Why did I choose these patterns? What are my options for scaling? How did we refactor our old implementation into this solution? And more… I’ll start with a couple of architectural drawings and the rest is all about the stuff we care about: running code. @ronaldharmsen ronald@nforza.nl #DevSum19

Ronald Harmsen Developer / Consultant / Trainer @ NForza Development >= 1993 Everything .NET >= 2004 Azure >= 2009 Others: DevOps, Software Quality, Software & Enterprise Architecture #DevSum19

The project where it all started #DevSum19

Our “adventure” Main dataset: approx. 750TB (billions of tiles:metadata/photographs) Initial application tests : ~273 days to calculate Planned run: every quarter 

Our “adventure” part 2 Issue with waiting for neighbors to be available Reloading of data Storage slow

Solution space Can we have ‘caching’ of state ? Process tiles independently ? Execute ‘multi-stage’ detections on tiles ? i.e. run algorithms for water, road, forest, buildings etc. independently? #DevSum19

The Actor Model #DevSum19 Lightweight Never share state Mailbox = message queue. So message-based or message-passing architecture One message at a time Simple threading scenario (code we write is single threaded, abstracted away in actor model) #DevSum19

Goals Resilient Scalable Fast Cloud based (Azure) #DevSum19 Resilient – we don’t want to completely start over when something crasheds #DevSum19

#DevSum19 We had some issues: Programming model How many people know how to do this? Slow deployments -> rolling updates of stateful services & hence actors #DevSum19

Azure Kubernetes Services (AKS) #DevSum19

Goals Resilient Scalable Fast Cloud based (Azure) Actor model ? Resilient – we don’t want to completely start over when something crashes Fast deployments / updates Wide spread. More knowledge available. #DevSum19

Which Actor Model ? #DevSum19 Akka. Orleans too much MS again Akka = open source, pluggable architecture Scalable -> remoting / clusters / sharding #DevSum19

Demo: akka.net actors #DevSum19 Resilient – we don’t want to completely start over when something crashes Fast deployments / updates Wide spread. More knowledge available. #DevSum19

What about resilience? Akka has some useful power-ups: Akka Remoting Akka Clusters Persistence Sharding Resilient – we don’t want to completely start over when something crashes Fast deployments / updates Wide spread. More knowledge available. #DevSum19

Akka Remoting #DevSum19

Akka Clusters #DevSum19

Akka Persistence - CQRS #DevSum19

Akka Persistence - CQRS #DevSum19

Akka Persistence #DevSum19

Akka Sharding #DevSum19

Putting it all together in code Resilient – we don’t want to completely start over when something crashes Fast deployments / updates Wide spread. More knowledge available. aka DEMO #DevSum19

Summing up Using Docker / Kubernetes for scaling Relatively simple programming model Easy to deploy the same architecture into Azure Resilient Resilient – we don’t want to completely start over when something crashes Fast deployments / updates Wide spread. More knowledge available. #DevSum19

And…. Last but not least – don’t forget to evaluate this session in the DevSum app!