GeoScience and µServices

Background Landmark would be one of the top 50 largest software companies Scientific and Engineering software for Energy sector Exploration Construction Production Neftex

What kind of developer are you? Experienced with Java (since 1.1 which is ~1998) Python hacker (what other kind is there?) Javascript noob (!! How hard can it be? Typescript, Uglify) Fortran deny all (literally don’t tell anyone) C++ dangerous (enough to be) C# not much (linux?) Wrote much UI and Server Architecture 

History Lesson

Happy Developers? Modern Programming Languages like Java and Python Thing Why we liked it? What Happened? 3G/4G Languages (Java, C#, Python, C++) Manage memory, Powerful APIs, UI toolkits We wrote lots of code IDEs (Xemacs, JCreator, Eclipse, IntelliJ, VisualStudio) It make our code more colourful and we could write more Frameworks (RCP, Struts, JSF, Django, ASP, etc. etc.) Do more stuff, Organise things We wrote lots of code and it was more efficient Incremental Programs (OSGi, Jigsaw, etc.) Our massive programs still worked We wrote lots of code and it didn’t all have to fit in memory!! Happy Developers? Modern Programming Languages like Java and Python Developers became happy They write more code

Introducing the Monolith Single Tier Software Application Horizontal Approach to Features Modularity is Code-Based Examples: Scientific Desktop Application Some Types of Server Application Thick client Multiple web page single server

Some problems I have hit with… Monolithic, or nearly monolithic, software Developers working horizontally, less efficiently Encapsulation of modules – entropy increases Multiple APIs doing the same task CORBA/RMI/JMS Swing/SWT/JavaFX DOM+SAX/Xpath/Castor/Gson/Jackson Ability to scale in various directions (scale cube) And… Deployment / Testing / Understanding / Agility Despite all this Powerful, Well used and understood, Profitable

Some I think which can be okay with… Monolithic, or nearly monolithic, software Testing Well defined interactions Limited set of interoperable features OSGi Services – one monolith multiple services Developer ownership, interoperability Cost

Summary We have lots and lots and lots of code We find it hard to change it Customers/Users get releases infrequently more defects lower scale solutions …depressed

Service Oriented Architecture (SOA) Application is a collection of loosely coupled services One service Logically represents a business activity with a specified outcome. Self-contained. black box for its consumers. May consist of other underlying services Idea been around in one form or another since early 2000’s µServices are one type of SOA

µServices Fine grained and light weight Modularity is enforced, each component easy to understand, develop and test Small autonomous development teams Vertical development rather than horizontal Less for one developer to deal with / understand / have to compile IMPORTANT Services can scale independently and on the fly Technology choices weakly linked and easy to change

Team structure Product Management (exactly 1) Development Lead (exactly 1) Developers (1-10) (UI, µservice, database) Testing (0-3) UX (As needed) DevOps (1) Architect (As needed)

This is all very well but… How do we apply it to scientific algorithms? How will they scale? Different algorithms have different requirements Scale out in the cloud What kind of cloud? We are looking at DC/OS, Kubernetes and native: AWS, Azure, Google Cloud etc. What about Big Data?

Case Study – Automatic Lithostratigraphy Python Research Code Machine learning Parallelizable algorithm each python process can work on one part of the problem prediction Java µService using JSON-REST and Kafka deployed in a Jetty Server to a docker container Large amounts of data but highly parallel each run being only O(10)xO(100,000), for instance 9 1D datasets of size 25,000. Scale in: Y - (web service front end and data splitter) + (analysis) X - Many (analysis) processes Z – No need to scale

Standard Python Project Containing R&D Code Module for exposing running one facet of the code by Kafka message (competitive consumer) Jenkins build runs the tests and deploys the python consumer to the cluster as a docker image A readme / getting started and other docs are contained in the project

Python client µservice code using JSON-REST Spring Boot, etc Self contained set of unit tests and resources The build is deployed to docker image for running on the cluster Separate build which Downloads dependencies for developer Builds on Jenkins a ‘fat’ jar Deploys to DCOS A readme / getting started and other docs are contained in the project

Artifactory and Aggregation Binary artifacts can be published and consumed = modularity old skool E.g. IO packages µservice design patterns E.g. Aggregator 20ms 20ms 20ms Aggregator 20ms 2ms 2ms 2ms 2ms 2ms = 62ms = 28ms

week6-alith-zoom.png1 Results of autolith in plotly (one prediction)