1. GeoScience and µServices

2. Background
Landmark would be one of the top 50 largest software companies
Scientific and Engineering software for Energy sector
Exploration
Construction
Production
Neftex

4. What kind of developer are you?
Experienced with Java (since 1.1 which is ~1998)
Python hacker (what other kind is there?)
Javascript noob (!! How hard can it be? Typescript, Uglify)
Fortran deny all (literally don’t tell anyone)
C++ dangerous (enough to be)
C# not much (linux?)
Wrote much UI and Server Architecture 

5. History Lesson

6. Happy Developers? Modern Programming Languages like Java and Python
Thing
Why we liked it?
What Happened?
3G/4G
Languages (Java, C#, Python, C++)
Manage memory, Powerful APIs, UI toolkits
We wrote lots of code
IDEs
(Xemacs, JCreator, Eclipse, IntelliJ, VisualStudio)
It make our code more colourful and we could write more
Frameworks
(RCP, Struts, JSF, Django, ASP, etc. etc.)
Do more stuff,
Organise things
We wrote lots of code and it was more efficient
Incremental Programs
(OSGi, Jigsaw, etc.)
Our massive programs still worked
We wrote lots of code and it didn’t all have to fit in memory!!
Happy Developers?
Modern Programming Languages like Java and Python
Developers became happy
They write more code

7. Introducing the Monolith
Single Tier Software Application
Horizontal Approach to Features
Modularity is Code-Based
Examples:
Scientific Desktop Application
Some Types of Server Application
Thick client
Multiple web page single server

8. Some problems I have hit with…
Monolithic, or nearly monolithic, software
Developers working horizontally, less efficiently
Encapsulation of modules – entropy increases
Multiple APIs doing the same task
CORBA/RMI/JMS
Swing/SWT/JavaFX
DOM+SAX/Xpath/Castor/Gson/Jackson
Ability to scale in various directions (scale cube)
And… Deployment / Testing / Understanding / Agility
Despite all this
Powerful,
Well used and understood,
Profitable

9. Some I think which can be okay with…
Monolithic, or nearly monolithic, software
Testing
Well defined interactions
Limited set of interoperable features
OSGi Services – one monolith multiple services
Developer ownership, interoperability
Cost

10. Summary We have lots and lots and lots of code
We find it hard to change it
Customers/Users get
releases infrequently
more defects
lower scale solutions
…depressed

11. Service Oriented Architecture (SOA)
Application is a collection of loosely coupled services
One service
Logically represents a business activity with a specified outcome.
Self-contained.
black box for its consumers.
May consist of other underlying services
Idea been around in one form or another since early 2000’s
µServices are one type of SOA

12. µServices Fine grained and light weight
Modularity is enforced, each component easy to understand, develop and test
Small autonomous development teams
Vertical development rather than horizontal
Less for one developer to deal with / understand / have to compile
IMPORTANT Services can scale independently and on the fly
Technology choices weakly linked and easy to change

13. Team structure Product Management (exactly 1)
Development Lead (exactly 1)
Developers (1-10) (UI, µservice, database)
Testing (0-3)
UX (As needed)
DevOps (1)
Architect (As needed)

14. This is all very well but…
How do we apply it to scientific algorithms?
How will they scale?
Different algorithms have different requirements
Scale out in the cloud
What kind of cloud? We are looking at DC/OS, Kubernetes and native: AWS, Azure, Google Cloud etc.
What about Big Data?

16. Case Study – Automatic Lithostratigraphy
Python Research Code
Machine learning
Parallelizable algorithm each python process can work on one part of the problem prediction
Java µService using JSON-REST and Kafka deployed in a Jetty Server to a docker container
Large amounts of data but highly parallel each run being only O(10)xO(100,000), for instance 9 1D datasets of size 25,000.
Scale in:
Y - (web service front end and data splitter) + (analysis)
X - Many (analysis) processes
Z – No need to scale

18. Standard Python Project Containing R&D Code
Module for exposing running one facet of the code by Kafka message (competitive consumer)
Jenkins build runs the tests and deploys the python consumer to the cluster as a docker image
A readme / getting started and other docs are contained in the project

19. Python client
µservice code using JSON-REST Spring Boot, etc
Self contained set of unit tests and resources
The build is deployed to docker image for running on the cluster
Separate build which
Downloads dependencies for developer
Builds on Jenkins a ‘fat’ jar
Deploys to DCOS
A readme / getting started and other docs are contained in the project

20. Artifactory and Aggregation
Binary artifacts can be published and consumed = modularity old skool
E.g. IO packages
µservice design patterns
E.g. Aggregator
20ms
20ms
20ms
Aggregator
20ms
2ms
2ms
2ms
2ms
2ms
= 62ms
= 28ms

21. week6-alith-zoom.png1
Results of autolith in plotly (one prediction)