GeoScience and µServices

Slides:



Advertisements
Similar presentations
Introduction to .NET Framework
Advertisements

July 23 th, 2005 Software Architecture in Practice RiSE’s Seminars Bass’s et al. Book :: Chapter 16 Fred Durão.
DNN LOVES JENKINS FOR CONTINUOUS INTEGRATION
CSCI 224 Introduction to Java Programming. Course Objectives  Learn the Java programming language: Syntax, Idioms Patterns, Styles  Become comfortable.
@2011 Mihail L. Sichitiu1 Android Introduction Platform Overview.
Cloud Computing & Amazon Web Services – EC2 Arpita Patel Software Engineer.
SOFTWARE DESIGN AND ARCHITECTURE LECTURE 09. Review Introduction to architectural styles Distributed architectures – Client Server Architecture – Multi-tier.
1 Geospatial and Business Intelligence Jean-Sébastien Turcotte Executive VP San Francisco - April 2007 Streamlining web mapping applications.
WEP Presentation for non-IT Steps and roles in software development 2. Skills developed in 1 st year 3. What can do a student in 1 st internship.
Distributed Information Systems. Motivation ● To understand the problems that Web services try to solve it is helpful to understand how distributed information.
Artifact Management Managing Maven and other artifacts in Perforce.
ASP.NET (Active Server Page) SNU OOPSLA Lab. October 2005.
Software Design 1.1 CPS 108, Spring 2006 l Object oriented programming and design, we'll use Java and C++  Language independent concepts including design.
Basic Java training Harinath Mallepally
 Can access all API’s made available by OS vendor.  SDK’s are platform-specific.  Each mobile OS comes with its own unique tools and GUI toolkit.
Portlet Development Konrad Rokicki (SAIC) Manav Kher (SemanticBits) Joshua Phillips (SemanticBits) Arch/VCDE F2F November 28, 2008.
Common Client Rich Client Platforms
Computer System Structures
Ramping Up On The SharePoint Framework (SPFx)
Platform as a Service (PaaS)
WebSphere Diego Leone.
“OSGi at the UK’s Biggest Science Lab.”
Joonas Sirén, Technology Architect, Emerging Technologies Accenture
DevOps Cloud Native Microservices
Containers as a Service with Docker to Extend an Open Platform
Scalable containers with Apache Mesos and DC/OS
Platform as a Service (PaaS)
Platform as a Service (PaaS)
Recipes for Use With Thin Clients
OpenLegacy Training Day Four Introduction to Microservices
N-Tier Architecture.
Mobile App Development
Docker Birthday #3.
Production Line Architecture
Production Line Architecture
Platform as a Service.
On the road: Test automation in practice for a BMW map update service
CS101 Introduction to Computing Lecture 19 Programming Languages
Google Web Toolkit - Gufran Mohammed
Introduction to Microservices Prepared for
The Improvement of PaaS Platform ZENG Shu-Qing, Xu Jie-Bin 2010 First International Conference on Networking and Distributed Computing SQUARE.
Cloud Computing.
Python Classes in Pune |
Using docker containers
The Business Case for Containerization
Developing for the cloud with Visual Studio
Lecture 1: Multi-tier Architecture Overview
Web Application Architectures
D7032E – Software Engineering
Google App Engine Ying Zou 01/24/2016.
Service Oriented Architecture (SOA)
SAMANVITHA RAMAYANAM 18TH FEBRUARY 2010 CPE 691
Distributed Systems through Web Services
Web Application Architectures
MORE ON ARCHITECTURES The main reasons for using an architecture are maintainability and performance. We want to structure the software into reasonably.
Software Development at ORTEC
Developing and testing enterprise Java applications
Java History, Editions, Version Features
Introduction to SOA and Web Services
CSC 142 Introduction to Java [Reading: chapters 1 & 2]
TechEd /23/2019 9:23 AM © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks.
Introduction to Web Services and SOA
Web Application Architectures
Azure Container Service
UNGP Methods & Developer Services
Building highly scalable enterprise systems (on Azure)
DBOS DecisionBrain Optimization Server
Running C# in the browser
Logical Architecture & UML Package Diagrams
Presentation transcript:

GeoScience and µServices

Background Landmark would be one of the top 50 largest software companies Scientific and Engineering software for Energy sector Exploration Construction Production Neftex

What kind of developer are you? Experienced with Java (since 1.1 which is ~1998) Python hacker (what other kind is there?) Javascript noob (!! How hard can it be? Typescript, Uglify) Fortran deny all (literally don’t tell anyone) C++ dangerous (enough to be) C# not much (linux?) Wrote much UI and Server Architecture 

History Lesson

Happy Developers? Modern Programming Languages like Java and Python Thing Why we liked it? What Happened? 3G/4G Languages (Java, C#, Python, C++) Manage memory, Powerful APIs, UI toolkits We wrote lots of code IDEs (Xemacs, JCreator, Eclipse, IntelliJ, VisualStudio) It make our code more colourful and we could write more Frameworks (RCP, Struts, JSF, Django, ASP, etc. etc.) Do more stuff, Organise things We wrote lots of code and it was more efficient Incremental Programs (OSGi, Jigsaw, etc.) Our massive programs still worked We wrote lots of code and it didn’t all have to fit in memory!! Happy Developers? Modern Programming Languages like Java and Python Developers became happy They write more code

Introducing the Monolith Single Tier Software Application Horizontal Approach to Features Modularity is Code-Based Examples: Scientific Desktop Application Some Types of Server Application Thick client Multiple web page single server

Some problems I have hit with… Monolithic, or nearly monolithic, software Developers working horizontally, less efficiently Encapsulation of modules – entropy increases Multiple APIs doing the same task CORBA/RMI/JMS Swing/SWT/JavaFX DOM+SAX/Xpath/Castor/Gson/Jackson Ability to scale in various directions (scale cube) And… Deployment / Testing / Understanding / Agility Despite all this Powerful, Well used and understood, Profitable

Some I think which can be okay with… Monolithic, or nearly monolithic, software Testing Well defined interactions Limited set of interoperable features OSGi Services – one monolith multiple services Developer ownership, interoperability Cost

Summary We have lots and lots and lots of code We find it hard to change it Customers/Users get releases infrequently more defects lower scale solutions …depressed

Service Oriented Architecture (SOA) Application is a collection of loosely coupled services One service Logically represents a business activity with a specified outcome. Self-contained. black box for its consumers. May consist of other underlying services Idea been around in one form or another since early 2000’s µServices are one type of SOA

µServices Fine grained and light weight Modularity is enforced, each component easy to understand, develop and test Small autonomous development teams Vertical development rather than horizontal Less for one developer to deal with / understand / have to compile IMPORTANT Services can scale independently and on the fly Technology choices weakly linked and easy to change

Team structure Product Management (exactly 1) Development Lead (exactly 1) Developers (1-10) (UI, µservice, database) Testing (0-3) UX (As needed) DevOps (1) Architect (As needed)

This is all very well but… How do we apply it to scientific algorithms? How will they scale? Different algorithms have different requirements Scale out in the cloud What kind of cloud? We are looking at DC/OS, Kubernetes and native: AWS, Azure, Google Cloud etc. What about Big Data?

Case Study – Automatic Lithostratigraphy Python Research Code Machine learning Parallelizable algorithm each python process can work on one part of the problem prediction Java µService using JSON-REST and Kafka deployed in a Jetty Server to a docker container Large amounts of data but highly parallel each run being only O(10)xO(100,000), for instance 9 1D datasets of size 25,000. Scale in: Y - (web service front end and data splitter) + (analysis) X - Many (analysis) processes Z – No need to scale

Standard Python Project Containing R&D Code Module for exposing running one facet of the code by Kafka message (competitive consumer) Jenkins build runs the tests and deploys the python consumer to the cluster as a docker image A readme / getting started and other docs are contained in the project

Python client µservice code using JSON-REST Spring Boot, etc Self contained set of unit tests and resources The build is deployed to docker image for running on the cluster Separate build which Downloads dependencies for developer Builds on Jenkins a ‘fat’ jar Deploys to DCOS A readme / getting started and other docs are contained in the project

Artifactory and Aggregation Binary artifacts can be published and consumed = modularity old skool E.g. IO packages µservice design patterns E.g. Aggregator 20ms 20ms 20ms Aggregator 20ms 2ms 2ms 2ms 2ms 2ms = 62ms = 28ms

week6-alith-zoom.png1 Results of autolith in plotly (one prediction)