1. Capstone Data Analysis Services: Micro-services Early Implementation (Vision and Architecture)
Ramesh Baral
Team: Marjani Peterson, Andre Guerrero
Mentors: Eric Nienhouse, Seth McGinnis, Rich Loft
TODOs
-have more images, wherever possible
Team: Marjani Peterson, Andre Guerrero
Mentors: Eric Nienhouse, Seth McGinnis, Rich Loft
September 17, 2018September 17, 2018

2. Vision Climate data used in research and planning Problems: Solution:
Complexity of climate data
Requires expertise to work with
Solution:
Encapsulate expertise into re-usable modules
Provision of an end-to-end server side climate data analytics system
Lowers time to solution
Climate data used in many societal contexts, research and decision making
Many real-world scenarios, such as, infrastructure planning and investments need to analyze the weather and climate data
It is difficult for scientists, resource managers, and concerned citizens to access and share the expert knowledge required for analyzing and drawing conclusions from weather and climate data.
Despite its usefulness, the atmospheric data model is generally complex and is only accessible to the scientists and the experts in that area
An end to end climate data analytics is required. Some of the existing tools required configuration and setup overheads. We would like to reduce those overheads
RDL: changed access to work with. It’s not hard to access, more difficult to understand. Changes overhead to time to solution. Made image in lower right bigger.

3. Example Use Case
A city manager needs to know the number of days the temperature of Austin, TX gets above 100°F, now and in the future.
A city manager needs to know what the CMIP5 climate model results can tell her about how often the temperature gets above 100°F in Austin, now and in the future, and compare it to observational data.
Traditionally, this task would require downloading gigabytes of data in an unfamiliar format and days of assistance from a specialist.
RDL: added TX and punctuation.

4. Example Workflow Re-use
Experts
Create workflow
Set parameters
Run workflow
Store workflow
Austin, TX ;
Expert Creates Workflow
Subsets data to local region
Extracts temp and precip
Averages fields across model suite
Converts units
Produces an output plot
Stores workflow in repository
Non-Specialist Re-Uses Workflow
Discovers workflow in repository
Changes location of interest
Runs workflow and views result
Non-experts
Re-use workflow
Fargo, ND;

5. Architecture
RDL: I couldn’t change the text. I think that you should change “intermediate result” at center right to “intermediate results”

6. Detailed Architecture
RDL: changed Detail to Detailed in title.

7. Workflow Patterns[1] Sequential Parallel Simple merge Structured loopB C A1 A2 B1 A1 C A B C
[1]

8. Workflow Spec and Engine
Problems:
Workflow specs coupled to the engine (e.g.: GirderWorker, Pinball, Celery)
Workflow engines
Client-side
Non-generic
Coupled with the execution scripts
Solution:
Simplicity
UI to generate workflow spec
Flexibility
Make generic workflow spec
Adaptor translates generic spec to selected engine
RDL: eg -> e.g. as e.g. is an abbreviation.

9. Example Workflow Spec
{ "inputs": [ {"type": "string“, "name": “myinputstring", "format": "text", "val": “teststring"}, { "type" : "number", "name": “myinputnumber", "format": "number", "val":30.25} ], "connections": [ {"input": " myinputstring ", "input_step": “mystep1", "name“:" step_inputstring “}, {"input": " myinputnumber ", "input_step": “mystep1","name“:“step_inputnumber”} ] }
TODO: make this more generic
RDL: This could be a time sink to explain. if time is an issue consider deleting this slide.

10. Workflow Adapter Purpose:
Translate generic workflow spec to Workflow engine standard
Validate the workflow:
Workflow inputs, outputs
Workflow connections’ inputs
Workflow connections’ steps
Translation includes the elaborated definition of steps including the injection of the script of tasks on run-time
Workflow connections’ inputs should either be from the user inputs or from within the tasks
Workflow outputs should be defined in both sections
TODOs – make this more focused on the requirement, the challenge of the workflow engine’s spec requirement and why we needed to design the adapter

11. Microservices Independent functional units Individually accessible
Separate process, and deployment
Different technology stack, and working team
RESTful microservices for each task
Distributed services across different VMs
Can be in different programming languages
Text, JSON input; JSON output
Segregated the core script from microservices
Accessible via web-interface and API
Eg:

12. Conclusion and Future work
Microservice based climate data analytics
Next:
Deployment
Automatic deployment
Deployment on other cloud technologies (OpenStack) and other providers
Refine UI for non-expert users
drag and drop
More microservices
distributed across multiple VMs
different technology stack (e.g.: Language)
RDL: added other.

13. References http://www.workflowpatterns.com

14. Q&A ?