Download presentation
Presentation is loading. Please wait.
Published byVictor Egger Modified over 6 years ago
1
Capstone Data Analysis Services: Micro-services Early Implementation (Vision and Architecture)
Ramesh Baral Team: Marjani Peterson, Andre Guerrero Mentors: Eric Nienhouse, Seth McGinnis, Rich Loft TODOs -have more images, wherever possible Team: Marjani Peterson, Andre Guerrero Mentors: Eric Nienhouse, Seth McGinnis, Rich Loft September 17, 2018September 17, 2018
2
Vision Climate data used in research and planning Problems: Solution:
Complexity of climate data Requires expertise to work with Solution: Encapsulate expertise into re-usable modules Provision of an end-to-end server side climate data analytics system Lowers time to solution Climate data used in many societal contexts, research and decision making Many real-world scenarios, such as, infrastructure planning and investments need to analyze the weather and climate data It is difficult for scientists, resource managers, and concerned citizens to access and share the expert knowledge required for analyzing and drawing conclusions from weather and climate data. Despite its usefulness, the atmospheric data model is generally complex and is only accessible to the scientists and the experts in that area An end to end climate data analytics is required. Some of the existing tools required configuration and setup overheads. We would like to reduce those overheads RDL: changed access to work with. It’s not hard to access, more difficult to understand. Changes overhead to time to solution. Made image in lower right bigger.
3
Example Use Case A city manager needs to know the number of days the temperature of Austin, TX gets above 100°F, now and in the future. A city manager needs to know what the CMIP5 climate model results can tell her about how often the temperature gets above 100°F in Austin, now and in the future, and compare it to observational data. Traditionally, this task would require downloading gigabytes of data in an unfamiliar format and days of assistance from a specialist. RDL: added TX and punctuation.
4
Example Workflow Re-use
Experts Create workflow Set parameters Run workflow Store workflow Austin, TX ; Expert Creates Workflow Subsets data to local region Extracts temp and precip Averages fields across model suite Converts units Produces an output plot Stores workflow in repository Non-Specialist Re-Uses Workflow Discovers workflow in repository Changes location of interest Runs workflow and views result Non-experts Re-use workflow Fargo, ND;
5
Architecture RDL: I couldn’t change the text. I think that you should change “intermediate result” at center right to “intermediate results”
6
Detailed Architecture
RDL: changed Detail to Detailed in title.
7
Workflow Patterns[1] Sequential Parallel Simple merge Structured loop
B C A1 A2 B1 A1 C A B C [1]
8
Workflow Spec and Engine
Problems: Workflow specs coupled to the engine (e.g.: GirderWorker, Pinball, Celery) Workflow engines Client-side Non-generic Coupled with the execution scripts Solution: Simplicity UI to generate workflow spec Flexibility Make generic workflow spec Adaptor translates generic spec to selected engine RDL: eg -> e.g. as e.g. is an abbreviation.
9
Example Workflow Spec { "inputs": [ {"type": "string“, "name": “myinputstring", "format": "text", "val": “teststring"}, { "type" : "number", "name": “myinputnumber", "format": "number", "val":30.25} ], "connections": [ {"input": " myinputstring ", "input_step": “mystep1", "name“:" step_inputstring “}, {"input": " myinputnumber ", "input_step": “mystep1","name“:“step_inputnumber”} ] } TODO: make this more generic RDL: This could be a time sink to explain. if time is an issue consider deleting this slide.
10
Workflow Adapter Purpose:
Translate generic workflow spec to Workflow engine standard Validate the workflow: Workflow inputs, outputs Workflow connections’ inputs Workflow connections’ steps Translation includes the elaborated definition of steps including the injection of the script of tasks on run-time Workflow connections’ inputs should either be from the user inputs or from within the tasks Workflow outputs should be defined in both sections TODOs – make this more focused on the requirement, the challenge of the workflow engine’s spec requirement and why we needed to design the adapter
11
Microservices Independent functional units Individually accessible
Separate process, and deployment Different technology stack, and working team RESTful microservices for each task Distributed services across different VMs Can be in different programming languages Text, JSON input; JSON output Segregated the core script from microservices Accessible via web-interface and API Eg:
12
Conclusion and Future work
Microservice based climate data analytics Next: Deployment Automatic deployment Deployment on other cloud technologies (OpenStack) and other providers Refine UI for non-expert users drag and drop More microservices distributed across multiple VMs different technology stack (e.g.: Language) RDL: added other.
13
References http://www.workflowpatterns.com
14
Q&A ?
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.