Download presentation
Presentation is loading. Please wait.
Published bySamuel Gray Modified over 9 years ago
1
Rochester Institute of Technology Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure Gregor von Laszewski, Andrew J. Younge, Xi He, and Fugang Wang Service Oriented Cyberinfrastructure Laboratory Rochester Institute of Technology Bldg 74, Lomb Memorial Drive Rochester, NY 14523-5608 11/9/2015http://cyberaide.org1
2
Agenda Introduction Related Research Problem Proposed Solution Design Implementation Use Case Conclusion 11/9/2015 http://cyberaide.org 2
3
Introduction What is Grid Computing? “Grid computing is a complex and diverse field where different technologies are constructed and combined to enable the use of distributed resources under administratively separate domains.” 11/9/2015http://cyberaide.org3
4
Grid Hierarchy 11/9/2015 http://cyberaide.org 4 Grid Fabric Middleware Upperware Condor PBS SGE LSF Globus TeraGrid OSG G-Lite EGEE BOINC Seti@Home CoG Kit Cyberaide Project Web Portals Science Gateways
5
Grid Fabric (Clusters) 11/9/2015http://cyberaide.org5 LSF PBS SGE Condor All clusters are batch queuing systems
6
Middleware Middleware combines multiple clusters together. – Administratively separate systems – Large dedicated clusters Globus Toolkit – Most popular of all Grid middleware technologies Developed at Argonne National Laboratory GT4 uses advanced WSRF Web Services G-Lite – Developed by the EGEE project – Used in the Large Hadron Collider experiments 11/9/2015http://cyberaide.org6
7
The TeraGrid NSF-funded national-scale Grid Infrastructure 11 Locations – LONI, NCAR, NCSA, NICS, ORNL, PSC, IU, PU, SDSC, TACC, UC/ANL 1.1Petaflops, 161 thousand CPUs, 60 Petabytes disk space Dedicated 10G fiber lines to each location Specialized visualization servers Uses Globus Toolkit 4’s basic WS services and security protocols Grid Infrastructure Group (GIG) at U. Chicago Commity for Teragrid planning, management, and coordination Science Gateways Independent services for specialized groups and organizations “TeraGrid Inside” capabilities Web Portals, desktop apps, coordinated access points Not Virtual Organizations (VOs) 11/9/20157http://cyberaide.org
8
TeraGrid View SDSC TACC UC/ANL NCSA ORNL PU IU PSC NCAR Caltech UNC/RENCI UW Resource Provider (RP) Software Integration Partner Grid Infrastructur e Group (UChicago) LONI NICS 11/9/20158http://cyberaide.org
9
TeraGrid User Portal11/9/20159http://cyberaide.org
10
Introduction What is a System Shell? “A system shell is a piece of software that provides a text based interface which abstracts complex services for users.” 11/9/2015http://cyberaide.org10
11
Shell environments Shells are used to abstract complex services Typically used in Operating Systems Hide the complexity of the underlying kernel and support services Provide a unified user environment Command-Line Interface – To many power users, this is preferred – More function and control available Shells can be scripted to automate common high-level tasks. 11/9/2015http://cyberaide.org11
12
Shells used System Shells shkshbashzshcshrc Distributed Shells PdshPsshJXTA Aspect- Shell Language Shells Python CLI iPythonIRBBeanShellGroovysh System shells are used as part of an OS to provide a CLI for users Distributed Shells execute tasks similar to system shells, however in Parallel Language Shells interprets users’s code and commands during runtime System shells are used as part of an OS to provide a CLI for users Distributed Shells execute tasks similar to system shells, however in Parallel Language Shells interprets users’s code and commands during runtime 11/9/2015http://cyberaide.org12
13
Problem A steep learning curve and high entry barrier limit the use of Grid computing within the scientific community. – Parallel programming is difficult – Using middleware technologies can be cumbersome and inefficient – Users need to be advanced programmers just to use the technology How do we simplify the Cyberinfrastructure environment for scientists? 11/9/2015http://cyberaide.org13
14
Requirements Simple, intuitive, and user friendly Unifying framework Extensible Interactive Fault tolerant Event management Support batch processing Integrates new Cyberinfrastructure 11/9/2015http://cyberaide.org14
15
Rochester Institute of Technology What if we combined Grids with Shells? Performance + Usability (have your cake and eat it too) 11/9/2015http://cyberaide.org15
16
Cyberaide Shell Simple to use system shell which provides access to the powerful Grids and advanced Cyberinfrastructure that is available today However, “This isn’t your father’s UNIX shell…” 11/9/2015http://cyberaide.org16
17
High Level Design11/9/2015http://cyberaide.org17
18
Architectural Design 11/9/2015http://cyberaide.org18
19
Implementation – Set – Submit – Person – Man – List – Experiment – Resources – Tasks – List – Experiment – Resources – Tasks 11/9/2015http://cyberaide.org19 Prototype implementation Demonstrates our design’s feasibility Uses Globus GT2 and GT4 technologies for backend services Supports basic commands
20
Command List 11/9/2015http://cyberaide.org20
21
Nested Shells Cyberaide Shell supports the idea of Nested Shells – “shell-within-a-shell” – Each command has its own interpreter which interrogates options dynamically – More efficient scripts Two Ways: 11/9/2015http://cyberaide.org21
22
Literate Semantic Objects (LSOs) New data management concept LSOs store attribute-value pairs – simple! Mutable during runtime Object inheritance – All objects inherent form this object Can be recorded in many different formats – XML, JSON, CSV – Converted to SQL Provide semantic information to the shell during runtime 11/9/2015http://cyberaide.org22
23
Extending Cyberaide Shell Creating a Service Oriented Architecture Expose shell a Web Service – Web Services Interoperability standar – WS-Security for endpoint to endpoint security with X.509 certificates – Uses Apache CXF and Jetty HTTP server to host Web service Various Client API’s possible Using Web service, its possible to build a Web Portal on top of the shell! 11/9/2015http://cyberaide.org23
24
Cyberaide Portal11/9/2015http://cyberaide.org24
25
Use Case11/9/2015http://cyberaide.org25
26
Future Work Implement other Cyberinfrastructure services – EGEE project and the Open Science Grid – Amazon Elastic Compute Cloud (EC2) Improve scheduling and workflow systems Improving integration with the Portal Develop language specific APIs 11/9/2015http://cyberaide.org26
27
Conclusion Cyberaide Shell helps scientists overcome the challenges faced when using advanced Cyberinfrastructure Combines the power of Grids with the usability of shells Enhances previous work in system shells with the use of semantic commands and nested commands Provides the shell as a Web service to allow for distributed usage across multiple platforms 11/9/2015http://cyberaide.org27
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.