September 10, 2015 XSEDE and OSG Kim Dillman Alain Roy

Slides:



Advertisements
Similar presentations
3rd Campus Grid SIG Meeting. Agenda Welcome OMII Requirements document Grid Data Group HTC Workshop Research Computing SIG? AOB Next meeting (AG)
Advertisements

Xsede eXtreme Science and Engineering Discovery Environment Ron Perrott University of Oxford 1.
1 US activities and strategy :NSF Ron Perrott. 2 TeraGrid An instrument that delivers high-end IT resources/services –a computational facility – over.
Joint CASC/CCI Workshop Report Strategic and Tactical Recommendations EDUCAUSE Campus Cyberinfrastructure Working Group Coalition for Academic Scientific.
Bosco: Enabling Researchers to Expand Their HTC Resources The Bosco Team: Dan Fraser, Jaime Frey, Brooklin Gore, Marco Mambelli, Alain Roy, Todd Tannenbaum,
Session 5 Intellectual Merit and Broader Significance FISH 521.
Background Chronopolis Goals Data Grid supporting a Long-term Preservation Service Data Migration Data Migration to next generation technologies Trust.
Dan Stanzione, Training lead CASC Meeting
Campus High Throughput Computing (HTC) Infrastructures (aka Campus Grids) Dan Fraser OSG Production Coordinator Campus Grids Lead.
Technology on the NGS Pete Oliver NGS Operations Manager.
National Science Foundation: Transforming Undergraduate Education in Science, Technology, Engineering, and Mathematics (TUES)
April 2009 OSG Grid School - RDU 1 Open Science Grid John McGee – Renaissance Computing Institute University of North Carolina, Chapel.
1 The eXtremeDigital (XD) Program Barry I. Schneider Office of Cyberinfrastructure National Science Foundation
1 Building National Cyberinfrastructure Alan Blatecky Office of Cyberinfrastructure EPSCoR Meeting May 21,
SCD FIFE Workshop - GlideinWMS Overview GlideinWMS Overview FIFE Workshop (June 04, 2013) - Parag Mhashilkar Why GlideinWMS? GlideinWMS Architecture Summary.
Ways to Connect to OSG Tuesday afternoon, 3:00 pm Lauren Michael Research Computing Facilitator University of Wisconsin-Madison.
PYs 4 and 5 Objectives for Education, Outreach and Training and External Relations Scott Lathrop
Key Project Drivers - FY11 Ruth Pordes, June 15th 2010.
Open Science Grid For CI-Days Internet2: Fall Member Meeting, 2007 John McGee – OSG Engagement Manager Renaissance Computing Institute.
Effective User Services for High Performance Computing A White Paper by the TeraGrid Science Advisory Board May 2009.
CI Days: Planning Your Campus Cyberinfrastructure Strategy Russ Hobby, Internet2 Internet2 Member Meeting 9 October 2007.
PCGRID ‘08 Workshop, Miami, FL April 18, 2008 Preston Smith Implementing an Industrial-Strength Academic Cyberinfrastructure at Purdue University.
Open Science Grid For CI-Days Elizabeth City State University Jan-2008 John McGee – OSG Engagement Manager Manager, Cyberinfrastructure.
SG - OSG Improving Campus Research CI Through Leveraging and Integration: Developing a SURAgrid-OSG Collaboration John McGee, RENCI/OSG Engagement Coordinator.
What is Cyberinfrastructure? Russ Hobby, Internet2 Clemson University CI Days 20 May 2008.
Miron Livny Center for High Throughput Computing Computer Sciences Department University of Wisconsin-Madison Open Science Grid (OSG)
October 21, 2015 XSEDE Technology Insertion Service Identifying and Evaluating the Next Generation of Cyberinfrastructure Software for Science Tim Cockerill.
Discussion Topics DOE Program Managers and OSG Executive Team 2 nd June 2011 Associate Executive Director Currently planning for FY12 XD XSEDE Starting.
Enabling Science Through Campus Bridging A case study with mlRho Scott Michael July 24, 2013.
10/24/2015OSG at CANS1 Open Science Grid Ruth Pordes Fermilab
Apache Airavata (Incubating) Gateway to Grids & Clouds Suresh Marru Nov 10 th 2011.
Russ Hobby Program Manager Internet2 Cyberinfrastructure Architect UC Davis.
Interoperability Grids, Clouds and Collaboratories Ruth Pordes Executive Director Open Science Grid, Fermilab.
Grid Middleware Tutorial / Grid Technologies IntroSlide 1 /14 Grid Technologies Intro Ivan Degtyarenko ivan.degtyarenko dog csc dot fi CSC – The Finnish.
Authors: Ronnie Julio Cole David
Cyberinfrastructure What is it? Russ Hobby Internet2 Joint Techs, 18 July 2007.
Ruth Pordes November 2004TeraGrid GIG Site Review1 TeraGrid and Open Science Grid Ruth Pordes, Fermilab representing the Open Science.
Joint Meeting of the AUS, US, XS Working Groups TG10 Tuesday August 3, hrs Elwood II.
Leveraging the InCommon Federation to access the NSF TeraGrid Jim Basney Senior Research Scientist National Center for Supercomputing Applications University.
GLIDEINWMS - PARAG MHASHILKAR Department Meeting, August 07, 2013.
1 NSF/TeraGrid Science Advisory Board Meeting July 19-20, San Diego, CA Brief TeraGrid Overview and Expectations of Science Advisory Board John Towns TeraGrid.
Cyberinfrastructure Overview Russ Hobby, Internet2 ECSU CI Days 4 January 2008.
Cyberinfrastructure: Many Things to Many People Russ Hobby Program Manager Internet2.
January 20, 2016 XSEDE at a Glance Aaron Gardner Campus Champion - University of Florida.
Fire Emissions Network Sept. 4, 2002 A white paper for the development of a NSF Digital Government Program proposal Stefan Falke Washington University.
Education, Outreach and Training (EOT) and External Relations (ER) Scott Lathrop Area Director for EOT Extension Year Plans.
Open Science Grid as XSEDE Service Provider Open Science Grid as XSEDE Service Provider December 4, 2011 Chander Sehgal OSG User Support.
Forging the eXtremeDigital (XD) Program Barry I. Schneider Program Director, Office of CyberInfrastructure January 20, 2011.
PEER 2003 Meeting 03/08/031 Interdisciplinary Framework Major focus areas Structural Representation Fault Systems Earthquake Source Physics Ground Motions.
All Hands Meeting 2005 BIRN-CC: Building, Maintaining and Maturing a National Information Infrastructure to Enable and Advance Biomedical Research.
1 Open Science Grid: Project Statement & Vision Transform compute and data intensive science through a cross- domain self-managed national distributed.
Building on virtualization capabilities for ExTENCI Carol Song and Preston Smith Rosen Center for Advanced Computing Purdue University ExTENCI Kickoff.
March 2014 Open Science Grid Operations A Decade of HTC Infrastructure Support Kyle Gross Operations Support Lead Indiana University / Research Technologies.
TeraGrid’s Process for Meeting User Needs. Jay Boisseau, Texas Advanced Computing Center Dennis Gannon, Indiana University Ralph Roskies, University of.
INTRODUCTION TO XSEDE. INTRODUCTION  Extreme Science and Engineering Discovery Environment (XSEDE)  “most advanced, powerful, and robust collection.
EGI-InSPIRE RI EGI Compute and Data Services for Open Access in H2020 Tiziana Ferrari Technical Director, EGI.eu
What is OSG? (What does it have to do with Atlas T3s?) What is OSG? (What does it have to do with Atlas T3s?) Dan Fraser OSG Production Coordinator OSG.
Ways to Connect to OSG Tuesday, Wrap-Up Lauren Michael, CHTC.
EPSCoR means “Experimental Program to Stimulate Competitive Research
Accessing the VI-SEEM infrastructure
XSEDE Value Added and Financial Economies
Clouds , Grids and Clusters
Joslynn Lee – Data Science Educator
WBS ESTEO Mission Extended Collaborative Support for Training, Education and Outreach prepares the current and next generation of researchers, engineers,
Recap: introduction to e-science
Introduction to XSEDE Resources HPC Workshop 08/21/2017
State of XSEDE: XSEDE14 John Towns PI and Project Director, XSEDE
Bringing HPC to Your Campus
Cyberinfrastructure for the Life Sciences
Introduction to Free HPC Resources: XSEDE
Presentation transcript:

September 10, 2015 XSEDE and OSG Kim Dillman Alain Roy

What is OSG? A multi-disciplinary partnership to federate local, regional, community and national cyberinfrastructures to help share computing and storage resources of research and academic communities at all scales We provide common services and support for more than 100 resource providers and scientific institutions using a distributed fabric of high-throughput computational services We do not own computing resources but instead provide software and services to users and resource providers to enable the effective use and sharing of their resources based on the principles of Distributed High Throughput Computer (DHTC) 2

OSG’s Community Focus 3

Where is OSG? 4

How big is OSG? From 5

6

Who uses OSG? 7

Who contributes to OSG? 8

Do you have a problem that needs HTC? Do you have local computing resources? – Use them first – You get local support – You gain experience with HTC When you need more resources, OSG is a good option to grow 9 How do you use OSG? Step 1: Make sure you can use HTC

How do you use OSG? Step 2: You must be in a Virtual Organization (VO) A VO is: – A collection of people and/or resources with a common purpose – Usually a scientific collaboration People + their computing & storage Which VO? – About 60 VOs in OSG: you might already be affiliated – Make your own for your collaboration – Join the “OSG” VO: miscellaneous 10

OSG runs a glidein factory, but you need a submission point (i.e. condor_schedd) Called a VO Frontend (see Igor’s talk) Larger VOs set up their own frontend Smaller VOs can use an OSG supplied frontend – We’re in a transition period right now – This will probably be hosted at UCSD in the near future Sites need to authorize your VO 11 How do you use OSG? Step 3: Get access to glidein VO Frontend

How do you contribute to OSG? Usually contribution is based on VO needs – Scientific collaboration wants to share within the collaboration: OSG provides the tools – Other VOs can utilize unused capacity Some sites are simply generous and share Install and maintain the OSG software on: – Front end job submission computer – Storage interface computer (optional) – Authorization service (optional) Everyone is welcome, but it takes effort to do it (25% of a person’s effort?) 12

Autonomy OSG provides: – Software – Support – Central Services Sites and VOs are autonomous: – They make decisions about their sites – They decide when to install, upgrade – They make operational decisions This is HTC: if we mandate less, more people will join and we will have access to more resources 13

What is XSEDE? 14 XSEDE is a comprehensive set of advanced, heterogeneous high-end digital services, integrated into a general-purpose infrastructure.

NSF eXtreme Digital (XD) Program NSF’s Transition from TeraGrid to XD High-Performance Computing and Storage Services High-Performance Remote Visualization and Data Analysis Services eXtreme Digital Resources for Science and Engineering (XSEDE)

16

XSEDE HPC Systems 17 (From: portal.xsede.org)

XSEDE Visualization Systems 18

XSEDE Storage Systems 19

XSEDE: HTC Systems 20

XSEDE Vision The eXtreme Science and Engineering Discovery Environment (XSEDE): enhances the productivity of scientists and engineers by providing them with new and innovative capabilities. Thus, XSEDE: facilitates scientific discovery while enabling transformational science and engineering, and innovative educational programs. XSEDE fulfills this vision by creating an advanced, capable, and robust cyberinfrastructure supported by the combined expertise of a distributed team of leading CI (cyberinfrastructure) professionals.

XSEDE Characteristics: XSEDE forms the foundation of a national cyberinfrastructure (CI) ecosystem – Its comprehensive suite of advanced digital services work with other high-end facilities and campus- based resources XSEDE integrates diverse digital resources – Its open architecture allows continued addition of new technology capabilities and services

XSEDE is about.... Increasing productivity – leading to more science – making the difference between a feasible project and an impractical one Transformative impact through – active, formal requirements gathering processes to understand the needs of the community – new and expanded extended collaborative support that includes external short-term contracting for expertise beyond the current team Novel and Innovative Projects: supports novel science areas, demographic diversity, innovative technologies, science gateway development, data repositories, and campus bridging 23

And… – National Training and Education and Outreach programs with the scope and scale to: increase diversity of topics, modes of delivery, and reach to new communities and audiences broaden participation among under-represented communities campus bridging for effective use of CI (cyberinfrastructure) resources – integrate with campuses through expanded Champions program and additional bridging activities establish academic certificate and degree programs – institutional incorporation of CS&E curricula; professional development certificate – prepare undergraduates, graduates and future K-12 teachers 24

XSEDE Components Coordination and Management Service Technology Audit and Insertion Service Extended Collaborative Support Service Training, Education and Outreach Service 25

Best of all you can use it for FREE!

Who Uses XSEDE? 27 CategoryUsers Graduate Student2,368 Faculty1,336 Postdoctoral1,008 University Research Staff506 Undergraduate Student505 High school5 Others328 Quarterly XSEDE user, allocation, and usage summary by field of science, in order by usage, excluding staff projects. Note: PIs, users may appear under more than one field of science. End of quarter XSEDE open user accounts by type, excluding XSEDE staff.

Science Highlights 1: PI: Markus Buehler Institution: MIT “We found that the structure of spider silk at the nanoscale can explain why this material is as strong as steel, even though the “glue” of the hydrogen bonds holding spider silk together at the molecular level is 100 to 1,000 times weaker than steel’s metallic bonds.” says Buehler. 28 Excerpts from “TeraGrid Science Highlights 2010”

Science Highlights 2: PI: Michael Deem, David Earl Institution: Rice University, University of Pittsburgh Identified millions of potentially new zeolites by searching computationally for properly configured, hypothetically stable structures. Zeolites are used to make everything from gasoline and asphalt to aquarium filters, laundry detergent and medical-grade oxygen. 29 Excerpts from “TeraGrid Science Highlights 2010”

Science Highlights 3: PI: Sorin Matei, David Braun Institution: Purdue University Purdue researchers led by Sorin Adam Matei are analyzing the entire collection of articles produced in Wikipedia from , and all their revisions – a computationally demanding task made possible by TeraGrid resources. “We looked at how article production is distributed across users’ contributions relative to each other over time. The work includes visualizations of patterns to make them easier to discern,” says Matei. 30 Excerpts from “TeraGrid Science Highlights 2010”

How do you use XSEDE? Step 1: The basics Setup an XSEDE User Portal Account – – You can do it right now: it’s free and easy Review available resources and determine which will match the needs of your research – Resources -> Systems Monitor 31

How do you use XSEDE? Step 2: Startup Allocation Submit a request (anytime) Easy to get—lightly reviewed Small: 200K hours Use to investigate the resources where your code will run Use to benchmarking your code to determine the best resource and how many SUs you will need to complete your research 32

How do you use XSEDE? Step 3: Research Allocation Write and submit a full research allocation request Can be submitted quarterly Peer-reviewed Can be very large You should know exactly what resources you need 33

How do you use XSEDE? Step 4: Use Log into individual resources – Can log in via User Portal (built in ssh + authentication) – Can download application to login (ssh + authentication) – Can request direct ssh access from individual sites When you run out: – You will be unable to submit more jobs – It is possible to request extensions to your allocation 34

Differences between XSEDE and OSG Authentication – OSG : certificates obtained by user, member of VO – XSEDE : User has certificate but never sees it. Transparent access via MyProxy with name/password System Access: – OSG: Access based on VO membership Jobs submitted remotely (usually glideins) – XSEDE Access based on allocations Direct login to resource to submit jobs (via User Portal or gsissh or ssh) 35

Differences (2) Amount of Compute Cycles: – OSG : only limited by excess cycles on available resources – XSEDE : based on granted allocations (different amounts available for request based on allocation type with research allocation requests peer reviewed) Job Types: – OSG: serial or single-node parallel – XSEDE: serial, shared memory, large parallel, large data Other: – XSEDE: Can request an allocation of “people” resources (ECSS) to assist with technical aspects of project/code – OSG: Can access all systems via a single submit host 36

Questions? Comments? 37