Updates on HPC and Data management at NCI

Slides:



Advertisements
Similar presentations
SCAR DIMS Implementation 31st July 2010 Kim Finney (Manager, Australian Antarctic Data Centre & Chief Officer, SCAR Standing Committee on Antarctic Data.
Advertisements

Understanding and Implementing WMO INFORMATION SYSTEM
Integrating NOAA’s Unified Access Framework in GEOSS: Making Earth Observation data easier to access and use Matt Austin NOAA Technology Planning and Integration.
VO Sandpit, November 2009 NERC Big Data And what’s in it for NCEO? June 2014 Victoria Bennett CEDA (Centre for Environmental Data Archival)
CLIMATE SCIENTISTS’ BIG CHALLENGE: REPRODUCIBILITY USING BIG DATA Kyo Lee, Chris Mattmann, and RCMES team Jet Propulsion Laboratory (JPL), Caltech.
Large-Scale Data Collection Metadata Management at the National Computational Infrastructure (NCI) Jingbo Wang 1, Irina Bastrakova.
Test Dev UR2 Improved Console Usability Window Maximization problems Memory leaks Rendering issues UR3 Improved Console Usability Improved.
Z EGU Integration of external metadata into the Earth System Grid Federation (ESGF) K. Berger 1, G. Levavasseur 2, M. Stockhause 1, and M. Lautenschlager.
About CUAHSI The Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) is an organization representing 120+ universities.
Jisc Data Spring Pitch: Cloud Workbench Ben Butchart EDINA.
© 2013 National Ecological Observatory Network, Inc. ALL RIGHTS RESERVED. THE NEON APPROACH TO DATA INGEST, CURATION, AND SHARING Christine Laney (Data.
EC Grant Agreement no GEOSS Interoperability for Weather Ocean and Water Enhancing the GEOSS Infrastructure for all the Stakeholders.
Climate Sciences: Use Case and Vision Summary Philip Kershaw CEDA, RAL Space, STFC.
NE II NOAA Environmental Software Infrastructure and Interoperability Program Cecelia DeLuca Sylvia Murphy V. Balaji GO-ESSP August 13, 2009 Germany NE.
AN ORGANISATION FOR A NATIONAL EARTH SCIENCE INFRASTRUCTURE PROGRAM Virtual Geophysics Laboratory (VGL): Scientific workflows Exploiting the Cloud Josh.
Neil Witheridge APAN29 Sydney February 2010 ARCS Authorisation Services Neil Witheridge Manager, ARCS Authorisation Services APAN29, Sydney, February 2010.
Digital Earth Communities GEOSS Interoperability for Weather Ocean and Water GEOSS Common Infrastructure Evolution Roberto Cossu ESA
Integrated Model Data Management S.Hankin ESMF July ‘04 Integrated data management in the ESMF (ESME) Steve Hankin (NOAA/PMEL & IOOS/DMAC) ESMF Team meeting.
Data Citation & Digital Object Identifiers DOIs. 2 DOIs for articles mints DOIs for Journal articles and some datasets.
THEME[ENV ]: Inter-operable integration of shared Earth Observation in the Global Context Duration: Sept. 1, 2011 – Aug. 31, 2014 Total EC.
Opendap dev - meeting, Boulder, Feb 2007 OPeNDAP infrastructure in European Operational Oceanography T Loubrieu (IFREMER) T Jolibois (CLS)
1ECOOP Plenary Meeting Athens 2008 EUROMISS EUROpean Marine Information System of Systems S Pouliquen, Thomas Loubrieu ECOOP plenary meeting.
NIST Data Science SymposiumMarch 4, 2014 NIST Data Science SymposiumMarch 4, Climate Archives in NOAA: Challenges and Opportunities March 4, 2014.
NQuery: A Network-enabled Data-based Query Tool for Multi-disciplinary Earth-science Datasets John R. Osborne.
Data formats and requirements in CMIP6: the climate-prediction case Pierre-Antoine Bretonnière EC-Earth meeting, Reading, May 2015.
Introduction Geospatial users workflow and needs Problem and research goals Contribution Publication Service: GEOSS Service Factory Prototype assessment.
1 Accomplishments. 2 Overview of Accomplishments  Sustaining the Production Earth System Grid Serving the current needs of the climate modeling community.
| nectar.org.au NECTAR TRAINING Module 2 Virtual Laboratories and eResearch Tools.
CUAHSI HIS: Science Challenges Linking small integrated research sites (
NESC Worshop – 07 September 2005 Development of a Marine Metadata Standard Greg Reed Executive Officer Australian Ocean Data Centre Joint Facility.
Jost von Hardenberg ISAC-CNR, Torino, Italy with Paolo Davini, Susanna Corti, and many others EUDAT User Forum, Rome,Italy 3-4 February, 2016.
Joseph Antony, Andrew Howard, Jason Andrade, Ben Evans, Claire Trenham, Jingbo Wang Production Petascale Climate Data Replication at.
GEOSS Common Infrastructure (GCI) The GEOSS Common Infrastructure allows Earth Observations users to search, access and use the data, information, tools.
CEOS Working Group on Information System and Services (WGISS) Data Access Infrastructure and Interoperability Standards Andrew Mitchell - NASA Goddard.
IODE Ocean Data Portal - technological framework of new IODE system Dr. Sergey Belov, et al. Partnership Centre for the IODE Ocean Data Portal.
Test Dev UR2 Improved Console Usability Window Maximization problems Memory leaks Rendering issues UR3 Improved Console Usability Improved.
Weigel, Berger, Kindermann, Lautenschlager EGU Versioning for CMIP6 in the Earth System Grid Federation Data preparation Initial registration.
A Quick Tour of the NOAA Environmental Software Infrastructure and Interoperability Group Cecelia DeLuca Dr. Robert Detrick visit March 28, 2012
Page 1 CSISS Center for Spatial Information Science and Systems IIB and GCI Meeting CSR Architecture and Current Registration Status Prof. Liping Di Director.
Enhancements to Galaxy for delivering on NIH Commons
The CUAHSI Hydrologic Information System Spatial Data Publication Platform David Tarboton, Jeff Horsburgh, David Maidment, Dan Ames, Jon Goodall, Richard.
GEOSS Component and Service Registry (CSR)
USGS EROS LCMAP System Status Briefing for CEOS
Pasquale Pagano (CNR-ISTI) Project technical director
Discovering and accessing data from a distributed network of data centres S. Mazzeo (ESA)
IS-ENES2: Contributions to HPC for climate modelling
INTAROS WP5 Data integration and management
GEO WP 1. INFRASTRUCTURE (Architecture and Data Management)
WGISS Connected Data Assets
The ECOPOTENTIAL Virtual Laboratory Platform Data Providers side event 7 Nov 2016, St. Petersburg Paolo Mazzetti Stefano Nativi Mattia Santoro Antonello.
Flanders Marine Institute (VLIZ)
Marketplace & service catalog concepts, first design analysis
Copernicus Programme European Commission CEOS Plenary 2017
European Open Science Cloud All Hands Meeting Pisa 8-9 March 2018
DATA SPHINX & EUDAT Collaboration
The system S.INTE.S.I.S.-Establishments
Cyberinfrastructure for the Life Sciences
ACCESS-CM2 Performance
WGISS Connected Data Assets Oct 24, 2018 Yonsook Enloe
FDA Topics Going Forward…???
EDDI12 – Bergen, Norway Toni Sissala
Brian Matthews STFC EOSCpilot Brian Matthews STFC
ARCCSSive Scott Wales & Paola Petrelli ARCCSS CMS
ExPLORE Complex Oceanographic Data
Introduction to RDS Datasets
Data Management Components for a Research Data Archive
RDA uptake activities and plans: ESGF
Digital Object Management for ENES: Challenges and Opportunities
Leveraging PIDs for object management in data infrastructures RDA UK Node Workshop, July Tobias Weigel (DKRZ)
WGISS WGISS Connected Data Assets Status Report October, 2019 CWIC Team Eugene Yu (GMU), Archie Warnock (A/WWW), Li Lin (GMU)
Presentation transcript:

Updates on HPC and Data management at NCI Dr. Ben Evans Associate Director, REI National Computational Infrastructure (NCI) Australia

Major ACCESS and related codes analysed ACCESS APS2 (Global, Regional, City) ACCESS-CM2 ACCESS-S1 ACCESS-OM UM 10.6/7 N512, N728, N1024, N1280 WRF MOM5 0.25, 0.1 (inc CICE5 and SIS1) MOM6 NEMO 0.25 4DVAR N216, N320 EnKF-C BODAS GC2 GC3 GITM WaveWatch3 NOAA MOST ROMS Common HPC profiling methodology across all elements: CPU and memory, performance, decomposition, interprocessor communication, IO Evaluation against system roofline measures Evaluation on new computational technologies (e.g., new accelerator technology) ACCESS Reference configurations for community registered in access-dev.nci.org © NCI Australia 2017 ACCESS Science Day. Ben Evans

Climate and Weather Datasets within NCI Discovery portal * Datasets catalogue entries * Queries of the data catalogue * Data access for each dataset (download and metadata review) * Data is quality tested (metadata specifications, functionality, and usability within community applications) Usage tracking that rely on the datasets and data services (nationally and internationally) ~4,000 unique users per quarter: 50M queries 20-50 Tbytes accessed via data services 1. Climate/ESS Model Assets and Data Products 2. Earth and Marine Observations and Data Products 3. Geoscience Collections 4. Terrestrial Ecosystems Collections 5. Water Management and Hydrology Collections https://datacatalogue.nci.org.au © NCI Australia 2017 ACCESS Science Day. Ben Evans

NCI Data Publishing Service Architecture Standards-compliant Community-compliant Third-party implementations NCI custom implementations Data also accessed via Raijin NCI VDI (Virtual Desktop Interface) ACCESS-dev integrated into NCI cloud CMIP, CORDEX at NCI will be published with both NCI standard services and ESGF Project for automated dataset (variable level) replication service (ESGF, BOM, ECMWF, …) Some upcoming changed directory layouts for data (particularly CMIP) but also other collections. © NCI Australia 2017 ACCESS Science Day. Ben Evans

NCI New attribute level discovery NCI Metadata Attribute Service: Nightly Indexing NCI /g/data POSIX, Buckets Checksummed data ESGF Solr (federation) OpenSearch Reporting /g/data Raijin Tenjin S3 EC2 NCI © NCI Australia 2017 ACCESS Science Day. Ben Evans

New Services: ARCCSSive (MkII) /g/data NetCDF Crawl MAS CMIP © NCI Australia 2017 ACCESS Science Day. Ben Evans

Feedback: Privacy Policy Feedback
Do Not Sell My Personal Information
About project: SlidePlayer Terms of Service

© 2025 SlidePlayer.com. Inc. All rights reserved.