The cf-python software library

Slides:



Advertisements
Similar presentations
1 NASA CEOP Status & Demo CEOS WGISS-25 Sanya, China February 27, 2008 Yonsook Enloe.
Advertisements

A Draft Standard for the CF Metadata Conventions Cheryl Craig and Russ Rew UCAR.
Data Formats: Using self-describing data formats Curt Tilmes NASA Version 1.0 Review Date.
HDF-EOS 2/5 to netCDF Converter Bob Bane, Richard Ullman, Jingli Yang Data Usability Group NASA/Goddard Space Flight Center.
RAMADDA for Big Climate Data Don Murray NOAA/ESRL/PSD and CU-CIRES Boulder/Denver Big Data Meetup - June 18, 2014.
MS698: Implementing an Ocean Model Benchmark tests. Dealing with big data files. – (specifically big NetCDF files). Signell paper. Activity: – nctoolbox.
Preparing CMOR for CMIP6 and other WCRP Projects
NCAS Unified Model Introduction Part 4a: UM tools and facilities University of Reading, 3-5 December 2014.
1 CF Unleashed: Introduction to Cf/Radial Joe VanAndel National Center for Atmospheric Research 2013/1/8 The National Center for Atmospheric.
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE) Increasing Accessibility and Interoperability of NASA Data Products with GIS Tools.
HydroShare: Advancing Hydrology through Collaborative Data and Model Sharing David Tarboton, Ray Idaszak, Jeffery Horsburgh, Dan Ames, Jon Goodall, Larry.
AS Computing Software definitions.
Data Formats: Using Self-describing Data Formats Curt Tilmes NASA Version 1.0 February 2013 Section: Local Data Management Copyright 2013 Curt Tilmes.
EARTH SCIENCE MARKUP LANGUAGE “Define Once Use Anywhere” INFORMATION TECHNOLOGY AND SYSTEMS CENTER UNIVERSITY OF ALABAMA IN HUNTSVILLE.
The Case for Data Stewardship: Preserving the Scientific Record Matthew Mayernik National Center for Atmospheric Research Version 2.0 [Review Date]
Atmospheric Data Analysis on the Grid Kevin Hodges ESSC Co-workers: Brian Hoskins, Lennart Bengtsson Lizzie Froude.
Metadata for the Coupled Ocean/Atmosphere Mesoscale Prediction System (COAMPS) using the Earth System Modeling Framework (ESMF) Peter Bosler University.
HDF5 A new file format & software for high performance scientific data management.
DM_PPT_NP_v01 SESIP_0715_AJ HDF Product Designer Aleksandar Jelenak, H. Joe Lee, Ted Habermann Gerd Heber, John Readey, Joel Plutchak The HDF Group HDF.
THREDDS Data Server Ethan Davis GEOSS Climate Workshop 23 September 2011.
Pursuing Faster I/O in COSMO POMPA Workshop May 3rd 2010.
ATMOSPHERIC SCIENCE DATA CENTER ‘Best’ Practices for Aggregating Subset Results from Archived Datasets Walter E. Baskin 1, Jennifer Perez 2 (1) Science.
U.S. Department of the Interior U.S. Geological Survey Management of Oceanographic time-series data at the Woods Hole Coastal and Marine Science Center.
Mid-Course Review: NetCDF in the Current Proposal Period Russ Rew
ViRBO and Autoplot. ViRBO Overview Virtual Radiation Belt Observatory Facilitate Search, Access, and Visualization of Radiation Belt Data Search – Develop.
Barcelona, 06 May 2015 s2dverification Seasonal to decadal forecast verification in R Overview Nicolau Manubens.
EARTH SCIENCE MARKUP LANGUAGE Why do you need it? How can it help you? INFORMATION TECHNOLOGY AND SYSTEMS CENTER UNIVERSITY OF ALABAMA IN HUNTSVILLE.
Database Essentials. Key Terms Big Data Describes a dataset that cannot be stored or processed using traditional database software. Examples: Google search.
Sherry Lake Candidate for Metadata Specialist for User Projects.
_______________________________________________________________CMAQ Libraries and Utilities ___________________________________________________Community.
NQuery: A Network-enabled Data-based Query Tool for Multi-disciplinary Earth-science Datasets John R. Osborne.
Data formats and requirements in CMIP6: the climate-prediction case Pierre-Antoine Bretonnière EC-Earth meeting, Reading, May 2015.
Introduction Amdocs’ clients line of work has to collect information from different, separated and independent devices, and each device has a lot of information.
The HDF Group Data Interoperability The HDF Group Staff Sep , 2010HDF/HDF-EOS Workshop XIV1.
Exporting WaterML from the Earth System Modeling Framework Xinqi Wang Louisiana State University NCAR SIParCS Program August 4, 2009.
The HDF Group Introduction to netCDF-4 Elena Pourmal The HDF Group 110/17/2015.
Part 6: UM tools and facilities
00/XXXX 1 Data Processing in PRISM Introduction. COCO (CDMS Overloaded for CF Objects) What is it. Why is COCO written in Python. Implementation Data Operations.
HydroShare: Advancing Hydrology through Collaborative Data and Model Sharing David Tarboton, Ray Idaszak, Jeffery Horsburgh, Dan Ames, Jon Goodall, Larry.
The HDF Group New Elements and Lessons Learned for New Mission HDF5 Products Ideas for new mission HDF5 data products 1July 8, 2013 Larry.
NCAS Computational Modelling Service (CMS) Group providing services to the UK academic modelling community Output of UM Diagnostics Directly in CF NetCDF;
1 Copyright © 2011 Tata Consultancy Services Limited Virtual Access Storage Method (VSAM) and Numeric Intrinsic Functions (NUMVAL and NUMVAL-C) LG - TMF148.
Santi Thompson - Metadata Coordinator Annie Wu - Head, Metadata and Bibliographic Services 2013 TCDL Conference Austin, TX.
CF 2.0 Coming Soon? (Climate and Forecast Conventions for netCDF) Ethan Davis ESO Developing Standards - ESIP Summer Mtg 14 July 2015.
Climate-SDM (1) Climate analysis use case –Described by: Marcia Branstetter Use case description –Data obtained from ESG –Using a sequence steps in analysis,
HydroShare: Advancing Hydrology through Collaborative Data and Model Sharing David Tarboton, Ray Idaszak, Jeffery Horsburgh, Dan Ames, Jon Goodall, Larry.
Automating the Audit: Updates from the Metadata Upgrade Project at the University of Houston Libraries Andrew Weidner, Metadata Librarian Santi Thompson,
Chapter 11 Analysis Methodology Spring Incident Response & Computer Forensics.
The CF Conventions: Governance and Community Issues in Establishing Standards for Representing Climate, Forecast, and Observational Data Russ Rew 1, Bob.
Enabling Digital Earth by focussing on ‘accessibility’ rather than ‘delivery’. Ryan Fraser CSIRO.
Big Data Analytics and Machine Intelligence Capability Team
Can Data be Organized for Science and Reuse?
Java Programming: From the Ground Up
Python data visualization
Big Data is a Big Deal!.
Data Are from Mars, Tools Are from Venus
SRNWP Interoperability Workshop
Experiment 01 C programming environment
Managing Copyrights in Invenio
Efficiently serving HDF5 via OPeNDAP
A platform for Linked Data publishing
Topics Introduction Hardware and Software How Computers Store Data
Accessing OECD Statistics WPFS December 2010
HDF-EOS Workshop XXI / The 2018 ESIP Summer Meeting
Peter Fox Data Science – ITWS/CSCI/ERTH 4350/6350 Module 3, Fall 2017
ExPLORE Complex Oceanographic Data
School of Information Studies, Syracuse University, Syracuse, NY, USA
Computer components is a programmable machine that receives input, stores and manipulates data, and provides output in a useful format. Computer The computer.
Overview of Computer system
ECMWF usage, governance and perspectives
Presentation transcript:

The cf-python software library 5th December 2014

Data, metadata and the CF conventions 3.56, 6.78, …, -0.32, 1.86

data + metadata data Data, metadata and the CF conventions 3.56, 6.78, …, -0.32, 1.86

Particularly suited to climate model outputs Data, metadata and the CF conventions data + metadata data 3.56, 6.78, …, -0.32, 1.86 CF conventions: A recognized standard for encoding Climate and weather Forecast metadata to support interoperability for earth science data from different sources Particularly suited to climate model outputs

The cf-python library An open source Python software package developed by NCAS to fully utilize the CF conventions Reads, manipulates and writes datasets from PP and netCDF files and UM output fields files by early 2015 cfpython.bitbucket.org

The cf-python library Read netCDF, netCDF and PP (and soon UM files) format files. Create fields. Write fields to netCDF files on disk. Aggregate collections of fields into a single field. Create, delete and modify a field's data and metadata. Select fields according to their metadata. Subspace a field's data to create a new field. Perform broadcastable, metadata-aware arithmetic, comparison and trigonometric operations with fields. Collapse fields by statistical operations. Sensibly deal with date-time data. Visualization with cfplot Copes with larger-than-memory datasets

An example with big data On a machine with a maximum of 32G of memory … Manipulate a larger-than-memory 4D field stored across multiple PP files and save it as a single CF-netCDF file:

cf-python is the library behind the cfa and cfdump tools for converting UM output to netCDF format and inspecting file contents The cf-python software is a useful python analysis tool