Still serving data with an old DODS server from the early 90's Jim Manning NOAA's Northeast Fisheries Science Center NERACOOS/NECOSP Data Management Workshop,

Slides:

Advertisements

Similar presentations

Visualizing Fitness for Purpose Bob Groman and Dicky Allison Biological and Chemical Oceanography Data Management Office Woods Hole Oceanographic Institution.

Advertisements

CT DEEP Long Island Sound Water Quality Data archive Todd Fake University of Connecticut NERACOOS/NECOSP Data Management Workshop, Sept. 26, 2012.

WP12. Hindcast and scenario studies on coastal-shelf climate and ecosystem variability and change Why? (in addition to the call text) Need to relate “today’s”

Tiffany C. Vance (NOAA/NMFS Alaska Fisheries Science Center)

Rome (26 Nov 2009) D4Science World User Meeting Moi Khim TAN The WorldFish Center Strengthening Information Access and Dissemination in Support of Effective.

Quality Assurance CS 615. Mission Statement The Quality Assurance team will provide assurance to stakeholders in CS-615/616 projects that their projects.

Low level CASE: Source Code Management. Source Code Management  Also known as Configuration Management  Source Code Managers are tools that: –Archive.

1 Presenting Regional and Urban Statistics Michal Peleg Israel Social Science Data Center The Hebrew University IASSIST/IFDO Conference.

Peter Cornillon University of Rhode Island Presented at the 10 September 2003 NVODS Workshop Washington DC The National Virtual Ocean Data System (NVODS):

10 March Setup Users. 10 March Setup Users Window Allows you to perform several user tasks –Enroll users –Enable/disable users –Set user access.

Comprehensive Large Array-data Stewardship System (CLASS) Web Site Tutorial Visit CLASS Site at

Data Management Needs and Challenges for Telemetry Scientists Josh M London Wildlife Biologist, Polar Ecosystems Program National Marine Mammal Laboratory.

Serving society Stimulating innovation Supporting legislation Meeting on OGC Sensor Observation Service (SOS) for INSPIRE Mickael.

CELTS Collection Exchange & Loan Tracking System Efficiency Tools, Inc. P.O. Box Carrollton, Texas Phone Let Efficiency.

Monitoring and Modelling the Spanish Coastal Waters. A new concept: The Operational PdE PORTUS System Workshop on “The status of coastal observing and.

CEOS/WGISS 20, Kyev, September 12-16, WTF-CEOP Implementation Plan #1 Status (WTF-CEOP first prototype, by JAXA) September 12, 2005 Osamu Ochiai.

MEDIN Data Guidelines. Data Guidelines Documents with tables and Excel versions of tables which are organised on a thematic basis which consider the actual.

Data Management Practices: BCO-DMO’s Successes and Challenges Bob Groman BCO-DMO Woods Hole Oceanographic Institution NERACOOS/NeCODP Data Management Workshop.

Spatial coherence of interannual variability in water properties on the U.S. northeast shelf David G. Mountain and Maureen H. Taylor Presented by: Yizhen.

EARTH SCIENCE MARKUP LANGUAGE “Define Once Use Anywhere” INFORMATION TECHNOLOGY AND SYSTEMS CENTER UNIVERSITY OF ALABAMA IN HUNTSVILLE.

Controlled Vocabularies (Term Lists). Controlled Vocabs Literally - A list of terms to choose from Aim is to promote the use of common vocabularies so.

Observing System Monitoring Center Integrating data and information across observing system networks.

Bringing it All Together: NODC’s Geoportal Server as an Integration Tool for Interoperable Data Services Kenneth S. Casey, Ph.D. YuanJie Li NOAA National.

MapServer-OGR-OPeNDAP: An Integrated System for Uniform Access to Land and Oceanographic Datasets Frank Warmerdam Consultant Thomas E. Burk University.

IOOS Data Management Integration Standards Plans in the Northeast Eric Bridger (GMRI) Sep

U.S. Department of the Interior U.S. Geological Survey Management of Oceanographic time-series data at the Woods Hole Coastal and Marine Science Center.

What has been lacking, until recently, is a successful method to develop, implement and sustain informatics solutions to modern application problems, such.

U.S. GLOBEC Pan-Regional Synthesis Workshop 1 Presentation to the U.S. GLOBEC Pan-Regional Workshop 29 November 2006 Bob Groman Data Access and Associated.

Ecosystem Based Approach to Management and Ocean Observing Kevin Friedland National Marine Fisheries Service, 28 Tarzwell Dr., Narragansett, RI 02882,

Glider data (*.*bd files) via Iridium or rsync from WHOI, Umaine or UMaryland Disk NetCDF/Acquire WMO ID IOOS Glider DAC ERDDAP OPeNDAP sequence or download.geoJson,.json.nc,.ncCF,.ncCFMA.csv,.tsv,

Drinking Water Infrastructure Needs Survey and Assessment 2007 Training.

Southeast Alaska Network Inventory and Monitoring Program Oceanography Glacier Bay National Park and Preserve.

Integrated Model Data Management S.Hankin ESMF July ‘04 Integrated data management in the ESMF (ESME) Steve Hankin (NOAA/PMEL & IOOS/DMAC) ESMF Team meeting.

Modelling 1: Basic Introduction. What constitutes a “model”? Why do we use models? Calibration and validation. The basic concept of numerical integration.

Soup-2-Nuts Alaska Department of Fish & Game Commercial Fisheries October, 2011.

The Rutgers IMCS Ocean Modeling Group Established in 1990, the Ocean Modeling Group at Rutgers has as one of it foremost goals the development and interdisciplinary.

OOI CI LCA REVIEW August 2010 Ocean Observatories Initiative External Observatory Integration Christopher Mueller, Matt Arrott, John Graybeal Life Cycle.

From Ocean Sciences at the Bedford Institute of Oceanography Temperature – Salinity for the Northwest.

Arctic Operational Oceanography at IMR Einar Svendsen Arctic GOOS planning meeting, September 2006 at NERSC, Bergen.

Observing System Monitoring Center (OSMC) Status Update April 2005 Steve Hankin – PMEL (co-PI) Kevin Kern – NDBC (co-PI)

IOOS National Glider Data Assembly Center

NQuery: A Network-enabled Data-based Query Tool for Multi-disciplinary Earth-science Datasets John R. Osborne.

U.S. GLOBEC Georges Bank 2007 Phase 4B SI Meeting April 23, 2007 GoMODP, Data Interoperability and the MapServer Interface to U.S. GLOBEC Data Presented.

Module 8: Managing Software Distribution. Collections Packages Programs Advertisements Collections Packages Programs Advertisements How Software.

Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP.

1 MSTE Visual SourceSafe For more information, see:

Observing System Monitoring Center (OSMC) Work in progress in brief June 2005 Steve Hankin, Kevin O’Brien – PMEL.

1 of 37 Next Steps SOS DescribeSensor implementation (SensorML) Testing/evaluation/refinement of existing work Build or borrow needed components … Reference.

Global Biodiversity Information Facility GLOBAL BIODIVERSITY INFORMATION FACILITY Hannu Saarenmaa EC CHM & GBIF European Regional Nodes Meeting Copenhagen,

NortheastOceanData.org Daniel R Martin Director of Geospatial Services I.M. Systems NOAA CSC NERACOOS/NECOSP Data Management Workshop, Sept. 26,

TOMS TRAINING Test Operations Management System 2016 PAWS Dec 2, 2015.

Do environmental factors affect recruit per spawner anomalies in the Gulf of Maine - Southern New England region ? Jon Brodziak and Loretta O’Brien NOAA.

Overview ABSTRACT As part of its Polar Satellite Program, NOAA operates the Jason-2 altimetry satellite and shares production of near real-time (NRT) products.

Morpho – metadata management software SEEK Training January 2004.

Using the Exchange Network One State’s Perspective Deb Soule New Hampshire Department of Environmental Services NERACOOS/NECOSP Data Management Workshop,

Access to IOOS Data Relevant to OOI Kathleen Bailey NOAA/NOS/IOOS January 6, 2016.

Maracoos DMAC and Asset Map June Maracoos DMAC Principles Data is served and integrated using open, interoperable data standards recommended by.

US GLOBEC Georges Bank Phase 4B Scientific Investigators’ Meeting 1 Presentation to the US GLOBEC Georges Bank Phase 4B Scientific Investigators October.

How Data Management Supports Ecosystem Science PIFSC Troy Kanemura Coral Reef Ecosystem Program Ecosystem Sciences Division April 4, 2016.

MySQL Getting Started BCIS 3680 Enterprise Programming.

Oceanic Monitoring and Sampling Phoebe Woodworth-Jefcoats Ecosystems and Oceanography Program Ecosystem Sciences Division PIFSC 5 April 2016.

Mantis Bug Tracker. MantisBT features Open source web-based bug tracking system. Track software defects. Mantis is often configured by users to serve.

OceanWatch Central Pacific Satellite Oceanography Products & Applications PIFSC Melanie Abecassis, PhD Ecosystems and Oceanography Program Ecosystems Sciences.

NQuery: A Network-enabled Data-based Query Tool for Multi-disciplinary Earth-science Datasets John R. Osborne 1, Kevin T. McHugh 2, and Donald W. Denbo.

Core LIMS Training: Entering Experimental Data – Simple Data Entry.

Data Browsing/Mining/Metadata

TOMS TRAINING Test Operations Management System 2017 PAWS

MERRA Data Access and Services

Integrating Data and Information Across Observing System

Mid Atlantic Water Property Measurements W. S

Presentation transcript:

Still serving data with an old DODS server from the early 90's Jim Manning NOAA's Northeast Fisheries Science Center NERACOOS/NECOSP Data Management Workshop, Sept. 26, 2012

NEFSC Oceanography Branch Falls under the “Ecosystem Processes Division” based in Sandy Hook, NJ The branch is based in Narragansett, RI but several of us are located in Woods Hole Primary functions are to describe & monitor: basic physical oceanographic processes (T,S, U&V) status of lower trophic levels the long-term variability of conditions on the shelf Primary data consumers: Fisheries biologist at our lab and elsewhere Academic researchers Data Requirements: ORACLE

About our data 1. Water Column Profiles (CTD) 2. Trawl Survey (merged with above) 3. Zooplankton 4. Derived Temperature and Salinity Anomalies by Region 5. Ship Track Surface Properties 6. Drifter Track Archive 7. Moored Time series (includes eMOLT & historic current meters) All of the above are shared (ie accessible) on our website 1,5,6,7 via OpeNDAP on old DOS server All of the above (except 3) visualize w/MapServer/gnuplot All of the above are updated at least a few times per year All of the above programs are multi-decades (except mine, 6&7) Most of the above documented on GCMD website

How we manage/serve our data Two options: 1) point & click through Perl/Mapserver Dataserver 2) download example MATLAB or PYTHON code to access it yourselves

Data Management Successes Original DODS server for relational database Python & Matlab users can bring the data directly into their working environments Dozens of data users have registered at least once to let us know they are using the data and what for Numerical modelers are using the eMOLT data, for example, to assimilation and validation of simulations

Data Management Challenges Political hurtles: NEFSC “Oceanography Branch” does not have a person trained in data management so we have to beg the “Data Management Branch” Overcoming the long-standing perception that there are more pressing issues in fisheries science than monitoring the effects of climate change (sector management, whales, etc. are more important to many) Technical hurtles: There is relatively little support in IOOS for serving relational databases? SOS required MySQL and therefore, for me, a duplicate database? Only one LINUX system admin guy who deals mostly with ~150 WINDOWS people Still waiting for a new LINUX box to be setup for updating our server so we can add THREDDS and Python functionality

Take home points NEFSC has collected a ton of data for decades eMOLT and Drifter Track archive is just a subset and is a relatively new addition Most of it is now served either through: Point & click MapServer interface OPeNDAP protocols We are still using an old DODS relational database server on a 10 year old LINUX box