NEES Grid Data Overview Comments to Charles Severance

Introduction The data approach has evolved significantly in the past year –Second version of the Data Repository (security, access control, performance improvements) –Data Turbine as unified real-time storage –NTCP has become increasingly capable –Model activity is bearing fruit (multiple) - Protégé / RDF / XML Schema –Data Curation summit has provided vision –We now have a notebook which captures metadata As we see more detail in these areas, we find new areas that need exploration

Boxology NEES Grid Data Approach Data Models Experiment Management Data Acquisition Experiment Monitoring Data Analysis Central Repository Local Repository Notebook

Data Lifecycle Data Models Experiment Prep Data Analysis Data Publishing Data Curation Data Discovery and Reuse Experiment Management Data Monitoring

Data/Metadata Capture Throughout Data Models Experiment Prep Data Analysis Data Publishing Data Curation Data Discovery and Reuse Experiment Management Data Monitoring

Data Models Data models are developed in RDF Local repository supports multiple simultaneous data models with cross-model linkages Metadata browser (aka Project browser) becomes the Project Browser, Notebook Browser, Site Specification Database Browser Metadata browser can federate multiple sources of Metadata

Project Model Proj Exp Trial Sensor Site PersonFacility Equipment Specimen Element Site Model Multiple Models Notebook ChapterEntry

Overall Data Modeling Efforts NEES Site ASite CSite B Equipment People Experiments Trials EquipmentPeople ExperimentsTrials Data Tsnumai Specimen Shake Table Specimen Geotech Specimen Centrifuge Specimen UnitsSensors Descriptions Site Specifications Database Project Description Domain Specific models Common Elements Data / Observations Ref. Source: Chuck Severance

Models + Data Model Repo Models Configure Data Load Configure RDF/ OWL RDF

Protégé - 2K Models + Data Model Repo Models Configure Data Load Configure RDF/ OWL RDF

Experiment Preparation Notebook –Allows the creation of material without needing a model –The model is pages, chapters, and “stuff” –It is all captured with data and metadata –A notebook can be attached to any object in the model structure (i.e. a project can have a notebook, a trial can have a notebook, etc…) Resources Discussions Project Browser –Setup basic structured metadata for the experiment - Trials, descriptions, sensors, etc… This material is captured in accordance to and with the data model

DOE ELN / Example

Setting up and Experiment Prior to running an experiment, the project browser will be used to create a trial, and experiment configuration, set up sensors, etc. In some cases, setup information may be done on the DAQ itself and the configuration information may be pulled from the DAQ

NEESgrid Experiment Data Flow NEESGrid Data Repository Project Browser Data Turbine Data Ingestion Experiment Control Streaming Viewer DAQ CD SiteSpecific ProjectRelated ExperimentalSetup ExperimentalElement DataElement Data Model DAQ Disk Stored Viewer

Experiment Management Simple reference implementations for –Experiment configuration (pull / push) –Experiment Start –Experiment Stop Some combination of LabView and CHEF code

DT Main System PTZ/ USB Still Capture DT Client BT848 Video Frames DT Client Capturing Video and Data Camera Control Gateway DAQ Data Capture DT Client Simulation Coordinator Site A Site B

DT Main System Data Monitoring Tools Still Image / Camera Control ~ <> ^ ^ <> Camera Control Gateway Creare viewers Still image camera control Thumb- nail

Working with Creare We want to leverage Creare’s live capture and viewers –Integrated Live Video and Data Viewer –Audio capability in addition to Video –JMF DataSource Capability - Use JMStudio SI will focus on the extraction, repository, data model, and stored viewer aspects

Data Stored in Data Turbine Video Stills Data Time Step* * Time Step is only present for Pseudo-dynamic Wall Clock Time

Data Extraction / Ingestion A tool will be developed to extract data from Data Turbine and place it in the NEES repository in the appropriate format –Video Channels –Image Channels –Data Channels The information will be stored in a format suitable for viewing using the stored viewer and appropriate metadata will be placed in the repository so that the information can be viewed This process is the primary new work in this plan

DT Main System Data Extraction For Analysis Data Extraction Pseudo-Dynamic Continuous Time Step Channel xyz Start Time Step 1 End Time Step 9999 ExportAuto Export NEES Data Repository

Pseudo-Dynamic Extraction Video Stills Data Time Step* Wall Clock Time

Continuous Extraction Video Stills Data Wall Clock Time

Stored Data Viewer Improvements Interactive Mode allowing reconfiguration of views within the Applet (insta-view) Linear combinations of data values Ability to launch from the Project Browser Looking at integration with notebook (i.e. launch from the notebook)

Central Repository / Curation Curation and the Central Repository are different than the local repository and the running / management of experiments Data must be packaged, kept, indexed, and maintained for the long term

Curation Flow Curation Bundle At some point, a project, experiment, etc is ready for curation. We must save all the information (models, notebooks, sensor data, etc) for transfer to the central repository

Data/Metadata Capture Throughout Data Models Experiment Prep Data Analysis Data Publishing Data Curation Data Discovery and Reuse Experiment Management Data Monitoring

Workflow in Central Repository The workflow of the central repository will be defined over time - here are some sample concepts –Incoming materials collect in an inbox –The curator processed the materials - adds required metadata, checks incoming data models, distinguishes information, and makes the bundle ready for publication –Some data is published immediately, other data is held for a period of time (perhaps to allow for publication) –Published data can be searched and viewed used and downloaded There are people in the curation loop The software for this is non trivial and will evolve over time with requirements Sometimes it will be necessary to alter/convert data to insure its value over time.

Workflow in Central Repository Curation Bundle Curation Bundle InBox Processed Published Search Hold for Time Need Conversion

Conclusion This is a significant adjustment in priority But not a significant shift in approach or architecture All of the elements which have been discussed can still be delivered - the elements described herein are just higher priority.