1. NEESGrid Data and MetaData Technology Kincho Law, Jun Peng, Jim Eng, Terry Weymouth, Paul Hubbard, Charles Severance

2. Goals n Data is online and persistent n Data and Metadata are supported together n Data migrates transparently including security, and metadata n Data is completely secure with access controls but security does not get in the way n Data provenance - how was it gathered, how has it been manipulated? n Data in support of research publication n Support for repeatable experiments n Data oriented research computation support n Support for workflow

3. The Slide Metadata Data Data Viewers Data Mappers Data Ingestors There is a layer is where we develop tools which take advantage and begin to depend on of the “meaning” of the data – where we begin to depend on the meaning of a second. Where we make a viewer capable of viewing a certain type of object. This is where we build things which make use of knowledge. This layer will never be complete but it is a large focus of the coming months. Concepts Search 09/2003

4. NEESgrid Data – Core Elements n Local Repository n Central Repository n JAVA APIs – Run locally on the same system as a repository or over OGSA Web Services –NEES File Management Services –NEES Meta Data Services n Data Viewers –Streaming (numeric, X/Y graph) –Stored (X/Y graph, 2-D structure, video)

5. NEESdata NEESpop Local Repository Core Elements API Central Repository Data Teamlets Data Acquisition Workstation API Data Teamlets API Data/MD Ingest Tools Data tools Data viewers Grid and Web Services

6. A Simple Experimental Scenario DAQ System Glue Test Specimen Labview Developer System Researcher System

7. Simulation System Code Simulation System Code A Simulation Scenario Developer System Simulation System Code

8. NCSA NEESPop (1.1) Colorado NEESPop (1.1) Incoming FTP NEESMost (Win XP) UIUC/Newmark NEESPop (2.0) LabView DAQ MatLab Host And Real-Time Target Control System Sim Controller CO LabView DAQ NSDS UIUC Test Specimen Matlab Computational Model Shore-Western Test Specimen Incoming FTP NSDS Repository CO NCSA UIUC Meta Series of files Complete file (aggregated) NTCP Site / Location ComputerProcess NCSA Ingest NTCP Matlab NTCP Ftp NTCP Wires NFMS/NMDS NSDS File I/O Plug In Ingest UIUC MOST Scanario

9. Overview of tools and technologies n Model development in RDF n Project Browser n Repository Browser n Electronic Notebook n Data Mappings n Data Viewer n Data Turbine n Data As Video n Still Cameras 11/2003

10. RDF Integration n Some of the data and meta data task force members are using Protégé-2000 to develop their models and expressing them in RDF. n RDF and NEESML are very similar but not identical so it may be challenging to ingest any arbitrary RDF n We expect that we will be able to map a subset of RDF to NEESML for ingestion or adapt an RDF parser (Jena or Raptor) to ingest that subset directly into the repository 11/2003

11. Models + Data Model Repo Models Configure Data Load Configure RDF/ OWL RDF

12. Protégé - 2K Models + Data Model Repo Models Configure Data Load Configure RDF/ OWL RDF

13. Repository Browser

14. Electronic Notebook n Collaborative effort with the DOE SciDAC –Electronic notebook - metadata entry –Data mapping –Data provenance –Data display –Slide data/metadata jakarta.apache.org/slide/ n Ultimate integration will be via JSR-170 n www.scidac.org/SAM/ n collaboratory.emsl.pnl.gov/docs/collab/sam/samtechoverview.html

15. DOE ELN / Example

16. DOE ELN / NEESgrid Integration (to date) SAM / Slide Repository Chef Grid Security NEESgrid Repository MyProxy Technology Celebration

17. DOE ELN / NEESgrid Integration (ultimate) Chef Grid Security NEESgrid Repository MyProxy

18. Mappings and the Data Viewer n NSDS (ISO 8601 Time channel) n Column data with time recorded as a column n Column – generate time n Column – generate time – trigger filter Channel units: g,g,in,kip Time ATL1 ATT1 2002-11-13T15:48:55.26499 -0.006409 0.004272 2002-11-13T15:48:55.36499 -0.005798 -0.003662 100.000 0.435 0.161 -1.016 -0.981 0.430 0.161 -1.016 -0.977 0.435 0.161 -1.016 -0.977 public class NEESDataMap { public static boolean repoMap(File mainFile, File mappingFile, String mapping) { // Code here }

20. Data Turbine n Commercial, free data streaming toolkit

21. Data Turbine (cont) n Existing data viewers will be adapted to access and display data from data turbine n Data acquisition software will be adapted to place information in Data Turbine Channels n Metadata elements will be developed to represent data turbine live, stored, and derived channels n New efforts (video as data) will be developed from the ground up using Data Turbine n outlet.creare.com/rbnb/ 11/2003

22. Video as Data n Follow on to initial demonstration at ORST n Experiment based development: Minnesota n Design phase complete n Joint effort, NEESGrid SI, ORST, Minnesota, UC Davis, Texas, Buffalo, and others

23. DT Main System CTL NTCP Plugin Data Turbine - Today NTCP Control DAQ Data Capture DT Client BT848 rbnbjcap DT Client NEES NSDS Driver Control Plugin AXIS / DT Gateway Axis

24. DT Main System Data Turbine NTCP Control Plugin Thumbs Control Make Smoothie Technology Celebration

25. DT Main System PTZ/ USB Still Capture DT Client DT Capturing Camera Control DAQ Data Capture DT Client Audio Encoder DT Client BT848 rbnbjcap DT Client Still Capture - Minnesota / Paul Hubbard Video capture - From Creare Audio capture - From Creare (TBD) Data Capture - From sites (upwards compatible) NEES NSDS Driver - Paul Hubbard Camera Control Plugin - Mich / Minn NEES NSDS Driver Each still capture produces two channels - Small 1-5fps stream + large single images when picture is taken Control Plugin

26. DT Main System User Views / Still Camera Quicktime Storage System Quicktime Slicing Tool Still Image / Camera Control ~ <> ^ ^ <> Control Plugin Data Viewer Thumbnail Process Thumbnail - uBuffalo / Umichigan Thumbnail viewer - Creare / Mich Camera Control Applet - Minn / Mich Quicktime Slicing tool - Mich (low) Stored Data Viewer - Mich JPEG Viewer - Creare Quickime Viewer - Apple Thumbnail + Audio + Data <>+

27. Minnesota Mock - up If you area developer and interested in following / helping / participating in this activity, join the mailing list neesgrid-dv@neesgrid.org

28. Tool List - To Do n Next release of repository n Integrate ELN into repository n DAQ Control Panel in CHEF –Set/Retrieve Metadata –Start / Stop –Ingest data from staging space n Data Turbine Control Panel in CHEF –Start / Stop / Configure Sources Video | Audio | Data | Thumbnail –Control permanent storage of video

29. To Do (cont) n NTCP Debugging and Monitoring in CHEF –Needed Data Turbine n Data as Video Client Tools in CHEF –New Monitor Tool –Still Image –Camera Control n Data Turbine Audio Capture n We may need to support XML Schema n QuickTime Capabilities –Archive, retrieve, slice, dice, convert, present –Probably will not be completed as part of SI effort

30. Data Model Work n Data Curation Summit –Understand issues form go-forward plan –Meeting 3/18/2004 n Data and Metadata Task Force –Finish the tsunami and centrifuge models n DSAC Committee –Meeting 3/19/2004

31. Summary n In September 2003, we met and “re-visioned” data n A bunch of requirements gathering and development has been done –The “high risk” elements are working now n There is more to do - We will run out of time –Evolutionary development approach - there will always be usable working code - we will stop when we run out of time

32. The people… n Gokhan Peckan - Data Models n Kincho Law - Data Models / Software design n Jun Peng - Data Models / Software design n Jim Eng - Parse / Ingest / RDF / Project Browser n Jim Myers - Electronic Notebook n Terry Weymouth - Data Turbine n Paul Hubbard - DAQ and NTCP n Joe Futrelle - Data / Metadata Repository