Online data access in the SeaDataNet V1 system by Dick M.A. Schaap – technical coordinator Bologna, September 08

CDI is a fine-grained index to individual data measurements (such as a CTD cast or moored instrument record). This is in contrast to EDMED, which is an inventory of data sets, each of which comprises a large number of measurements. CDI paves the way for direct online data access through file downloads or online data requests. Present CDI V0 system: > CDI entries from 36 Data Centres from 29 countries Common Data Index CDI

Common Data Index CDI V0

Central and homogeneous metadatabase, but for data access the users are confronted with the various user interfaces of the SeaDataNet data centres (which can provide online search and access / ordering facility / request) CDI V0 CDI url Data centre User interface User interface Data centre User interface Data centre User interface data

Central and homogeneous metadatabase plus transparent access to the distributed data sets via a unique user interface at the SeaDataNet portal CDI V1 CDI User interface Download Manager data Shopping basket Request Status Manager Download Manager Download Manager Central User Register

CDI Metadatabase and User Interface: Searching and browsing of metadata of data sets, managed by SeaDataNet Data Centres SeaDataNet Central User Register with User Roles db + web service : Details of users, their organizations and addresses, license period, Id-Passwords, SeaDataNet Roles of users SeaDataNet Shopping Basket: Preparing a user request of multiple data sets, handling validation of users and roles via the Central User Register, and routing requests to the Request Status Manager SeaDataNet Request Status Manager: Processing and administration of all requests and data deliveries (downloads), for users in communication with data centres. For data centres to oversee all transactions. CDI V1 System Modules

SeaDataNet Data centre Download Manager: Handles communication with SeaDataNet Request Status Manager and takes care that requested files are made ready for downloading by User (if cleared! ) in the agreed formats via a local website address. System Modules SeaDataNet Portal Data centre search shop Data download Metadata + transaction data

Order flow diagram Search in CDI directory Results Requests list Submit Requests (specify filetype? ) User Check CDI Meta database YES User + Role + Requests Shopping Basket NO Info about Conditions, Registration etc Log in Register Users + Licenses User Register User Roles WS

CDI V1 Search – home page

Search – Results page

Search – Detail page (full)

Search – records selected for Shopping basket

Shopping basket

Submit Shopping list => User Authentication

User identified and request for motivation and order confirmation

Request confirmed and link to RSM (also by )

Request Status Manager (RSM) Submit extended Request Form with motivation Shopping Basket YES User + Role + Requests Requests + user info in RSM; Split by DC Status: Pending Confirmation to User with RSM info Validation of access via business matrix ABCD U V W Combi logic +

Data access restrictions in CDI V1 CCCollection cost charge FECommercial charge LILicence MOMoratorium NANo access NCDistribution cost OGOrganisation RSRestricted SRAcademic UKUnknown UNUnrestricted See Vocabulary L081 for definitions Multiple restrictions may apply

User Roles in AAA Register, of relevance for data access SDRN02Public SDRN03Academic SDRN04Commercial SDRN05National and local government SDRN06Pan-National government See Vocabulary C866 for definitions Multiple roles may apply

Business Matrix Restriction Role ABCD U V W OK TBD NO

Request Status Manager (RSM) Accepted requests in Download Manager Validation of access via business matrix Data Centre Status: OK Status: Ready for Download Status: TBD to DC manager Validate in RSM Status: OK Status: Access Denied Access Denied Download Manager Status: NO Retrieving files + preparing formats (if database) + placing/updating daily ZIP file in user Download directory + notifying RSM of ‘ready for download’ status

Request Status Manager (RSM) Data Centre Download URL User Download webpage at Data Centre with option for downloading daily zip files + option for removing daily zip files Status: Downloaded Download Manager Retrieving files + preparing formats (if database) + placing/updating daily ZIP file in user Download directory + notifying RSM of ‘ready for download’ status If Removed User Check via AAA OK Not OK No access

Request Status Manager and Download Manager RSM has Web service to communicate with the Download Managers, locally installed RSM keeps IP-addresses, EDMO codes, data manager , trigger addresses, generic download paths and V1/interim V1 labels of all connected Data Centres in config file Download Manager is Java programme, that must be locally installed Download Manager also has a config file with IP-address of RSM service, EDMO code, paths to support directories with data sets, database mapping files, and coupling table

Functionality Download Manager V1.1 Communication with RSM Web service Retrieving and processing order lists after being triggered Retrieving data files, related to requested CDI’s, via local_CDI_ID and local Coupling Table Support for retrieving data sets from file management systems and from database systems, following the data set requests and the corresponding indicators in the local Coupling Table If data sets are stored in database, then producing the data files in the agreed ODV format via read-method Note: Data sets from a file management system must be pre-processed in SDN transport formats (ODV and optional MedATLAS for ASCII type data, or NetCDF for grid type data) Combining data set files in a new daily zip file or updating an existing daily zip file with the latest data sets and placing the daily zip files in a user download directory Generating dynamic user download webpage, following SDN template, to give user overview and option of downloadable zip files Note: the download URL is composed of the generic download path of the Data Centre as specified in the RSM configuration file (at installation) + the user login. Protecting the download URL via the AAA login service (CAS) Notifying the RSM Web service once a User has removed daily zip files (because the user has completed the download successfully) by indicating the status ’Downloaded’ for the associated data set requests Keeping a local log

Request Status Manager - Homepage

Submit Shopping list => User Authentication

User RSM – Overview of request per Data Centre and Status

User RSM : Status = Ready for Download + Download option

User personal Download webpage at Data Centre

User RSM: Status = All

All submitted requests, their users, the data sets concerned, and processing of these requests is administered in the Request Status Manager SeaDataNet Data centres can also log on to the Request Status Manager and oversee all requests, handle ‘tbd’ requests and prepare a number of online reports. Monitoring & Tracking

Data centre RSM: overview Standing Orders

Data centre RSM: overview To be Discussed Orders

Data centre RSM: overview History Orders

SeaDataNet : URLs CDI V1 retrieval CDI V1 RSM CDI V1 RSM Web service TESTER: WSDL: wsdl

SeaDataNet V1 Data Access planning CDI V1 interface, database and XML import are ready Download Manager V1.1 ready Coming months all TTT partners will install the DM V1.1 and configure their infrastructure for being a full V1 data centre. Preparing CDI V1 XML records following the upgraded CDI XML schema and using the new MIKADO software Installing the Download Manager software Choosing between pre-processed data files OR conversion via the Download Manager Converting data files to SeaDataNet formats, in case of choice for pre-processed mode Operational by end 2008

SeaDataNet interim V1 Implementation by other partners in 2008 Coming months all other partners will also start populating the CDI V1 database: Preparing CDI V1 XML records following the upgraded CDI XML schema and using the new MIKADO software Not yet installation of Download Manager, but data centres will be informed by the Request Status Manager by of all requests Data centres will process all requests manually using the Request Status Manager Effect is, that users will experience a common approach to all data centres. The difference is behind the scenes. In 2009 partners gradually moving to full V1 status by installing DM and delivering data sets in SeaDataNet Formats (conversion via new NEMO tool).

Thank you