1. ODI database maintenance
06/03/2017
ODI database maintenance
ESA/CNES FP Days, ESTEC
D. Heynderickx, DH Consultancy, Belgium
P. Wintoft, Solar Analytics, Sweden

2. Project overview Activity title: ODI database maintenance
06/03/2017
Activity title: ODI database maintenance
Funding programme: INFRA
Contract value: 25 k€ ( /16/NL/LF/as)
Start and end dates: Jul 2016 – Jun 2017
Contractor team: DH Consultancy, Solar Analytics
Name of the TO in TEC-EES: Hugh Evans
ESA/CNES FP Days, ESTEC

3. ODI overview
06/03/2017
Open Data Interface (ODI) is a database system for retrieving, processing and storing space environment (and other) data and metadata in a MySQL (MariaDB) database.
Development started in 2008
Compliant with CDF and SPASE data models.
Server and client software for data downloading and pre/post-processing, and data retrieval.
Currently support for ~150 datasets.
Extensible with user added functionality.
A licence can be requested at
ESA/CNES FP Days, ESTEC

4. ODI server and client Server Client
06/03/2017
Server
Client
Tools to create dataset tables, download data, ingest metadata and data.
Tools to setup automatic download and ingestion.
Hooks to trigger user code upon download and ingestion.
Implemented in PHP and MySQL.
Client can connect to different ODI servers.
Supports both local (SQL) and remote connections (HTTP).
Supported languages: PHP, Java, Python, Jython, IDL, Matlab, Excel
Data and metadata export tool to CDF
ESA/CNES FP Days, ESTEC

5. ODI software (I) IPR belongs to ESA, licence available on request
06/03/2017
IPR belongs to ESA, licence available on request
Server
PHP engine for database communication and process flow
Requires read/write access to MySQL database
Download and parser scripts
NASA/GSFC CDFexport tool to ingest CDF data
Python tool to ingest NetCDF data
cron/quartz for automatic download and ingestion
User may add functionality triggered by hooks on download and ingestion
ESA/CNES FP Days, ESTEC

6. ODI software (II) Coordinates for Earth orbiting spacecraft
06/03/2017
Coordinates for Earth orbiting spacecraft
Download of TLEs from
Generation of spacecraft coordinates (GEI) using NASA/JPL SPICE library (
Magnetic coordinates for spacecraft in Earth’s magnetosphere
UNILIB library for calculation of L, L*, MLT, …
For fixed or record varying pitch angle(s)
IGRF + OPQ field models
A single command triggers the whole processing suite (manually or in cron/quartz job), e.g.: php get_ingest.php goes_gp_mag_1m_rt
ESA/CNES FP Days, ESTEC

7. @php %ODI_DATASETS%\dataset\goes_rt_coordinates.php goes_gp_mag_1m_rt
ODI software (III)
06/03/2017
raw_data_dir=GOES/RT/geomag
file_name_pattern=*Gp_mag_1m.txt
platform=GOES_RTP
platform_type=satellite
instrument=SEM
skeleton_file=Gp_mag_1m.skt
parser_file=../../parser/GOES_SEM_RT_TLE.php
availability=public
download_script=wget_GOES_RT Gp_mag_1m geomag
SPACETRACK_satnum=29155
UNILIB_PREFIX=
cron_schedule=8 * * *
User defined process hooks
Scripts (batch, shell or php) started by triggers
Pre/post download, ingestion
post_ingest.bat example: copy GOES GEI and magnetic coordinates from one dataset to others
ESA/CNES FP Days, ESTEC
@php %ODI_DATASETS%\dataset\goes_rt_coordinates.php goes_gp_mag_1m_rt

8. ODI software (III) Client
06/03/2017
Client
Direct local connection (php/MySQL) requires read access to database
Remote connection over HTTP/REST
Java SE and MySQL Connector/J JDBC driver
APIs for php, Java, IDL,Matlab, Jython, Python
Standardised procedure syntax
Outputs in language specific objects
Excel interface
ESA/CNES FP Days, ESTEC

9. ODI client/server interaction
06/03/2017
Client
Client
Client
Client
REST
REST
SQL
SQL
ODI server
ODI server
ESA/CNES FP Days, ESTEC

10. ODI Java interface Web Excel REST server External REST server
06/03/2017
REST server
External REST server
ODI MySQL
Java ODI
Remote
Local
ESA/CNES FP Days, ESTEC
ODI Client
Client layer
Java
Py4j server
IDL
Jython
Matlab
Python

11. Jython example ~ peter$ jython
06/03/2017
~ peter$ jython
Jython (2.5:c56500f08d34+, Aug , 14:48:36)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.8.0_05
Type "help", "copyright", "credits" or "license" for more information.
>>> import se.irf.lund.odi.OdiClient as client
>>> odi = client.newInstance()
>>> odi.listDatasets()
{ace_mfi_h0={odi_name=ace_mfi_h0, odi_table_name=dataset_ace_mfi_h0, num_recs= , ...
>>> odi.selectDataset('ace_mfi_h0')
>>> odi.listVariables()
[{name=Time, key=epoch, type=null}, {name=B-field magnitude, ...
>>> odi.readData()
{cdf_epoch=[ E13, E13, E13, ...
ESA/CNES FP Days, ESTEC

12. Matlab example Notice connection over HTTP 06/03/2017
ESA/CNES FP Days, ESTEC

13. $ jupyter notebook
06/03/2017
ESA/CNES FP Days, ESTEC

14. ODI local app running in browser
06/03/2017
ESA/CNES FP Days, ESTEC

15. 06/03/2017
Excel interface
ESA/CNES FP Days, ESTEC

16. Database structure
06/03/2017
ESA/CNES FP Days, ESTEC

17. Datasets supported in ODI distribution (~150)
06/03/2017
ACE: archive and real time EPAM, SWEPAM, MAG, SIS
DSCOVR: real time IMF and plasma data (JSON streams)
GOES: archive and real time SEM data
SREM: PROBA1, Integral, GioveB, Rosetta, Herschel, Planck
Magnetic and solar indices (Kp, Dst, F10.7, ISN, OMNI, …)
Interplanetary particle datasets: HELIOS, IMP8, Voyager, Pioneer, Wind
Radiation belt missions: AZUR/EI-88, CRRES/MEA/HEEF/PROTEL, UARS/PEM, SAMPEX/PET, NOAA/POES/SEM2, XMM/ERMD, PROBA-V/EPT, RBSP/HOPE/MAGEIS/REPT, HIMAWARI/SEDA
“Proprietary” datasets: MIR/REM, STRV1B/REM, AMPTE/UKS, EQUATOR-S, ISEE1/WIM/KED, Meteosat/SEM
ESA/CNES FP Days, ESTEC

18. ODI applications ESA projects EC FP7 projects
06/03/2017
ESA projects
SEPEM, SEDAT, SPENVIS, SAAPS, JHelioViewer
SRREM, RENELLA, VALIRENE
SSA P2-SWE-II
EC FP7 projects
SEPServer
SPACECAST, SPACESTORM
ESA/CNES FP Days, ESTEC

19. Current Activity Development of a Python native client interface
06/03/2017
Development of a Python native client interface
Development of reporting, plotting and status tools
Enhancements of the REST interface
Update of the Excel client connection
Functionality for ingesting netCDF data files
Enhanced support of SPASE metadata (
Conversion into ISTP/CDF metadata
Definition of extensions to define dependent variables
Support for new datasets (including DSCOVR)
ESA/CNES FP Days, ESTEC