1. ODI: Open Data Interface (Status report)
16/05/2018
ODI: Open Data Interface (Status report)
ESA FP Days, ESTEC
D. Heynderickx, DH Consultancy, Belgium
P. Wintoft, Solar Analytics, Sweden

2. ODI overview
16/05/2018
Open Data Interface (ODI) is a database system for retrieving, processing and storing space environment (and other) data and metadata in a MySQL (MariaDB) database.
Development started in 2008 (Swedish Institute of Space Physics, DH Consultancy, Solar Analytics)
Compliant with CDF and SPASE data models
Server and client software for data downloading and pre/post-processing, and data retrieval.
Currently support for ~150 datasets.
Extensible with user added functionality.
A licence can be requested at
ESA FP Days, ESTEC

3. Project overview Activity title: ODI database maintenance
16/05/2018
Activity title: ODI database maintenance
Funding programme: INFRA
Contract value: 25 k€ ( /16/NL/LF/as CCN1)
Start and end dates: Nov 2017 – Oct 2018
Contractor team: DH Consultancy, Solar Analytics
Name of the TO in TEC-EES: Hugh Evans
Current developments are marked green in slides
ESA FP Days, ESTEC

4. ODI server software (I)
16/05/2018
PHP engine for database communication and process flow
Requires read/write access to MySQL database
Download and parser scripts
NASA/GSFC CDFexport tool to ingest CDF data
Python tool to ingest NetCDF data
Ingestion of FITS and PDS datasets
cron/quartz setup for automatic download and ingestion
User may add functionality triggered by hooks on download and ingestion
ESA FP Days, ESTEC

5. ODI server software (II)
16/05/2018
Coordinates for Earth orbiting spacecraft
Download of TLEs from
Generation of spacecraft coordinates (GEI) using NASA/JPL SPICE library (
Magnetic coordinates for spacecraft in Earth’s magnetosphere
UNILIB library for calculation of L, L*, MLT, …
For fixed or record varying pitch angle(s)
IGRF + OPQ field models
A single command triggers the whole processing suite (manually or in cron/quartz job), e.g.: php get_ingest.php goes_gp_mag_1m_rt
ESA FP Days, ESTEC

6. ODI server software: hooks
16/05/2018
raw_data_dir=GOES/RT/geomag
file_name_pattern=*Gp_mag_1m.txt
platform=GOES_RTP
platform_type=satellite
instrument=SEM
skeleton_file=Gp_mag_1m.skt
parser_file=../../parser/GOES_SEM_RT_TLE.php
availability=public
download_script=wget_GOES_RT Gp_mag_1m geomag
SPACETRACK_satnum=29155
UNILIB_PREFIX=
cron_schedule=8 * * *
User defined process hooks
Scripts (batch, shell or php) started by triggers
Pre/post download, ingestion
post_ingest.bat example: copy GOES GEI and magnetic coordinates from one dataset to others
ESA FP Days, ESTEC
@php %ODI_DATASETS%\dataset\goes_rt_coordinates.php goes_gp_mag_1m_rt

7. ODI client software (I)
16/05/2018
Direct local connection (php/MySQL) requires read access to database
Remote connection over HTTP/REST (server/client), JSON output
HAPI (NASA Heliospheric API: server/client
Java SE and MySQL Connector/J JDBC driver
APIs for php, Java, IDL, Matlab, Jython, Python
Standardised procedure syntax
Outputs in language specific objects
Excel interface
ESA FP Days, ESTEC

8. 16/05/2018
Excel interface
ESA FP Days, ESTEC

9. Database structure
16/05/2018
ESA FP Days, ESTEC

10. Datasets supported in ODI distribution (~150)
16/05/2018
ACE: archive and real time EPAM, SWEPAM, MAG, SIS
DSCOVR: real time IMF and plasma data (JSON streams)
GOES: archive and real time SEM data
SREM: PROBA1, Integral, GioveB, Rosetta, Herschel, Planck
Magnetic and solar indices (Kp, Dst, F10.7, ISN, OMNI, …)
Interplanetary particle datasets: HELIOS, IMP8, Voyager, Pioneer, Wind
Radiation belt missions: AZUR/EI-88, CRRES/MEA/HEEF/PROTEL, UARS/PEM, SAMPEX/PET, NOAA/POES/SEM2, XMM/ERMD, PROBA-V/EPT, RBSP/HOPE/MAGEIS/REPT/RPS, HIMAWARI/SEDA
“Proprietary” datasets: MIR/REM, STRV1B/REM, AMPTE/UKS, EQUATOR-S, ISEE1/WIM/KED, Meteosat/SEM
ESA FP Days, ESTEC

11. ODI applications ESA projects EC FP7 projects
16/05/2018
ESA projects
SEPEM, SEDAT, SPENVIS, SAAPS, JHelioViewer, HERMES, GALEM
SRREM, RENELLA, VALIRENE
SSA P2-SWE-II, VSWMC2 (P2-SWE-XIV), P2-SWE-XIII
EC FP7 projects
SEPServer
SPACECAST, SPACESTORM
ESA FP Days, ESTEC

12. Current activity: summary
16/05/2018
Updates of the client interfaces
Enhancements of the REST interface, new HAPI interface
Functionality for ingesting FITS and PDS data files
Parsing of SPASE metadata (
Support for new datasets (e.g. RBSP/RPS)
Software maintenance
Dataset maintenance
ESA FP Days, ESTEC