ALMA Integrated Computing Team Coordination & Planning Meeting #2 Santiago, January 2014 ASDM relational database Rafael Hiriart / Jorge Avarias

ICT-L December 2013 Original Goals Our main motivation to create this database was to improve DataCapturer and QuickLook.  DataCapture was running out of memory, observations were reaching a limit of ~40 minutes with 32 antennas.  The original DataCapture “incremental writes” architecture – based on incremental writes at the level of the ASDM and XML Store – was dropped because of lack of resources in HLA/ASDM and Archive. It was suggested at this time to explore a relational DB solution.  We implemented a workaround for the Pointing table, the biggest ASDM table. By saving this table incrementally to a file and streaming it at the end of the observation to the binary store, we now support long observations (several hours).  We continued working on DC relational database aiming to improve QuickLook. This subsystem, which currently relies in a complicated collaboration between its own components, DataCapture an Quicklook, can be refactored to be a simple application that queries the database, performs a little processing and plot results.

ICT-L December 2013 Status First version of the relational database has been delivered (ICT-150) as part of release It is working in the OSF. A new version of QuickLook is being delivered with  This has become quite urgent because of frequent “freezes” of QuickLook running in the OMC.  Besides, QuickLook plots don't scale well with the number of antennas. A more interactive application is required, able to support filtering on antennas, scans, baselines, etc.

ICT-L December 2013 Demo

ICT-L December 2013 Implementation Based on Hibernate  The Hibernate mapping file is generated using Java reflection to know what are the columns and the keys of every table.  ASDM Java row classes cannot be used directly as mapping classes for Hibernate.  New wrapper classes were created to circunvent problems: Define the no-param constructor. Setters need to be public. Some getters need to be fixed (they throw runtime exceptions). Database is auto-generated by Hibernate, currently deployed in Scheduling database. Custom Hibernate types were implemented to handle simple types like angle, temperature, speed, etc.; and complex types like timeInterval, arrays, IDL enumerations.

ICT-L December 2013 Problems/Improvements Wrapper classes were manually (and painfully) produced. Mappings are created statically.  The system is fragile over changes in the ASDM.  The ideal way to create these artefacts would be to generate them as part of the HLA/ASDM build. Relationships between tables (FK) are not supported. Arrays are stored in the database as strings following the ASDM array serialization ( …...). Only Java is supported.

ICT-L December 2013 Our plan We stop here!  The mappings, as they are, fullfill our requirements to refactor QuickLook.  We don't have plans (or resources) to continue dedicating to this task. Jorge is in transition to 100% Scheduling. Other resources in Control are fully allocated. We are well aware of the interest to use this database in a broader context, but The database could need significant work/modifications to be useful for other applications (e.g., QA). DB is temporary. If a permanent DB is needed, APO needs to be involved. Maintainance, migrations, data duplication, etc., will need to be addressed. These developments are well outside Control group's scope and expertise.

ICT-L December 2013 Decision from Last Leads Meeting ICT-2048, action for Alisdair Manning: Provide a list of options to ICT management about possible delays to "fund" making the ASDM relational database permanent, and integrated into the code generation system. To be discussed in Alisdair's presentation.