Firenze, September 2006ASAC Meeting- Computing IPT1 ALMA Computing IPT Science Software Status Update Gianni Raffi, ESO ALMA Science Advisory Committee September 2006

Firenze, September 2006ASAC Meeting- Computing IPT2 Time Now 1.07 Computing Summary Schedule CDR 4 (Data from IPS as _________) CPA #1 PARRR CPA #2 CDR 7 CDR 8 Computing IPT Management and Science Software Requirements High Level Analysis and Software Engineering Common Software and Archive Executive, Correlator and Control SW Schedule, Observation Prep, Offline, and Telescope Calibration Software Integration and Test 3 Antennas Integrated IR 4.1 (Single antenna commissioning) IR 5.1 (Interferometer support) IR 7.0 (Commissioning feedback) IR 6.0 (CPA punch list) R6.0 R5.1 R4.1 R7.0 R9.0 IR9.0 Full Science Support R4.0 IR 4.0 (Single antenna modes) R3.0 1st Antenna Integration Interferometry at ATF IR Array Software >> Observatory Software >>

Firenze, September 2006ASAC Meeting- Computing IPT3 Requirements Management Goal:: Evaluate development progress with respect to requirements Provide reporting. => Done via Central database (DOORS) with requirements derived from all sources (science, technical, operational) that allows us to: –Manage requirements (addition, editing, deletion, prioritization, review) –Assess the status of software development (reporting) –Analyze the impact of requirements changes –Coordinate requirement tracking for all computing subsystems. Complete functional team work and release development with integration tests (by ITS team) Validate implementation with user tests (by SSR subsystem scientists) following each release.

Firenze, September 2006ASAC Meeting- Computing IPT4 SSR Testing of Science Software Extracted from D.Shepherd’s page on TWiki at: Requirements available on SSR TWiki (unless explicitly mentioned below) All functional subsystems shown for completeness (except ACS framework and ObOps) EXEC –R3.1 - Executive internal pre-test complete. –R4.0 - user test scheduled to begin October 20. Scheduler –R3.1 - Optical pointing test completed in March 2006 –R4.0 - Testing planned for early fall This will test the scheduling simulator, particularly with weather simulation. TelCal –R3.1 - Features developed during this release tested August 21, Optical pointing reduction tests completed March –R4.0 – holography reduction tests scheduled late Archive –R3.1 – Basic archive function tested during optical pointing integrated user test in March –R4.0 – Tests of archive queries (projects, monitor point data) scheduled for integrated user test late 2006 –Requirements: Use Case development for Science Archive continues. Granular requirements for Archive starting to be produced. ALMA Science Archive (ASA) requirements about ready.

Firenze, September 2006ASAC Meeting- Computing IPT5 SSR Testing Status (cont.) Correlator –R3.0 - Testing planned for April, June, and July slipped to October Pipeline –R3.1 - End to end user test (from flagging to imaging) is planned for the pipeline in Fall –Quick Look: R3.0 - Operations test should be possible. It is not yet planned. ObsPrep –R3.1 – User test about to conclude. Control –R3.1 – Optical pointing features associated tested during integrated user test in March –R4.0 – Holography features scheduled to be tested late 2006 during integrated user test. –Requirements: adopting EVLA control requirements (modified for ALMA) for ALMA control granular requirements. Passed System Requirements Review. Offline –User Test 4: R2.1 - Single baseline data reduction and analysis - supporting the ATF. Test complete, report written. –User Focus Group: completed March 2006, provided input on python user interface. –User Test 5: Will test single dish reduction & imaging. Scheduled for Dec 2006 – date dependent on ACA requirements. Note: Control and correlator are exercised not only at ATF, but also every day in the Prototype System Integration (PSI) lab.

Firenze, September 2006ASAC Meeting- Computing IPT6 Function Based Team (FBT) development In principle planning of R4.0 FBTs Works to functionality across subsystems (~2 months) Based on Technical Use Cases Interface issues clarified soon Allows more frequent integration Some parallelism possible

Firenze, September 2006ASAC Meeting- Computing IPT7 Latest Developments Concluded R3.1 release and integrated in April Includes Optical Pointing e2e functionality (tested in March at ATF) Installed and tested on both prototype antennas Installed also in the Prototype System Integration Laboratory in Socorro Included integrated regression tests FBTs so far: Optical Pointing (completed in March) Shared Simulation of Single Field Interferometry (SFI) (April) Monitor DB optimization (July) On-going FBTs: –ALMA Project Data Model (APDM) re-factoring (Aug) Validation almost complete. Currently on hold behind higher priority Holography work. –Holography (Aug). Hw for tests in the lab early Sept., at ATF late Sept. –Single Field Interferometry II (to be used at ATF just after first fringes- achieved with PSI scripts)

Firenze, September 2006ASAC Meeting- Computing IPT8 OPT User Test (mar.’06) (1 st integrated user test of ALMA software) evaluated by members of the Computing IPT (CIPT) in March primary software limitations were found to be jlog instability and slowness, lack of coherent logging levels across subsystems, archive access (query mechanism not documented at the level of end users), lack of access to archived monitor points, missing CCL offset commands, and uncaught exceptions (e.g. Control, TelCal) that caused software crashes. Many of the interfaces and system responses have been designed for the developer. Approximately 170 defects, issues, and enhancements were identified in this report. Roughly 40% (70) of those are defects (14 (8%) priority 1 and 56 (32%) lower priority).

Firenze, September 2006ASAC Meeting- Computing IPT9 Next Developments (*) R4.1 ( up to June 2007): –Single Dish (SD) Function Based Team (Feb.) (w/o nutator) TP mode 1: OTF mapping in continuum Total Power TM mode 2: radio pointing measurements using TP –Test SFI with h/w at ATF (completion of SFI II work) Readiness Review (external) by subsystem (May) R5.0 ( up to Dec 2007) –SD: Nutator, Spectral line (prepare for equipment in Chile) –Interferometry: WVR, fast switching, Pointed mosaics –Respond to AIV use of software in Chile PAR (Dec.07) – Delivery of modes developed thus far to Chile During all of 2007 support in Chile for antenna: –Acceptance and integration –System characterization using continuum TP (*) Development plan will be adapted to availability of h/w CIPT formal user test will occur twice next year

Firenze, September 2006ASAC Meeting- Computing IPT10 Archive Software Persistency support for raw data, pipeline results (images and logs), SBs and all other User input, site characteristics, monitor data, calibration data, catalogs, and all other auxiliary information required for ALMA to observe. Quick look results are temporarily stored in the Archive. Items not specific to ALMA Science Archive: User Repository: Manage database of registered users, support proprietary periods. User query, retrieval and monitoring capability. (event data R4.0) Data replication between OSF, Santiago, ARCs (both via network and physical media). Use of NGAS supported data replication. Requires ALMA testing. Request Handler: Manage packaging, request tracing and delivery of data products to the scientific user community (implementation planned for Q4 2007) High-availability system with failover (test implementation Q3 2007) Provision for safe backup of all data; when more than one Archive is operational this may be accomplished via data mirroring between Archives. Support 6 MB/s average and 60 MB/s peak (sustainable for many hours at a time) data rates.

Firenze, September 2006ASAC Meeting- Computing IPT11 Archive Replication between SOC and ARCs Full description on Archive Twiki at: 1.Bulk data replication taken care of by NGAS. 2. Transaction based replication for Oracle DB OSF- SCO 3. Standard DB incremental backups sent via NGAS.

Firenze, September 2006ASAC Meeting- Computing IPT12 ALMA Science Archive (ASA) Scientific view on ALMA observations through ALMA data products (cubes). –Preview image, possibly via an interface to Science Pipeline. –Science Archive Requirements document was drafted and has been discussed by the SSR. –Many use cases for the ASA have been drafted and were given to the developers. ( ) Support for storage and association of ‘advanced products’. –Catalogue Support: offer access to catalogues (calibrators, spectral lines) and calibration database (R4.0) Consists of two major parts: –ASA portal: Main access point to all query and retrieval services. –ASA data repositories and ASA services. VO aware by sharing technologies and standards. –ASA VO registry (planned) –ASA VO SkyNode (planned)

Firenze, September 2006ASAC Meeting- Computing IPT13 ALMA Science Archive (ASA) Operations Archive (Santiago) Dataset ALMA Science Archive Commands Archive Subsystem State Queries Archive Users, PI, VO, Staff ALMA Staff Query Responses Data Processed Data Quality Data Correlator Data Reduction Pipeline Exercise use cases for each user type: ALMA staff, Archive Users/PI and Reduction Pipeline Test interfaces (query structure, ease of use); transfer content (connections between APDM structures in ASA made & distributed correctly); & performance. Test data transfer from Operations Archive and ASA.

Firenze, September 2006ASAC Meeting- Computing IPT14 Support for ASA users Staff & PIs: User repository development: manage database of registered users Add a user to the repository: Manager view of users: User edits his/her contact information:

Firenze, September 2006ASAC Meeting- Computing IPT15 Archive table browser prototype list of ALMA Science Data Model (ASDM) tables entities table populated by Correlator (actual contents not shown)

Firenze, September 2006ASAC Meeting- Computing IPT16 Observing Preparation Support creation of Phase 1 proposals and Phase 2 programs, with submission through the ALMA archive. –“APDM” refactored (R4) –Visual Spatial Editor – first version. Visual Spectral Editor also exists. Final versions of both for R6. –Support for Director’s time (R5) –“Special SBs” for fixed period scheduling now available. (R4) –Phase I “bureaucracy” and textual cases (R5). –Phase I – II transition (creation of Phase II from approved Phase I) (R5) Support all ALMA Observing modes –So far optical pointing and holography observatory modes, and Single Field Interferometry (SFI) observing mode. –infrastructure in place to allow rapid development for further modes, R5-6 and on. Provide a mode which operates in terms of science goals with automated generation of SBs. –Science goal-based input for Phases I and II – First “wizard” now available for SFI. Will expand as modes are developed. –A generic framework for science input collection and SB generation is in place. –Alternative science goal presentation (“workbooks” - more flexible for users than a wizard) under development – R4. –Almost complete infrastructure will allow science goal input and SB generation to be extended to more observing modes. R5-6 and on.

Firenze, September 2006ASAC Meeting- Computing IPT17 Observing Preparation (cont.) Exposure time and data rate calculators shall be provided. –ETC complete – in maintenance. Supports all arrays. –Data rate estimator first version available, requires more user friendly presentation (R5.1) –Execution likelihood estimator (R5) Sensible default values and basic validity checking will be provided. –Infrastructure to support validation and setting of (sensible) defaults is in place. –Needs to be rolled out for all items (R4.1) Provide an expert mode with access to technical parameters of interest to observers (e.g., detailed control of correlator configuration). –Almost complete for main array – in maintenance as other specs emerge. Still details to be determined for ACA/TP arrays. (R5) Local disk persistency (provided by MicroArchive). In maintenance. Source list and catalog (target and spectral) support –Infrastructure in place with interim catalogs. –To be moved to Observatory Characteristics (R4) –Final catalogs in place by agreement with Operations (R6?) Widely portable software. –Supported now, will continue.

Firenze, September 2006ASAC Meeting- Computing IPT18 Spatial Visual Editor

Firenze, September 2006ASAC Meeting- Computing IPT19 Observatory Operations Software (ObOps) Passed its PDR at CDR4 (June 2006). Still at.5 FTE/yr effort Infrastructure: Ops Support Database, Data Access Module, etc. (R4-R9) Shift Log Tool: record all day- and night-time observing activities –core features (R4) –basic features (R5) –full implementation (R6) Observatory Characteristics Database GUI (R4.1) Phase One Manager: support for proposal review process (R6) ProjectTracker: control and monitor operations information flow (R6) User Portal: integrated access to operations info –internal (ALMA only) access (R7) –public access (R9) QualityTool: data quality assurance (R7) DataPacker –raw data delivery (R7) –data package production and delivery (R9) PublicationsDatabase (R9)

Firenze, September 2006ASAC Meeting- Computing IPT20 Offline software Priority 1 requirements. In particular for the next release: –Single dish analysis, High dynamic range imaging (R4.0) –Standard calibration (R4.0) –CASA framework doc. Portable to a range of Unix-like operating systems. Efficient enough in operations to satisfy user expectations and for use with the Science Pipeline. GUI,CLI, and script interfaces (Image viewer & data visualization GUIs available) Readily installed by end users for processing execution. GUI tools for raw data and image inspection, flagging, arbitrary plotting. Filler for ALMA Export Data Format for Interferometric data (single dish ASDM filler to be developed when ASDM format defined). –ASDM extensions, EDF doc (R4.0) Extensive support for multi-channel, multi-polarization, multi-spectral window, multi-pointing flagging, calibration, imaging, and analysis for interferometric, single dish, and combined data from existing telescopes. Simulation capability. Processing history, interactive warnings, and re-execution facilities. Direct access to the ALMA Archive and the VO. Atmospheric modeling for the ALMA bands Extensive user documentation

Firenze, September 2006ASAC Meeting- Computing IPT21 CASA AIPS++ –Glish interface; unknown, unsupported outside of NRAO –Tasking system based on Glish –GUI system based on Glish/Tk; limited widgets, not robust! –Difficult for external developers to contribute –Multi-CD binary distribution –Large monolithic libraries with cross dependencies –No namespace –Freeze CASA –Python interface (community standard); IPython extensions –Direct binding to Python; ACS, other frameworks readily possible –Hierarchical set of small libraries with clearly defined dependencies –Namespace protection for integration with other code –RPM distribution mechanism; auto-updates possible –Robust –Inherits all application code improvements in robustness and performance.

Firenze, September 2006ASAC Meeting- Computing IPT22 DataCapture Maps telescope, observing information into Alma Science Data Model (ASDM) for post-processing in near-real time.

Firenze, September 2006ASAC Meeting- Computing IPT23 CASA Timeline 2006 –NRAO: NAUG testing (AIPS++) –Community: Project tests –ALMA: external science testing: +single-baseline commissioning test) (CASA) –ALMA: external science testing: +single dish reduction (CASA) –ALMA: commissioning support (CASA) –ALMA/EVLA: user interface review (CASA) –EVLA: external science testing: +full polarization imaging; antenna pointing calibration – AIPS++ frozen 2007 –NRAO: User support (CASA) –ALMA: commissioning support (CASA) –EVLA: external science testing: + wide band calibration imaging (CASA) –ALMA: P1 SSRs complete 2008 –EVLA: external science testing: RFI (CASA) –NRAO: user support (CASA) –ALMA: commissioning support –EVLA: P1 SSRs complete –ALMA/EVLA: commissioning support –Community: CASA released/distributed for early ALMA/EVLA science 2011 –Community: CASA released/distributed; full user support (ALMA/EVLA: P1/P2s)

Firenze, September 2006ASAC Meeting- Computing IPT24 Offline User Test 4 (report Apr.06) Test delayed by SSR (not developers) so the first framework changes and python commands could be evaluated in a realistic environment of single baseline interferometry. CASA is a python binding of the C++ functionality that was in AIPS++. This initial version of CASA had python commands that had a one-to-one correspondence with AIPS++ Glish commands. first test of the CASA infrastructure and python interface. focus was to exercise and verify functionality for –editing, visualization, data reduction and analysis for single baseline interferometry – –capabilities that are needed at the ALMA Test Facility and for commissioning support. Testers with extensive experience commissioning radio telescopes and/or operating telescopes. In general, single-baseline data reduction procedures worked with good stability and accuracy. Additional visualization capabilities to support commissioning activities still needed.

Firenze, September 2006ASAC Meeting- Computing IPT25 Offline Development & Test Schedule Single Field VLA & PdBI 13mm 1/3mm User Acceptance Tests Small Multi-field Mosaic Interferometer/TP combo Single-BL Commissioning at ATF Increased capability VLA & BIMA mosaics Complete 1st Python interface, new framework Single dish reduction & imaging Python tasks, new framework VLA+GBT & BIMA+12m Community use, command line python Limited release, mostly command line, some GUIs User Focus group Tools  Tasks Delay: late arrival of ACA requirements. Delay: SSR decision to allow testing of 1 st python interface.

Firenze, September 2006ASAC Meeting- Computing IPT26 Offline Key Planned Development for 2006/2007: Commissioning Plan shows heavy use of the offline package. –Change development priorities/details (e.g.: single baseline analysis software must be ready earlier than expected; more plotting capabilities needed). –Completed commissioning support test in February 2006 (1 st step before using ATF) Single-dish reduction & imaging – will be based on ASAP package –Single-dish user test scheduled for late 2006 (dependent on ACA requirements). –Also evaluate task-based reduction & imaging for interferometric data. Complete ALMA interferometric ASDM filler and new viewer (more robust). Formal tests have been limited to specific datasets to verify development along a defined path. This has been necessary for development but need to expand to more open use once python command line interface is stable – 2007 – 2009: Staged plan to release CASA to astronomers for scientific use. Not final yet, however, key components are: Nov 2006: CASA user point person starts (guide user interface development, documentation & help new users) : Increase number of NRAO scientists & a few external scientists reduce own data Mid 2007: Early release of CASA simulator Mid-2008: Release to the public.

Firenze, September 2006ASAC Meeting- Computing IPT27 Science Pipeline Goals Automated flagging, calibration and imaging with associated quality information for common ALMA observing modes. Pipeline processing shall be data driven). Main purpose is to produce: calibrated, scientifically useful images. Following information will be calculated or available for use in pipeline processing: sideband ratios, temperature scale from an atmospheric model, bandpass calibration, instrumental polarization, phase rms, antenna efficiencies from source with known flux, and time variable amplitude and phase curves. (heuristics computing bandpass, gain, and flux calibrations for single field interferometry data, (R3.1->) Choose: whether or not to use WVR corrected if both are available, to reuse previous calibrations if necessary, to select appropriate deconvolution algorithm, to do self-calibration or not, to do continuum subtraction or not (Choices non-existent or hardcoded, (R4.1->). Automatically assemble all needed information from the Archive (i.e., assemble appropriate sources from multiple SBs over multiple epochs / configurations). (automatic ASDM retrieval for observing modes with simple observing unit set structures supported, (R2.1->) User friendly processing summary and processing script which can be modified and used in later reprocessing. (processing summary logs are written by current pipeline heuristics scripts, automatic script logging may be available from CASAPY, (R4.0->) Scripting interface to allow operations to modify processing heuristics without changes in compiled code. (Scripts will be written in Python, (R4.0) Manual mode (e.g., for reprocessing particular projects). (R5.0->) First version of pipeline processing scripts. (E2E for: single field interferometry (R4.0->), for: single dish: (R4.1->)

Firenze, September 2006ASAC Meeting- Computing IPT28 Status: Heuristics Single field interferometry (prototype e2e heuristics) –calibrator source baseline based flagging heuristics –bandpass, gain, flux calibration solution heuristics –calibrator source imaging heuristics –some quality assessment measures –calibrator source antenna based flagging heuristics –target source flagging and imaging heuristics Single dish (baseline fitting heuristics) –including line detection and masking heuristics Software –general Python classes for flagging, calibrating, and imaging SFI data –general Python classes for baseline fitting (some cleanup required) Heuristics development slower than expected (based on 2004 planning) Reasons: Technical: CASA framework conversion happening at same time; Developmental: more difficult to get flagging heuristics correct than expected; Personnel: ramp up to expertise needed to develop interferometric heuristics slow.

Firenze, September 2006ASAC Meeting- Computing IPT29 Status Pipeline Infrastructure Quicklook –supports parallel observing sessions (independent subarrays) –provides clean separation of processing and display functions –working telescope calibration results monitor and GUI Science –working heuristics development framework (IPython, CASAPY) –working set of data processing engines (CASA, ASAP) –heuristics not yet connected to infrastructure –ASDM filler prototype –separate ALMA / CASA build environments Modules full set of tool methods available to heuristics from CASAPY for –table, ms, constants, quanta, autoflag, calibrator, and imager subset of tool methods available to heuristics from CASAPY for –image, measures prototype visibility display tool already available to heuristics from CASAPY –msplot Java based table browser available in latest stable –to be integrated with CASAPY

Firenze, September 2006ASAC Meeting- Computing IPT30 Heuristics User Test 3 – Browser Snapshots PdBI Spectral line From R3 User test conclusions (mar.’06): The quality of the automatic flagging was judged to be good by the majority of the testers and the calibration heuristics were found to be a reasonable first implementation

Firenze, September 2006ASAC Meeting- Computing IPT31 Pipeline - What is ahead for next year ? Science –E2E single field interferometry heuristics running in the ALMA environment –E2E single dish heuristics –prototype cluster computing environment QuickLook –user friendly telescope calibration monitor and display –real time visibility display using real time filler –some bad data detection heuristics integrated with alarm system

Firenze, September 2006ASAC Meeting- Computing IPT32 Dynamic Scheduling Software to automatically choose best SB given scientific priority, current environmental conditions, current and future configurations, and the total pool of possible observations Complete interactive - and queue - scheduling (R4.0) (R4.0). Planning mode simulator (incl. weather data and quality - R4.0) (R4.0) SB interruption and re-execution. (Special SBs for fixed-time scheduling - R4.0) (R5-6) Manual override of automated choices. (R4.1) Mechanisms for operations to change the weight of factors within the dynamic scheduler without significant programming. (R4.1) Scheduler reporting facilities. (GUI for master scheduler, Exec plug-in - R4.1) (R4.1) Heuristics delivered by CIPT will require optimization by operations. (R5) Multiple sub-arrays supported (including taking antennas off-,on-line - R4.1) (R4.1 ) ACA projects operating independently (R5-6) Medium term scheduling (R5-6) Long term scheduling (R7-8) Receiver setup optimizations (R7-9) [Breakpoint facility – low priority.] (R7-8) Accounting of time spent for various purposes (source, programs, observatory calibrations). (R?)

Firenze, September 2006ASAC Meeting- Computing IPT33 Operator Master Client Scheduler

Firenze, September 2006ASAC Meeting- Computing IPT34 TelCal Online (and Offline) Calibrations Telescope Calibration takes scans (mainly special calibration scans) and performs data reduction on them. Results are used by Control for further observations with improved parameters and by the Quick Look Pipeline to display and calibrate quick look data. Some array calibrations are not required to be performed on-line (e.g. Holography, pointing models). Holography (Pointing & focus offset, illumination pattern, surface err.) (R4.0) Atmospheric calibration (Temperature scale) (R4) WVR values (R5) Sideband ratios (R4) Phase calibration (phase rms, calibration curves) (R4) Amplitude calibration (calibration curve) (R4) Bandbass calibration (R5) Reference pointing (publish/reduce scan results) (done) Focus (partly done) Delay (publish/reduce delay scans) (R4.0) Pointing Model (optical done) Baseline measurement (positions from delays) (R4.0) Skydip Interferometry solvers (done) Automatic activation (according to intent) (R4.0)