Computing ALMA Board Meeting November 2015 Jorge Ibsen Head of ADC, ICT Lead Contributions from: ADC Management (JAO): Achermann, Parra, Saldias, Shen,

Slides:



Advertisements
Similar presentations
ALMA Cycle 2 Capability Jongsoo Kim ALMA EA Korea node.
Advertisements

National Radio Astronomy Observatory June 13/14, 2005 EVLA Phase II Proposal Review EVLA Phase II Computing Development Bryan Butler (EVLA System Engineer.
ITIL: Service Transition
Atacama Large Millimeter/submillimeter Array Expanded Very Large Array Robert C. Byrd Green Bank Telescope Very Long Baseline Array.
CSV Plan status Robert Laing ESO Science IPT. 2 Recent changes Complete rewrite to take account of rebaselining and revised delivery schedule Proposal.
ALMA TACand the proposal process Lister Staveley-Smith Member, ALMA Review Panel.
AOP version E Science Operations For the ASAC meeting March 9-10, 2010 Lars-Ake Nyman 1.
MSIS 110: Introduction to Computers; Instructor: S. Mathiyalakan1 Systems Design, Implementation, Maintenance, and Review Chapter 13.
Configuration Management
System Design/Implementation and Support for Build 2 PDS Management Council Face-to-Face Mountain View, CA Nov 30 - Dec 1, 2011 Sean Hardman.
2008 Decmeber 5 ALMA-JT Science Meeting 1 ALMA Project News Koh-Ichiro Morita ALMAJ Project Scientist.
Atacama Large Millimeter/submillimeter Array Expanded Very Large Array Robert C. Byrd Green Bank Telescope Very Long Baseline Array.
CXC Implementing 2007 NRC Portals of the Universe Report Chandra X-ray Center Recommended Best Practices Roger Brissenden and Belinda Wilkes 25 April 2012.
December 2007Chile Observatories Earthquake Preparedness Workshop1 Atacama Large Millimeter/submillimeter Array ALMA Eduardo Donoso.
Hunt for Molecules, Paris, 2005-Sep-20 Software Development for ALMA Robert LUCAS IRAM Grenoble France.
Chapter 2 The process Process, Methods, and Tools
Software Configuration Management
Descripción y Areas del Dpto. de Computación de ALMA (ADC) Tzu-Chiang Shen Gerente del Grupo de Software Departamento de Computación ALMA Joint ALMA Observatory.
ALMA Common Software Basic Track Introduction to the ACS Framework.
ALMA Common Software Basic Track Software Engineering Basics.
ALMA Operations and the North American ALMA Science Center Al Wootten NRAO.
Workshop in ALMA Logs Prepared by Juan Pablo Gil – Arturo Hoffstadt
ICT Coordination and Planning Meeting #1 (17-19 April 2013) ALMA Dashboard 1.0 Giorgio Filippi The Atacama Large Millimeter/submillimeter Array.
ICT-CPM November 2014 ALMA Integrated Computing Team Coordination & Planning Meeting #4 Santiago, November 2014 Software Engineering and Quality.
EA ARC Ken Tatematsu East-Asian ARC Manager. ARC organization Difference between ARCS: NA: concentrated in Charlottesville Europe: distributed in different.
Principles of Information Systems, Sixth Edition Systems Design, Implementation, Maintenance, and Review Chapter 13.
A Search for Hydroxlyamine (NH 2 OH) Towards IRC+10216, Orion-S, Orion(KL), SgrB2(N), SgrB2(OH), W512M, W3(IRS5) R. L. Pulliam NRAO / North American ALMA.
ALMA Integrated Computing Team Coordination & Planning Meeting #2 Santiago, January 2014 Control Group Planning Rafael Hiriart, Control Group Lead.
The ALMA Software and Release Management Ruben Soto Software Operations Group & Release Manager Joint ALMA Observatory.
ALMA Integrated Computing Team Coordination & Planning Meeting #4 Santiago, November 2014 Reporting Software Principles M Chavan, ESO R Soto, ALMA.
ALMA Software B.E. Glendenning (NRAO). 2 ALMA “High Frequency VLA” in Chile Presently a European/North American Project –Japan is almost certainly joining.
ALMA Common Software Basic Track Component implementation guidelines.
Doug Tody E2E Perspective EVLA Advisory Committee Meeting December 14-15, 2004 EVLA Software E2E Perspective.
Software Quality Assurance
ALMA Common Software Basic Track Test Driven Development Unit testing and TAT.
ALMA Common Software Basic Track Logging and Error Systems.
ALMA Integrated Computing Team Coordination & Planning Meeting #4 Santiago, November 2014 Telescope Calibration Planning Dominique Broguière.
ALMA Common Software Basic Track A walk through ACS functionality.
1 Computing Challenges for the Square Kilometre Array Mathai Joseph & Harrick Vin Tata Research Development & Design Centre Pune, India CHEP Mumbai 16.
Response to ASAC Report of May 2004 Response to ASAC The Atacama Large Millimeter Array Tom Wilson and Al Wootten Science IPT.
ICALEPCS’ GenevaACS in ALMA1 Allen Farris National Radio Astronomy Observatory Lead, ALMA Control System.
2007Sep06 EAC Butler - Software Overview 1 Software Overview Bryan Butler.
ALMA Integrated Computing Team Coordination & Planning Meeting #1 Santiago, April 2013 Telescope Calibration Planning Dominique Broguiere.
ALMA Polarization Commissioning and Verification Status Kouichiro Nakanishi (Joint ALMA Observatory/NAOJ) on behalf of ALMA Polarization Commissioning.
Atacama Large Millimeter/submillimeter Array Expanded Very Large Array Robert C. Byrd Green Bank Telescope Very Long Baseline Array Data Processing Progress.
Principles of Information Systems, Sixth Edition 1 Systems Design, Implementation, Maintenance, and Review Chapter 13.
ALMA and the Call for Early Science The Atacama Large (Sub)Millimeter Array (ALMA) is now under construction on the Chajnantor plain of the Chilean Andes.
ICALEPCS 2005 Geneva, Oct. 12 The ALMA Telescope Control SystemA. Farris The ALMA Telescope Control System Allen Farris Ralph Marson Jeff Kern National.
Atacama Large Millimeter/ submillimeter Array - ALMA ASAC Charges For Oct 31 ASAC Report to ALMA Board Al Wootten JAO Interim Project Scientist.
ALMA Common Software Basic Track Project Lifecycle.
Gustaaf van MoorselEVLA Advisory Committee Meeting December 14-15, 2004 EVLA Computing End-to-end (E2e) software.
ALMA Common Software Basic Track Configuration Database.
ALMA Common Software Basic Track Component/Container Model and Lifecycle Management.
PDS4 Project Report PDS MC F2F University of Maryland Dan Crichton March 27,
Jeff Kern NRAO/ALMA.  Scaling and Complexity ◦ SKA is not just a bigger version of existing systems  Higher Expectations  End to End Systems  Archive.
Dashboard upcoming features A Hales, ALMA and M Chavan, ESO
ACA TP Spectrometer Manabu Watanabe (NAOJ)
ITIL: Service Transition
Software Project Configuration Management
EGEE Middleware Activities Overview
ALMA Common Software Basic Track
ALMA Software Scheduling subsystem Planning for cycle5 onwards
PRTS & KPI Nick Whyborn – Vasco Cortez
Shift Log Tool Refactoring
Obsprep Planning, 2017 Alan Bridger
Upgrade to Oracle12c in February 2017 José Parra
IEEE Std 1074: Standard for Software Lifecycle
DBA Coordination Group Update José Parra
Welcome K. Y. Lo Director, NRAO
Atacama Large Millimeter Array Science IPT Review
Presentation transcript:

Computing ALMA Board Meeting November 2015 Jorge Ibsen Head of ADC, ICT Lead Contributions from: ADC Management (JAO): Achermann, Parra, Saldias, Shen, Soto ICT Management (ESO, NAOJ, NRAO): Griffith, Kern (NA), Kosugi, Watanabe (EA), Chavan, Schmid (EU)

Composed by onsite and offsite activities carried out by: ALMA Department of Computing (ADC) Composed by Archive and Pipeline Operations (APO), Information Technology (IT) and Software (SG) groups Funded by JAO ADC budget (staffing and running costs) Integrated Computing Team (ICT) Successor of ALMA Computing team for the operations phase (Jan 2013 onwards), consisting of Regional ICTs from the three Executives and SG from the JAO OFF Offsite Operations – Software Development and Maintenance exclusively funds the Regional ICTs from the three Executives from 1 January 2013 Interfaces with many other onsite and offsite groups: ADE, DSO, IET, ISOpT, AMT Computing in a nutshell

ALMA Core Processes

Computing directly contributes to all ALMA Core Processes Phase 1 Proposal Generation : Call for Proposals; Proposal Submission, Review & Scheduling, User Support & Notification; Phase 2 Program Generation : User Support, Scheduling Block creation, Submission & Validation; Observation Execution : System Calibration, Site Conditions Monitoring, Quick- Look Quality Assurance, Dynamic Scheduling and Scheduling Block Execution; Maintenance : Preventive & Corrective Maintenance, Performance Trending, Fault Correction, Array Re-Configuration; Archive and Pipeline Operations : Archive Maintenance (Science & Engineering Data), Detailed Quality Assurance, Data Product generation & Delivery; Science Research Archive : Web-Based, Virtual Observatory Compliant Interface to Public Data, Project-Independent search & Retrieve Tools.

Body responsible for: Deploying, operating, supporting, administering, maintaining in coordination with the executives and enhancing all ALMA computing infrastructure and services collocated in Chile hardware- and software- wise. This includes, among other systems, the networking and server information technology infrastructure and services, general-purpose software, and the domain-specific software developed by the executives and used by the observatory for scientific proposals handling and data acquisition, processing and delivery to the ALMA executives. Defining computing standards, and developing and enforcing policies internal to JAO, aiming to get a reasonable level of commonality with ALMA partners whenever it is feasible. In particular, JAO computer and network security policies have significant importance to ensure that the systems in Chile are protected from unauthorized and malicious attempts to gain access to them. ADC Responsibilities

ADC Organization 37 FTEs (2 ISM, 35 LSM) ~ 50% at OSF, 50% SCO

The ICT’s main responsibilities are the on-going software support, maintenance and internally funded feature development for all delivered ALMA Subsystems. The two core principles are:  Responsibility for maintenance stays with the Executive who was responsible for the same deliverables during construction  ICT-CL is responsible for all operational matters, first level support and diagnostics, and the ALMA software release process. Maintenance is considered to include corrective, adaptive, perfective and preventive tasks for the delivered software. It is anticipated that some ALMA subsystems (like f.i. pipeline) will require significant updates as operational experience with ALMA is acquired. Significant changes in scope (VLBI capability, new bands, significant changes in data rate) must be considered separately as part of the ALMA Development Program or another externally funded program. ICT Responsibilities

ICT Organization Total effort ~64 FTE/year between 2015 and Staffing levels are based on the current budget and FTE-cost projections. If FTE costs vary they are adjusted to meet the budget rather than vice versa.

Operations Review in April, ICT Leads meeting in May to discuss 2020 items, ICT Planning meeting in December ACA hardware and servers upgraded in Feb, remaining (three) outstanding issues were worked out by May Since Feb, users logged into trusted portals (ESO, NRAO) don't need to explicitly log onto ALMA's Science Portal VLBI fringes observed using ALMA/APEX baseline (ALMA Phasing Project milestone) by Feb Proposal submission process successfully completed by April. Last hours were uneventful and system correctly coped with high demand Proposals Review Committee meeting successfully completed by June. Small issues were resolved swiftly during the meeting Sub-arrays capability delivered and ready to begin acceptance Cycle 3 Observations started in October. Early software issues being monitored and resolved 2015 in a nutshell

7 Incremental software releases were verified (ICT) and validated (Science) during the year. These releases included online and offline functionality Acceptances CYCLE3-OFF accepted for Cycle 3 Call for Proposals Included also updates for PT, SLT, AQUA, Ph1M, User Registry, NGAS, and Pipeline infrastructure CYCLE3-OFF (based on and.3) accepted for Cycle 3 APRC meeting Included also updates for the OT Phase 2 Preparation for Cycle CYCLE3-OFF (based on ) accepted for offline applications to be used during first part of Cycle 3: New versions of PT, AQUA, AQ, SLT, OT, deployed as part of this acceptance Software Releases (I)

Next offline acceptance planned for March CYCLE3-ON release accepted for performing CYCLE-3 online observations: CONTROL, TELCAL, BL CORR (single array) accepted without significant issues. BL CORR sub arrays: Most significant issues were resolved and final acceptance is planned for November or December depending on pressure form PI projects 2015 ACA: HW and SW issues are currently under investigation. Dedicated mission planned for end of November Software Releases (I)

To allow sub-arrays, certain functionality was migrated to a layer which performs slightly worse in very limited cases Improvements were made to allow for better linear scaling of the autocorrelations Hardware and Software problems have prevented the completion of ACA acceptance Action plan has been defined (next slide) ACA enhanced capabilities in Cycle 3

Enhanced 3 bit linearity correction by using digital power meter reading Finer time resolution to update correction value 1920ms reduced to 1ms: now applicable even for fast raster scan Introduction of 4 bit linearity correction Frequency profile synthesis (FPS) is now done in the CDP computer (before was in ACA Correlator hardware) Finally sub-array is available in ACA Hardware and Software problems have prevented the completion of ACA acceptance Action plan has been defined (next slide) ACA enhanced capabilities in Cycle 3

Recovery Mission Plan: Send both ACA Subsystem Scientist and ICT Group Lead to the OSF by Nov/Dec Create hot-line between OSF and NAOJ to get every technical support, and, Extensive test on AOS2-STE without any interaction with Science work. ACA trouble shooting toward Cycle 3

Implemented for Cycle 2 and 3 Ph1M using an algorithm developed by ALMA science operations Algorithm presented to ESAC and ASAC No comments have been received to date Stats* show that algorithm is not producing the expected results False positives: 98% (flagged as potential duplicates, but were not duplicates in the end) False negatives: 65% (flagged in the end as duplications, but not detected by the tool) Duplication checking for OT/Archive has therefore been delayed Infrastructure to use the same algorithm as Ph1M is in place ALMA science operations must come up with a better algorithm Once this is done, computing work involved in integrating it to Ph1M, OT and Archive is relatively straightforward Duplication checking * From Cycle 3 Ph1M, in comparison with recommendations of the ARP/APRC (not quality checked!)

AQUA Used daily for the QA0 process Support for QA2 is being developed, requires changes in the life- cycle of Observing Projects and support from the Pipeline Trending to be included later, requires efficient query of produced data SnooPI Replaces the current “public view” of the Project Tracker To be delivered 2016/Q1 Modern “Single Page” application with REST-ful back-end AQUA and SnooPI

SnooPI

SnooPI (2)

Control Cycle 3 - focus was on stability, including increased robustness to hardware errors Cycle 4 will be delivered for testing in December (including portion for solar observing) Emphasis on reducing the bug backlog, which should also help addressing operational reliability Correlator Correlator - delivered, commissioning team recommended acceptance Will continue with sub-arrays, any high-impact bugs found will be addressed at a high priority vs. other correlator bugs Control

Dynamic Scheduling Algorithm and What To Observe tools were run in parallel in the second half of Cycle 2 – list of enhancements were generated and prioritized Important enhancements were implemented - DSA is on track for being used in Cycle 3 ICT and ISOpT collaborating to improve usability of Scheduling for Astronomers on Duty Scheduling

Separate presentation prepared by Jeff Kern Pipeline

Additional material

The core activities are in the following categories:  Software maintenance:  Corrective, adaptive, perfective, and preventive maintenance  Further development on deliverables:  Completion of construction features  Internally funded features  Externally funded features  Release planning, integration and testing  Software fault tracking (troubleshooting, analysis, and diagnosis)  Software operations support The ICT Groups at the Executives are primarily responsible for first two categories, each one for their software deliverables in construction; and ICT-CL is primarily responsible for the remaining categories. ICT Core Activities

 Software Requirements Gathering  Key aspects described in the Software Requirements Management Plan  Coordinated and refined through a number of Planning, Coordination and Review Meetings  ALMA Software Delivery Process  A strict and well-defined release and acceptance process for delivering software to the ALMA observatory.  Fully defined in the document with the same name, largely based in incremental releases  The ICT Release Manager and the JAO Acceptance Manager organize this process.  Change Request Processes  All requests for change in requirements or scope for ICT software are dealt with by the Software Change Control Board (SCCB) ICT Processes

ALMA Operations

ALMA Project Lifecycle

Note that the number of products or services is no indication on complexity or workload.  ACA Control (ICT-EA):  ACA 7m/12m Antenna Software, ACA Correlator Software, ACA DMC Software, NAOJ Holography receiver Software  Common Infrastructure (ICT-EU):  ACS GUIs, Tools and Utilities, ACS Manager, Component, Container, Client, ACS Third Party Products, Alarm System and Monitoring, ALMA Science Archive Query Interface, ALMA Science Data Model, Archive Online Chain, Archive Pipeline Integration, Bulk Data Handling, Data Packer, Data Tracker, Harvester, Logging and Error System, Monitor Data Store, NGAS Archive, Notification Channel, Oracle Archive, Project Code Generator, Request Handler, Source Catalog, TMCDB  Control (ICT-NA):  Antenna, LO and Timing Hardware, Baseline Correlator Software, Data Capturer, Hardware Configuration Database, Hardware Monitor and Control Points, Alarms, Quick Look  Data Processing (ICT-NA):  CASA, Science Pipeline Products and Deliverables (I)

 Integration and Release Management (ICT-CL):  ALMA Software Build Procedures, Release Management, Standard Test Environment, Standard Web Environment  Observatory Interfaces (ICT-EU):  ALMA Dashboard, ALMA Quality Assurance (AQUA), Integrated Reporting, Internal Infrastructure, Observing Project Lifecycle, Observing Tool (OT), Operator Master Console (OMC), Phase 1 Manager (Ph1M), Project Tracker, Registration, Science Portal, Sensitivity Calculator, Shift Log Tool, Single Sign-On, Submission Service, User Registry, Web Shift Log Tool  Scheduling (ICT-NA):  Online Scheduler, Scheduling Planning Tool  Software Engineering and Quality Management (ICT-EU):  Auxiliary Services (NRI, Twiki, JIRA), Computing Standards, Makefile and Build System, Operating System, Packages and Utilities, Software Repository, Site Replication, Standard Test Environment Support, User Accounts and Mailing Lists  Software Operations (ICT-CL):  Hardware Monitoring, Operations Software Support Products and Deliverables (II)

 Telescope Calibration (ICT-EU):  Antenna Positions, Delay Scan Results, Holography, Online and Offline Corrections, Pointing Models, WVR Corrections Products and Deliverables (III)

The Atacama Large Millimeter/submillimeter Array (ALMA), an international astronomy facility, is a partnership of Europe, North America and East Asia in cooperation with the Republic of Chile. ALMA is funded in Europe by the European Organization for Astronomical Research in the Southern Hemisphere (ESO), in North America by the U.S. National Science Foundation (NSF) in cooperation with the National Research Council of Canada (NRC) and the National Science Council of Taiwan (NSC) and in East Asia by the National Institutes of Natural Sciences (NINS) of Japan in cooperation with the Academia Sinica (AS) in Taiwan. ALMA construction and operations are led on behalf of Europe by ESO, on behalf of North America by the National Radio Astronomy Observatory (NRAO), which is managed by Associated Universities, Inc. (AUI) and on behalf of East Asia by the National Astronomical Observatory of Japan (NAOJ). The Joint ALMA Observatory (JAO) provides the unified leadership and management of the construction, commissioning and operation of ALMA.