SOFIA DCS History and Overview Ian Gatley

SOFIA March DCS Preliminary Design Review2 The South Pole CARA Project: A DCS demonstration A data cycle system was produced by the RIT consortium for the CARA project (Harper, PI) at the South Pole. The South Pole DCS has the following history: –The South Pole IR telescope operated for 5 years as a PI project. –NSF indicated that the project needed to make telescope time with their camera publicly available. –NOAO agreed to run the proposal selection. –NOAO selected 21 proposals, 13 of which were actually implemented. –Gatley, a member of the CARA consortium, and his colleagues at RIT developed and implemented the data cycle pipeline for this experiment.

SOFIA March DCS Preliminary Design Review3 South Pole DCS: Implementation Both raw and calibrated data were provided to the PIs. Data were archived so they could be used for further global processing –For example, Kastner kept track of L-band photometry in order to assess the site's performance at this band. Some months of observations (IR imaging) took 1 month of processing at RIT. The process was not optimized, and it was written only for the CARA mode of observing. Rhody and Gatley spearheaded the project. Kastner (who planned to use the data, and who came to RIT as a faculty member in July 1999) improved and refined the process by interacting with Rhody, and by actually using the pipeline.

SOFIA March DCS Preliminary Design Review4 South Pole DCS: Philosophy Commercial software tools were used whenever possible. Software written by the consortium defines the interaction between tools and/or packages and the data. In addition, standard packages employed in other astronomy pipelines were used. Examples of commercial software include: –XML (extensible markup language); –CORBA (Common Object Resource Broker Architecture); –IDL Lib_Astro; –JAVA; –SQL (a database query language)

SOFIA March DCS Preliminary Design Review5 South Pole DCS: Portability and Scalability Because this was a straightforward application, Harvey Rhody had all of the DCS for CARA on his computer at RIT. The system is portable, and could be configured to run on several resident computers with balanced loads or on separated computers over the internet. This experiment represents a proof of concept: –the RIT consortium has designed and executed a DCS.

SOFIA March DCS Preliminary Design Review6 SOFIA DCS History HAWC presented their pipeline designs to SOFIA because: –Gatley and Harper (the South Pole DCS principals) were also principals in the HAWC experiment, –Harper (PI HAWC) proposed an end-to-end design, –Gatley and Harper had already teamed to successfully implement the data pipeline for the CARA project. Becklin was interested in a common pipeline approach for all the SOFIA instruments Since RIT already had demonstrated competence, they were asked to describe plans for a DCS to SOFIA

SOFIA March DCS Preliminary Design Review7 Slides from Becklin presentation Nov 98

SOFIA March DCS Preliminary Design Review13 Conceptual Design Review (June 1999)

SOFIA March DCS Preliminary Design Review14 The CoDR material is on the web Username = science Password = sofia

SOFIA March DCS Preliminary Design Review15 The DCS Mission The DCS shall provide the infrastructure to support the operation of facility instruments on SOFIA using best current knowledge and tools, and supporting continuous improvement in an efficient, extensible, and modular architecture.

SOFIA March DCS Preliminary Design Review16 Results of Conceptual Design Review

SOFIA March DCS Preliminary Design Review17 Need for DCS The CoDR committee finds that a DCS is necessary for making the facility instruments on SOFIA accessible to the largest possible astronomy community. Therefore, the Director should consider the development of the DCS of equivalent importance to the development of the FSIs.

SOFIA March DCS Preliminary Design Review18 Recommendations of CoDR Committee 1. The USRA should commission the development of a DCS, and the first priority for the DCS is the recording and archiving of all data necessary for complete analysis, calibration and future access to observational data obtained with the FSIs. 2. In order for the FSIs to be readily accessible to the community of US astronomers, the second priority for the DCS is to establish a select number of observing modes supported by pipeline reduction, including quicklook analysis, for each of the FSIs.

SOFIA March DCS Preliminary Design Review19 Recommendations of CoDR Committee 3. The Director should enable a close working relationship between the DCS team(s) and the FSI teams. 4. The DCS should become the vehicle for providing on-line documentation and information necessary for the GI to propose and carry out observations with the FSIs. 5. The DCS should be implemented as a system of distributed modules, integrated by an extensible hardware and software infrastructure, as presented at the DCS Conceptual Review.

SOFIA March DCS Preliminary Design Review20 Top Level Requirements The recommendations of the CoDR committee ARE the top level requirements for the DCS

SOFIA March DCS Preliminary Design Review21 Top level requirements (1) The DCS shall record and archive all data necessary for complete analysis, calibration and future access to observational data obtained with the FSIs. The DCS shall support a select number of observing modes with pipeline reduction, including quicklook analysis, for each of the FSIs.

SOFIA March DCS Preliminary Design Review22 Top level requirements (2) The DCS shall provide on-line documentation necessary for the GI to propose and carry out observations with the FSIs. The DCS shall be implemented as a system of distributed modules, integrated by an extensible hardware and software infrastructure.

SOFIA March DCS Preliminary Design Review23 The DCS in plain English The program architecture planned consists of "dumb" modules (e.g. the parser) that need descriptions of what to do, and where and how to find the appropriate tools. Scripts are written which are particular to the data-taking mode for the instrument, and which utilize these tools on the data. All of these components are also modular. The scripts can be written by RIT (or the PI).

SOFIA March DCS Preliminary Design Review24 The CARA example Experiments “scripted” for execution Observations made by the operator (Charlie Kaminsky). –Standard modes of operation, common standards Data sent by satellite to White Sands and then by Internet. Dark-frame subtraction, linearization, flat-fielding, co-addition, background-subtraction etc., performed as required, using IDL. –Saved the method used, the raw data, and the final product intermediate products not generally retained; intermediate products from some point in the path can be retained if desirable Images calibrated. Data written in the desired format*. * There would be a stage to put the product into IPAC archival form for SOFIA Data archived. Data products disseminated.

Overview of the PDR The "Core Development" DCS has been provided priorities by the June '99 Conceptual Design Review. This core development must have the capability to capture the data, reduce it, and provide documentation, all within a modular infrastructure utilizing extensible software/hardware tools. The DCS consortium will present their view of the required large components, and the software tools and programs needed to get the components to work together efficiently in the pipeline. The DCS consortium will also present the collected requirements for support of extensibility, including (presently unfunded, but envisaged): expanded scope for facility instrument support; additional participation (universities, ARC, GSFC, and IPAC); tools for USRA FSI flight planning; tools for USRA to manage the request for general investigator telescope time; tools for visualizing the use of SOFIA facility instrumentation; unified approach to USRA control of the facility instruments; Support of PI instruments.

SOFIA March DCS Preliminary Design Review26 Interface Management and Continuous Improvement With Facility Science Instruments With MCCS With Archives (IPAC, etc) With General Investigators Supporting change and continuous improvement –The strength of SOFIA and the fundamental difference between this and the other Great Observatories.

SOFIA March DCS Preliminary Design Review27 Summary Proof of concept exists and this team has track record. But the scope of the CARA DCS was very much less than that of the SOFIA DCS. The initial design of the SOFIA DCS is focussed on one mode per FSI instrument, and deals solely with data collection, reduction, and archiving. The core DCS will be delivered to the SSMOC, and the next stages of the USRA program will build upon that deliverable. An augmentation proposal (Becklin PI) is being submitted so that the SSMOC can support DCS expanded scope.