IRAC-Data Processing Pipeline Review / January 26, 2001 Jason Surace (1) IRAC Pipeline Data Analysis and Pipeline Validation Plan Jason Surace January 26, 2001

IRAC-Data Processing Pipeline Review / January 26, 2001 Jason Surace (2) Pipeline Module Testing Nearly all IRAC pipeline modules have been tested in some way with “real” data, i.e. produced by the flight instrument. But, testing with real data has been somewhat haphazard and not very systematic. We have had to sift through 10s of Gb of test data. Instrument not completed or in stable environment - not clear what is best characterization data. Analysis has been lengthy - lots of problems have been turned up both by us and SAO-IT. IT is still analyzing data and working on characterizing instrumental signatures. Short on manpower. IT pipeline work pre-empted by instrument delivery. How to reach a point where we can check off each module as “finished”?

IRAC-Data Processing Pipeline Review / January 26, 2001 Jason Surace (3) Pipeline Validation Plan 1) Instrument team selects and provides a “validation” data set, where they have pre-analyzed the data. 2) SSC processes this raw data via pipeline. 3) This data is then analyzed by SSC-IST/SAO-IT. 4) If results match, we’re done. If not, we go back to IT and iterate by asking for a new algorithm or a clarification of requirements. In this way, the results of each module and thread will be validated and approved by IT. The pipeline has renewed visibility currently with IT as instrument construction ramps down.

IRAC-Data Processing Pipeline Review / January 26, 2001 Jason Surace (4) Example - LINCAL IRAC data must be corrected for non-linear response of detector. Science requirement is a very stringent 1% linearity over usable data range of detector. Previous LINCAL effort based on a multivariate function delivered by SAO: S'=S (C + A/Sqrt(B-S)) This was tested using the comprehensive performance test (CPT) data generated via ASIST at GSFC. This was intended to be the definitive test data set.

IRAC-Data Processing Pipeline Review / January 26, 2001 Jason Surace (5) Early Analysis Problems However, testing with CPT data showed data was too unstable to demonstrate requirement. Camera was turned on and off each day during multiday test, may have been other problems. Normalized mean count levels for supposedly identical linearity frames taken on different days during CPT. Note that on a given day the stability is very high (drift < 0.5% per 6 hours).

IRAC-Data Processing Pipeline Review / January 26, 2001 Jason Surace (6) Validation Data At SSC request, Rochester provided a new linearity dataset taken under laboratory conditions which they believe can demonstrate the 1% requirement. Jason writes new software to analyze this dataset and prototype several different ways to linearize the data. (See Input: TCAL with increasing exptimes Derived “linear” part (flat-field+lamp pattern) Derived “non-linearity” for rate method.

IRAC-Data Processing Pipeline Review / January 26, 2001 Jason Surace (7) A Comparison of Methods % error as a function of full-well depth for individual pixels. LINCAL (blue) shows behavior seen in previous tests. Scatter and offset are never below a few percent and hence fails the requirement - functional form does not fit well. Red is a new proposed quadratic solution. Both models break down completely within 10% of full- well capacity.

IRAC-Data Processing Pipeline Review / January 26, 2001 Jason Surace (8) So We Redo It New solution will be a quadratic function: S' = A (-A+sqrt((A^2)-(4*B*S)))/(2 * B)) Quadratic solution expected to exceed 1% requirement up to 90% full-well capacity.

IRAC-Data Processing Pipeline Review / January 26, 2001 Jason Surace (9) The Next Iteration There is a plan to acquire new linearity test data during CTA testing. We have a more uniform illuminator available, and the data will be taken in a flight-like manner and environment. The new LINCAL will be ready to test this once the data arrives. Rec/Del for all IRAC algorithms and data from SAO-IT is 02/01/01.