1. Russ Laher and Merhdad Moshir (1) S12.0 Test Readiness Review, March 8, 2005 Downlink Segment Test Status (S12.0) Russ Laher and Mehrdad Moshir March 8, 2005

2. Russ Laher and Merhdad Moshir (2) S12.0 Test Readiness Review, March 8, 2005 Summary D/L-group software changes implemented for S12.0  105 pipeline-related CRs  67 database-related CRs/AIs  31 ARs D/L segment-testing to check specific CR/AR/AI changes  Small cluster of 12 pipeline-drones (mini “ops-like” environment)  SODB (fresh S11 database via EdJackson)  Archiver run on pipeline results for end-2-end verification ISTs doing science verification of pipeline products and testing of SPICE QA team verification of QA sub-system upgrades Test plan documents written by CogEs to help I&T

3. Russ Laher and Merhdad Moshir (3) S12.0 Test Readiness Review, March 8, 2005 What’s New for S12.0 Functionality/infrastructure to process MIPS-24 SUR-mode TPM data Processing of MIPS-24 flatfield AORS through both flatfield and science pipelines New MIPS-24 automated “spot” and scan-mirror-dependent flatfield picking New MIPS-24 scan-leg pointing-refinement pipeline MIPS:Ge calibration products are now being archived! New IRAC and MIPS-24 sky median product Post-BCD IRAC position-refinement pipeline now generates warning, not error, when number of input images < 5 Pipeline-picker upgrade to handle IRS PeakupImage data Improved and more flexible ensemble creation New database functionality  new vBest’ing machinery for telemetryFiles records  getCalProducts now returns files with specified releaseLevel range  12 new QA diagnostics are now being populated in Qa_ptg-xfer table Ingest now populates SCLK, Frametime, and exposuretime for IRAC in DCEs DB table, to be used for simplifying IRAC first-frame-correction DB query Updated the following software modules: irs_tune, droopop, profile, basecal, ridge, extract, qaloader, darkest, slremov, snestimator Various updates to wrappers, cal files, CDFs New off-line superboresight pipeline: refines original boresight pointing-history file by mapping the IRAC pointing-refinement results back to the original pointing-history data; aim is to use for re- processing; more testing is needed before it can be used in operations SPICE upgrades for picking version-dependent calibration files, selecting input files via HTTP, and using a remote server for processing QA sub-system upgrades to SDAT, database functionality, and other web-based tools

4. Russ Laher and Merhdad Moshir (4) S12.0 Test Readiness Review, March 8, 2005 Tests Under S12.0 IRS  All Sci (BCD&BQD), DarkCal, DarkEns pipelines; used fallback cal; all data run through archiver  Details in IRS S12 Test Plan Document MIPS-Ge  Sci (scanmap, photometry, sed, tpm), darks, flats; used fallback cal; all data run through archiver  Details in MIPS-Ge S12 Test Plan Document MIPS ‑ 24  SUR-mode sci scanmap (slow, med, fast cases); SUR- and RAW-mode photometry; SUR- and RAW-mode TPM; flat-field cal w/ pointing transfer, non-linearity cal w/ pointing transfer; SUR & RAW dark cal w/ pointing transfer ; used fallback cal; all data run through archiver  Details in MIPS-24 S12 Test Plan Document IRAC  sub-, full-array science and cal pipelines for all exposure modes, frame times, and combinations of internal and external repeats; product records and file system contents checked for data run thru each pipeline; used metadata and fallback cal; all data run through archiver  Details in IRAC S12 Test Plan Document Post-BCD  FIFcreation; pointing refinement;single_frame_pnt_src_ext;multi-frame pnt-src_ext; mosaic pnt- src_ext; overlap correction; image_interp, outlier; image_reinterp; mosaic_coadd for IRAC, MIPS24, 70 &160; all data run through archiver  Details in PostBCD S12 Test Plan Document QA, SDAT  Loader run for all tested pipelines, SDAT tool run, qa pages and statistics pages brought up  Details in QA/Infrastructure S12 Test Plan Document

5. Russ Laher and Merhdad Moshir (5) S12.0 Test Readiness Review, March 8, 2005 S12.0 Functions/Test Data Flows

6. Russ Laher and Merhdad Moshir (6) S12.0 Test Readiness Review, March 8, 2005 S12.0 Capabilities Sub-systemCapability IngestSupports DCEs, Pntng, ECSV, Rpt files, Mem Dmps. Sped up, SigEvents, FEIdelete, ReRun, Archiver naming, HK compression Pipeline Execreal-time data reduction; reprocessing Instrument pipelinesAll three instruments science and calibration Calibration TransferFall-back rules (incl NIT) for all three instruments, three nearest in time rules for all three instruments, blessed status flag, asymmetric discriminant selection, interpolation rule Instrument Performance MonitoringCreate/Populate HK tables in the HK DB; web-based viewing and data retrieval; updated menus Quality AssessmentCreation and populating of QA tables; SDAT web-based QA inspection; analysis and display functions; support archive release CDFPopulating of version controlled CDF files on the pipeline drones; retrieve CDF in pipelines, version tracking in the SODB Imaging post-BCD pipelinePoint source extraction, pointing refinement (w/ & w/out 2MASS, USNO, UCAC, FIRST), tiling, overlap correction, outlier identification and mosaic coadding, bandmerging

7. Russ Laher and Merhdad Moshir (7) S12.0 Test Readiness Review, March 8, 2005 S12.0 Success Criteria Run, re-run requests that failed on OPS Interact with the SODB via perl & c-APIs Deposit raw data into the archive file system Create pipeline manifests & ensembles Run manifests through APES Run multiple jobbers on 12 pipeline drones Use CDF to get configuration files Use CalTransfer to get calibration data (NIT and/or fallbacks) Execute post-BCD pipelines including pointing refinemnet Put QA statistics into QA tables Put processing results in sandbox and set file pointers in DB Use SDAT to access and view results on segment test Get “better” BCDs & post-BCD prods IST sci verification of seg test data & buy off (incremental from IOC) Pass the running of archiver

8. Russ Laher and Merhdad Moshir (8) S12.0 Test Readiness Review, March 8, 2005 Applicable SISs Calibration SOSDL-SIS-CL-3000 : darkcal output FITS file, dark image SOSDL-SIS-CL-3001 : lincal output FITS file, linearity model cube SOSDL-SIS-CL-3002 : lincal output table file, pixel cal data SOSDL-SIS-CL-3003 : lincal output cmask image SOSDL-SIS-CL-3004 : darkcal output cmask image SOSDL-SIS-CL-3005 : pmask definitions SOSDL-SIS-CL-3006 : Electronic Nonlinearity Calibration File SOSDL-SIS-CL-3007 : MIPS-Ge stim_latent SOSDL-SIS-CL-3008 : function loadcalkeywords SOSDL-SIS-CL-3009 : IRAC-loadcalkeywords version-1 SODB Table SOSDL-SIS-CL-3010 : darkest cmask SOSDL-SIS-CL-3011 : CalTrans Rule List SOSDL-SIS-CL-3012 : IRS Umask SOSDL-SIS-CL-3013 : IRS Lmask SOSDL-SIS-CL-3014 : IRAC flatcal data cube SOSDL-SIS-CL-3015 : IRAC flatcal cmask SOSDL-SIS-CL-3016 : IRS flatcal cmask SOSDL-SIS-CL-3017 : IRS Omask SOSDL-SIS-CL-3018 : IRS Flux Conversion Table SOSDL-SIS-CL-3019 : IRS FOV PSF Table SOSDL-SIS-CL-3020 : IRS COADD2D AMASK SOSDL-SIS-CL-3021 : IRS COADD2D FMASK SOSDL-SIS-CL-3022 : IRS WAVSAMP TABLE SOSDL-SIS-CL-3023 : MIPS-Ge cmask Data Tools SOSDL-SIS-DT-3004 : profile.tbl (IRS BQD) SOSDL-SIS-DT-3005 : ridge.tbl (IRS BQD) SOSDL-SIS-DT-3006 : extract.tbl (IRS BQD) SOSDL-SIS-DT-3007 : irs_tune.tbl (IRS BQD) Executive SOSDL-SIS-EX-3000 : plinexyz.cdf SOSDL-SIS-EX-3001 : run time environment variables Quality Assessment SOSDL-SIS-QA-3000 : QA file (*.qa) format Products SOSDL-SIS-PD-3000 : cvti2r4 output FITS file, 32-bit real data SOSDL-SIS-PD-3001 : cvti2r4 output FITS file, dmask SOSDL-SIS-PD-3002 : muxbleedcorr output DCE FITS file SOSDL-SIS-PD-3003 : imfliprot output FITS file SOSDL-SIS-PD-3004 : MIPS Ge bmask SOSDL-SIS-PD-3005 : imagest output bmask SOSDL-SIS-PD-3006 : tranhead keyword-translate-only list SOSDL-SIS-PD-3007 : Tile/BCD association files SOSDL-SIS-PD-3008 : Mosaic-geometry interface SOSDL-SIS-PD-3009 : BCD delta file SOSDL-SIS-PD-3010 : PREDICTSAT s-mask SOSDL-SIS-PD-3011 : dldeltafile library SOSDL-SIS-PD-3012 : PRF/PSF map SOSDL-SIS-PD-3013 : Point source extraction table SOSDL-SIS-PD-3014 : Detection/Tile/BCD Association File SOSDL-SIS-PD-3015 : imagest output slope image SOSDL-SIS-PD-3016 : latimflag l-mask SOSDL-SIS-PD-3017 : fiducial frame definition parameters SOSDL-SIS-PD-3018 : mosaic_interpolation table SOSDL-SIS-PD-3019 : matched-point-source refined position table SOSDL-SIS-PD-3020 : fiducial frame input parameters SOSDL-SIS-PD-3021 : bandmerged point source output file SOSDL-SIS-PD-3022 : bandmerger pointer dump file SOSDL-SIS-PD-3023 : band-pair registration uncertainties SOSDL-SIS-PD-3024 : bandmerge statistics SOSDL-SIS-PD-3025 : MIPS24 bmask SOSDL-SIS-PD-3026 : FPGen Command Line Input SOSDL-SIS-PD-3027 : FPGen SODB Entries file (input) SOSDL-SIS-PD-3028 : FPGen Template Input file SOSDL-SIS-PD-3029 : Bandmerge (QA) spectral combination counts Pointing Transfer SOSDL-SIS-PT-3000 : pixel-scale table file SOSDL-SIS-PT-3001 : Ptg Refinement Absolute Source List Input Fmt SOSDL-SIS-PT-3002 : MIPS Scan Mirror Parameters Table SOSDL-SIS-PT-3003 : Instrument FOV Table SOSDL-SIS-PT-3004 : Super Boresight Pointing History File SOSDL-SIS-PT-3005 : Refined DCE Position Table File

9. Russ Laher and Merhdad Moshir (9) S12.0 Test Readiness Review, March 8, 2005 Pipeline-Related CRs (105)

10. Russ Laher and Merhdad Moshir (10) S12.0 Test Readiness Review, March 8, 2005 Pipeline-Related CRs (105)…

11. Russ Laher and Merhdad Moshir (11) S12.0 Test Readiness Review, March 8, 2005 Database-Related CRs/AIs (67) AI #1469 - Change in product types table needed AI #1449 - Ensemble-creation rule changes for MIPS:Si AI #1421 - Change release level from 99 to 100 for a few archive Pids AI #1402 - Data content for new ensRules DB table (set make='f' for ruleId=350) AI #1388 - Add "bcd_median.fits" and "bcd_median_uncert.fits" to ProductTypes AI #1364 - Backfill externalName fields in S12.0 database AI #1363 - Data content changes to productTypes database table (S12.0 DB only) AI #1338 - Remove ruleId=226 from ensRules and ensplScripts DB tables, etc. AI #1212 - Backfill vBest in telemetryFiles database table AI #1219 - New data contents for calExposureTypes database table AI #1215 - New operations-TFS fileType "calexposuretypes" AI #1212 - Backfill vBest in telemetryFiles database table AI #1201 - Update DB productTypes table with comments AI #1191 - Data content for new ensRules and ensPlScripts DB tables AI #1172 - update the product types table per CR 6073 CR #6419 - getCalPRoducts SP needs two release levels min and max CR #6374 - Update instrumentId in runPipelinePicker DB stored-functions CR #6370 - dnlProc_CR6370.sql, addQA_ptg_xfer_CR6370.pm CR #6368 - dnlProc_CR6368.sql (loading mips1fallback) CR #6367 - QA_ptg_xfer_CR6367.sql CR #6363 - Modify pipeline picker to handle new IrsPeakupImage mode CR #6360 - New overloaded updateDce DB stored procedure for Ingest CR #6328 - New S12.0 database capabilities for QA subsystem CR #6327 - New database role qaSuperUserRole CR #6325 - New database table QA_superUsers CR #6324 - New perl-API function getQaAnalystsAndSuperUsers CR #6323 - New DB stored-function getQaAnalystsAndSuperUsers() CR #6318 - New overloaded setDceTimeParameters DB stored procedure - REJECTED CR #6307 - Schema update for mips1fallback in ops and sodb_it databases CR #6284 - New Ingest C-API for populating Dces database table for IRAC CR #6283 - New Dces.atimeend field in database (integer) - REJECTED CR #6282 - Update PlScripts.plscriptnum for plscriptid=1042 CR #6281 - Make plScripts.comment more precise for plScriptId=33 CR #6270 - Remove ruleId=226 and update ensemble-creation document. CR #6250 - vBest'ing machinery for ensembleSets CR #6242 - Modify createEnsembles.pl to have -u option CR #6223 - New database stored function getQaStatusForRequest CR #6224 - New database stored procedure setQaStatusForRequest CR #6225 - New perl-API function getQaStatusForRequest CR #6226 - New perl-API function setQaStatusForRequest CR #6212 - PlScripts-table data-content changes to undo AI #1183 CR #6210 - New DB stored-function getBestTelemetryFilesForCampaign CR #6188 - New perl-API function getRefinedPointingFilesForCampaign CR #6187 - New DB stored-function getRefinedPointingFilesForCampaign CR #6180 - Upgrade perl-API function lockEnsembleProductsVBest CR #6179 - Upgrade DB stored-function lockEnsembleProductsVBest CR #6173 - New perl-API function unlockEnsembleProductsVBest CR #6148 - Add vBest'ing machinery to DB SP registerTelemetryFile CR #6136 - Upgrade database stored-function getDceIdsForRequest CR #6123 - add plscriptId=1550 and 1650 with exptype=ffl CR #6120 - Upgraded perl-API function setQaStatusForCampaignAndPid CR #6106 - New perl-API function getQaStdComments CR #6105 - New database stored-function getQaStdComments CR #6104 - Upgrade stored-procedure setQaStatusForCampaignAndPid CR #6093 - New perl-API function getQaStatusForCampaignAndPid CR #6092 - New DB stored-function getQaStatusForCampaignAndPid CR #6077 - New perl API getLatestEnsembleProduct CR #6076 - New DB stored-function getLatestEnsembleProduct CR #5913 - Mod. to ensemble-creation ruleId=110 CR #5827 - Overloaded addEnsemble and addEnsembleNTC DB stored functions CR #5818 - New perl-API functions for ensemble creation CR #5816 - New DB stored procedures for ensemble creation CR #5815 - New database tables for ensemble creation CR #5738 - New "ruleId" field in Ensembles tables CR #5614 - Stored functions for unlocking vBest CR #5092 - Schema update for QA_irsflatcalprep CR #1209 - Define and implement database procedures for ensemble processing

12. Russ Laher and Merhdad Moshir (12) S12.0 Test Readiness Review, March 8, 2005 ARs (31) 5847 Archive Laher_Russ Archive system does not find fallback calibration files for MIPS-Ge

13. Russ Laher and Merhdad Moshir (13) S12.0 Test Readiness Review, March 8, 2005 Issues to be tackled in S13 SDAT fully functioning under Apache proxy server Improvement of MOPEX documentation for March 22 Observers’ Support Workshop and more extensive improvements over the next few months Port post-BCD tools to Max OS 10 (goal is by August Observers’ Support Workshop Database Query of IRSPeakUpImage database table for Final Product Generation – is this needed? Simplified time-delay calculations for IRAC first-frame correction MOPEX GUI development Bandmerge upgrade

14. Russ Laher and Merhdad Moshir (14) S12.0 Test Readiness Review, March 8, 2005 Lines of code

15. Russ Laher and Merhdad Moshir (15) S12.0 Test Readiness Review, March 8, 2005 Lines of Code

16. Russ Laher and Merhdad Moshir (16) S12.0 Test Readiness Review, March 8, 2005 S12.0 LOC breakdown

17. Russ Laher and Merhdad Moshir (17) S12.0 Test Readiness Review, March 8, 2005 Sample of new tool for setting QA status

18. Russ Laher and Merhdad Moshir (18) S12.0 Test Readiness Review, March 8, 2005 Sample of new ptg diff Q/A statistics

19. Russ Laher and Merhdad Moshir (19) S12.0 Test Readiness Review, March 8, 2005 Sample of new ptg diff Q/A statistics