Developing JWST Pipelines at STScI Robert Jedrzejewski

Who we are The Science Software Branch at STScI 16 members Most have an astronomy background 6 have PhDs Combined experience in group: 125 years Combined experience at STScI: 200 years

What we do Develop HST calibration pipelines STSDAS/TABLES PyRAF, PyFITS, STScI_Python HST Exposure Time Calculators Other smaller projects (Gemini/GOODS/Hubble Legacy Archive/GoogleSky/JWST Backplane Stability…)

Development Experience Python Java C/C++ Fortran spp/cl IDL (Perl/Assembly/Tcl…)

Our preferred development model Python! We find we can be extremely productive writing in Python Speed is occasionally an issue, so we use C extensions when necessary Very little pipeline code requires performance optimization

Development style Use version control (subversion) Use regression tests + nightly builds + web reporting tools Trac for problem tracking/wiki for information dissemination Unit/doc tests Multiple platforms (Linux/Mac/Solaris/Windows)

How we did HST pipelines Calfoc, calfos, calhrs, calwfpc, calwp2 –First generation pipelines, written in spp, read GEIS files Calstis, calnic(a/b) –Second generation, written in C using hstio (which wraps IRAF imio libraries) to read multiple extension FITS files Calacs –Borrowed much code from calstis imaging Calwfc3 –Borrowed much code from calacs, calnic Calcos –Third generation, written in Python (+ c where needed) Later pipelines were more likely to be used by IDTs for calibrating ground test data

More on HST pipelines Pipeline operation is data-driven –Calibration steps as header keywords: FLATCORR=PERFORM/OMIT/COMPLETE/SKIPPED –Reference file names as header keywords FLATFILE=oref$g _flt.fits This decouples some of the intelligence from the code –No need to rebuild code if step or reference file changes

Multidrizzle Multidrizzle is used by the ACS and WFPC2 pipelines to combine images with small position offsets (dithered), removing cosmic rays It is a Python application that can be used with ACS, STIS, WFPC2, NICMOS and WFC3 data This breaks from our ‘tradition’ of having 1 calibration pipeline program for each instrument

How we see the JWST Pipelines A series of calibration steps Calibration Step Input stage Output stage Reference File

Early design ideas No need to have separate pipeline programs for each JWST instrument –Many calibration steps depend on detector, and JWST instruments use detectors of the same type –We can use the same code, instead of having to replicate it (and maintain it) in more than one place –Some calibration steps will probably be identical for all JWST data (e.g. the MASKCORR step, where a static mask from a reference is applied to the DQ array of the data)

Try not to make the mistakes we made with HST Use the same keywords for the same quantities Use the same file/association structure Use the same algorithms to do the same calibration –Unless a team shows that a given algorithm does not work for their instrument –Even then, try and keep as much code common as possible, only breaking out the code that is different –Sometimes it is possible to encapsulate the differences in the reference files, keeping the code the same

JWST Pipelines (continued…) Python gives us object-oriented capabilities –‘input_stage’ and ‘output_stage’ are objects that encapsulate information on their state and on how to calibrate themselves –For example, they might be NIRSPEC IFU data objects, or MIRI imaging data objects –When executing a given step, they may use their own custom method, or else defer to a method that they inherit from a more ‘generic’ datatype –E.g. MIRI imaging data and NIRCAM imaging data may both use the flatfield() method of the JWSTImagingData class, from which they both inherit

JWST Pipelines (continued…) The inheritance hierarchy encapsulates information about what is the same and what is different about JWST data types –We can mix in behaviors from different types of object, as necessary –But, to the extent that is possible, we try and keep as much the same as possible –The people who inherit this project will thank us

What goes in? IDTs and instrument teams at STScI will figure out: –Which steps are needed, and their ordering –Which instruments/modes use the steps –What each step does –What calibration reference data are needed –What tests the code needs to pass

Facilitating the process Calibration data will be in a “public” repository This will include: –Code –Test data –Documentation

Facilitating… We will encourage everyone to try out our algorithms as we develop them And we encourage everyone to contribute their own algorithms We’ll handle keeping teams synchronized by versioning and providing different builds –E.g. Team A may still be testing build X, when team B needs to test the next stage of functionality in build X.1 –When Team B is ready to test the functionality in build X.1, there may already be build X.2 (which includes the functionality in build X.1 as well as new functionality) –In the end, all the teams will test the same code

Facilitating How do we know that the code does the ‘right’ thing? –Teams provide test data with test results –Then we know that the result is correct because it reproduces team-supplied answers –Test results could be actual data (e.g. FITS files) Pixels in pipeline-calibrated data should be identical within +/- –Or results of analysis Aperture photometry should be the same to within +/-

Interfacing with other languages If teams develop code that does a lot of fancy processing, we can try to include it by wrapping Python talks to C/C++ using C extensions An existing C function can be wrapped so that Python objects can be passed to C/C++, and C objects passed back to Python We can wrap relatively simple C functions –Arguments are arrays or primitive datatypes (integer/float/string…) –No objects as arguments –Structs are OK, as long as they are simple (flat) –Play nice with memory

Wishlists We don’t need to feel constrained by HST What are the biggest deficiencies in HST? –Best reference files and best calibration steps can be determined by querying a service Don’t need to rely on HST archive to find these out –Reference files can be downloaded as needed –Even calibration code can be updated as needed (don’t need to wait 6 months for the next STSDAS release)

Wishlists Tell us what you want! –The earlier the better –Some aspects of the overall architecture are still flexible And not just pipeline calibration code –We are going to need tools for data analysis, evaluation, interpretation, visualization –Reference file generation