SPACE TELESCOPE SCIENCE INSTITUTE Operated for NASA by AURA COS Pipeline Language(s) We plan to develop CALCOS using Python and C Another programming language? Why? - What is Python and why are we using it? Risks Development dependencies and schedules

COS PIPELINE CDR P. Greenfield July 24, 2001CALCOS Development Environment Space Telescope Science Institute 2 of 12 What is Python? An Open Source, freely available “scripting” language.An Open Source, freely available “scripting” language. Extremely productive.Extremely productive. Many powerful libraries available. Many powerful libraries available. Comparable to Perl in capabilities. Comparable to Perl in capabilities. Easy to learn and read—well documented with many books available.Easy to learn and read—well documented with many books available. Available on just about any computing platform.Available on just about any computing platform. Portable Portable Available on far more platforms than IRAF (e.g., MS Windows) Available on far more platforms than IRAF (e.g., MS Windows) Robust implementation Robust implementation Wide user and developer baseWide user and developer base Approaching a million users, and growing Approaching a million users, and growing Includes many large commercial and scientific users Includes many large commercial and scientific users Supports both procedural and object-oriented coding.Supports both procedural and object-oriented coding. Unlike Tcl or Perl, scales well to larger programming projects Unlike Tcl or Perl, scales well to larger programming projects Extendable with C, C++, or Fortran.Extendable with C, C++, or Fortran.

COS PIPELINE CDR P. Greenfield July 24, 2001CALCOS Development Environment Space Telescope Science Institute 3 of 12 Why are we using Python? The basis of the Science Software Group’s efforts to open up the IRAF environment by developing a new Command Language for IRAF using Python.The basis of the Science Software Group’s efforts to open up the IRAF environment by developing a new Command Language for IRAF using Python. The successful effort convinced us that Python was a desirable applications language for astronomy data analysis and reduction.The successful effort convinced us that Python was a desirable applications language for astronomy data analysis and reduction. Success of IDL in astronomy indicates that similar or better capabilities will be successful.Success of IDL in astronomy indicates that similar or better capabilities will be successful. Why not IDL?Why not IDL? Not freely distributable Not freely distributable Python is a much better programming language Python is a much better programming language Not as extendable as Python; it would not have been possible to implement CL for IRAF using IDL, for example. Not as extendable as Python; it would not have been possible to implement CL for IRAF using IDL, for example.

COS PIPELINE CDR P. Greenfield July 24, 2001CALCOS Development Environment Space Telescope Science Institute 4 of 12 Why we are using Python for CALCOS Python is already planned for other calibration pipelines: Python is already planned for other calibration pipelines: IRAF CL scripts are not suitable for pipeline use (no error handling!); Python scripts invoking IRAF tasks are the only alternative (short of rewriting script as one large executable). Python scripts are being used for automatic Reference file generation. IRAF CL scripts are not suitable for pipeline use (no error handling!); Python scripts invoking IRAF tasks are the only alternative (short of rewriting script as one large executable). Python scripts are being used for automatic Reference file generation. For similar reasons, implementation of ACS dither-combining capability requires Python wrappers for the Drizzle task (and now part of CALACS) For similar reasons, implementation of ACS dither-combining capability requires Python wrappers for the Drizzle task (and now part of CALACS) Python programs are more easily written, tested, and modified than C programs because of Python’s interpreted, dynamic and concise nature (5-10 times more productive).Python programs are more easily written, tested, and modified than C programs because of Python’s interpreted, dynamic and concise nature (5-10 times more productive). Instruments always have unanticipated calibration requirements. Python programs are generally much easier to adapt than C programs.Instruments always have unanticipated calibration requirements. Python programs are generally much easier to adapt than C programs.

COS PIPELINE CDR P. Greenfield July 24, 2001CALCOS Development Environment Space Telescope Science Institute 5 of 12 Why we are using Python for CALCOS (cont.) Python code is more easily understood and contributed to by instrument scientists than is C making it more likely they will contribute to the review and refinement of the algorithms.Python code is more easily understood and contributed to by instrument scientists than is C making it more likely they will contribute to the review and refinement of the algorithms. Previous calibration pipelines use ~80% of their code for bookkeeping and setup, which is much better accomplished by Python.Previous calibration pipelines use ~80% of their code for bookkeeping and setup, which is much better accomplished by Python. Python errors, when they occur, are much easier to diagnose than C errors, particularly C memory and pointer errors.Python errors, when they occur, are much easier to diagnose than C errors, particularly C memory and pointer errors. CALCOS will be portable to more platforms (e.g., MS Windows).CALCOS will be portable to more platforms (e.g., MS Windows). Unlike WFC3, no great amount of code reuse of existing C code possible.Unlike WFC3, no great amount of code reuse of existing C code possible. There are real risks in keeping the status quo!There are real risks in keeping the status quo!

COS PIPELINE CDR P. Greenfield July 24, 2001CALCOS Development Environment Space Telescope Science Institute 6 of 12 Non-Risks Safety (dynamic typing)Safety (dynamic typing) Speed (interpreted language)Speed (interpreted language) “Lack of experience”“Lack of experience”

COS PIPELINE CDR P. Greenfield July 24, 2001CALCOS Development Environment Space Telescope Science Institute 7 of 12 Risks I/O: Current C programs rely on IRAF for data I/O. But IRAF is not usually compatible with other programs such as Python.I/O: Current C programs rely on IRAF for data I/O. But IRAF is not usually compatible with other programs such as Python. Use of Python requires removing dependence on IRAF I/O.Use of Python requires removing dependence on IRAF I/O. Best solution is to have Python handle I/O (though CFITSIO is always a fallback).Best solution is to have Python handle I/O (though CFITSIO is always a fallback). SSG is developing the PyFITS module to do data I/O.SSG is developing the PyFITS module to do data I/O. Fully functional PyFITS requires a reworked array module to handle FITS tables.Fully functional PyFITS requires a reworked array module to handle FITS tables. The primary risk is the dependency on these modules.The primary risk is the dependency on these modules.

COS PIPELINE CDR P. Greenfield July 24, 2001CALCOS Development Environment Space Telescope Science Institute 8 of 12 Risks (cont.) Minor dependence on memory requirements. We assume workstations will have 1GB memory (enabling the ability to read whole data sets into memory for processing). This requirement will be reduced as new capabilities are added to PyFITS but these are not part of the baseline requirements.Minor dependence on memory requirements. We assume workstations will have 1GB memory (enabling the ability to read whole data sets into memory for processing). This requirement will be reduced as new capabilities are added to PyFITS but these are not part of the baseline requirements. Minor risk on updating changes to CALCOS based on changes to new modules. Since we control these modules the changes are expected to be small, especially later on in the coding phase. All such interface issues should be settled long before CALCOS is distributed to the user community.Minor risk on updating changes to CALCOS based on changes to new modules. Since we control these modules the changes are expected to be small, especially later on in the coding phase. All such interface issues should be settled long before CALCOS is distributed to the user community. Tru64 IEEE floating point has problems that must be addressed.Tru64 IEEE floating point has problems that must be addressed.

COS PIPELINE CDR P. Greenfield July 24, 2001CALCOS Development Environment Space Telescope Science Institute 9 of 12 Software Dependencies Numarray: New implementation of array module:Numarray: New implementation of array module: Status: Complete enough for CALCOS I/O use. Status: Complete enough for CALCOS I/O use. RecordArray: Needed for efficient table access.RecordArray: Needed for efficient table access. Effort: 8 FTE weeks Effort: 8 FTE weeks Started: 15 May 2001 Started: 15 May 2001 Status: completed. Status: completed. CharArray: Needed to access text fields in tables.CharArray: Needed to access text fields in tables. Effort: 3 FTE weeks Effort: 3 FTE weeks Started: 4 June 2001 Started: 4 June 2001 Status: completed Status: completed

COS PIPELINE CDR P. Greenfield July 24, 2001CALCOS Development Environment Space Telescope Science Institute 10 of 12 Software Dependencies (cont.) C API for Numarray: Needed by C code to access data in Python ArraysC API for Numarray: Needed by C code to access data in Python Arrays Effort: 1 FTE week Effort: 1 FTE week Started: 20 June 2001 Started: 20 June 2001 Status: completed Status: completed Integrate Numarray and other array classes with PyFITSIntegrate Numarray and other array classes with PyFITS Image Data Image Data Effort: 1 FTE week Effort: 1 FTE week Started: 18 June 2001 Started: 18 June 2001 Status: completed Status: completed Table Data Table Data Effort: 2 FTE weeks Effort: 2 FTE weeks Start: 1 Sept 2001 Start: 1 Sept 2001 Planned completion: 15 Sept 2001 Planned completion: 15 Sept 2001

COS PIPELINE CDR P. Greenfield July 24, 2001CALCOS Development Environment Space Telescope Science Institute 11 of 12 Software Dependencies (cont.) Schedules indicate completion of work needed for use with CALCOS. Most of these modules have other work planned to enhance their capabilities but are not required or essential for CALCOS.Schedules indicate completion of work needed for use with CALCOS. Most of these modules have other work planned to enhance their capabilities but are not required or essential for CALCOS.

COS PIPELINE CDR P. Greenfield July 24, 2001CALCOS Development Environment Space Telescope Science Institute 12 of 12 Dependencies Schedule DependencyStartFinishStatus Numarray 15 March 15 June Complete (for I/O) RecordArray 15 May 10 July Complete CharArray 4 June 15 July Complete C API 20 June 15 July Complete PyFITS/ImageIntegration 18 June 25 June Complete* PyFITS/Tablesintegration 1 Sept 15 Sept Not started.