1. The Astro-wise pipeline Python Data processing - the low-level interfaces Data administration - persistent objects

2. Python ● Object Oriented scripting language ● clean, easy-to-understand syntax ● extensive library ● powerful built-in data types (str, dict, list, file) ● byte-code interpreted, dynamically typed Rapid development and easy maintenance

3. Python, a 1-slide course wordcount = {} for line in file('my_thesis.txt'): for word in line.split(): if word not in wordcount: wordcount[word] = 1 else: wordcount[word] += 1 print wordcount

4. Image processing ● eclipse libraray in C (ESO, N. Devillard) ● wrapped with SWIG ● fits images and headers as built-in data types >>> from eclipse import image, header >>> img = image('science.fits') >>> hdr = header('science.fits') >>> flat = image('flat.fits') >>> img = img / flat >>> hdr['FLATFLD'] = 'flat.fits' >>> img.save('flatfielded.fits', hdr) ● Co-addition with SWARP (IAP, E. Bertin)

5. Catalog Processing ● Sextractor (IAP, E. Bertin) ● LDAC (OmegaCAM, E. Deul) – Astrometry – Photometry ● Common interfaces, but no Python types ● Persistent Sourcelists (see KGB)

6. Astro-wise Pipelines ● Pipelines are about data-administration, not about data-processing ● Persistent objects – Object's state persists across program boundaries – Python classes representing SQL tables – Should also work without back-end ● Distinguish meta-data and bulk (FITS) data – meta data in persistent objects – bulk (FITS) data through file-server

7. The make metaphor ● 'Making' objects – think unix Makefile – targets and dependencies (recursive) >>> bias = BiasFrame() >>> bias.raw_frames = [RawBiasFrame('bias1.fits'), RawBiasFrame('bias2.fits'),... RawBiasFrane('biasN.fits')] >>> bias.make()

8. An example with Queries ● Queries are Python expressions (see Danny) ● Dependencies can be filled through queries >>> bias = BiasFrame() >>> query = ((RawBiasFrames.chip.name == 'ccd50') & (RawBiasFrames.DATE_OBS > a_date-1) & (RawBiasFrames.DATE_OBS >> bias.raw_frames = list(query) >>> bias.make() >>> bias.store() # the bulk FITS data >>> bias.commit() # the meta data

9. Image Pipeline