1. Aus-VO: Progress in the Australian Virtual Observatory Tara Murphy Australia Telescope National Facility

2. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 Aus-VO: who we are Science infrastructure effort, funded by ARC LIEF grants Science infrastructure effort, funded by ARC LIEF grants The Aus-VO Partner Institutes are: The Aus-VO Partner Institutes are: –CSIRO Australian Telescope National Facility –The University of Melbourne –The University of Sydney –Anglo-Australian Observatory –Swinburne University of Technology –The University of Queensland –Monash University –The University of New South Wales –Victorian Partnership for Advanced Computing –ANU Mount Stromlo Observatory & Supercomputing Facility Plus collaborators: Plus collaborators: –CSIRO Mathematical and Information Sciences –Australian Partnership for Advanced Computing

3. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 APAC Astronomy Grid Grid infrastructure effort Grid infrastructure effort Funded by the Australian Partnership for Advanced Computing (APAC) Funded by the Australian Partnership for Advanced Computing (APAC) Active members & partners are: Active members & partners are: –Melbourne and Swinburne –UNSW and Sydney –ANU Project Leader: David Barnes Project Leader: David Barnes Research Leader: Rachel Webster Research Leader: Rachel Webster

4. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 Aus-VO: our focus Bring legacy astronomy archives on-line Ensure future project compliance Provide access to archived realizations of simulations Describe all data fully, and support a small and well-chosen set of interoperability protocols Develop tools and interfaces to find, acquire, process and visualize data Build national grid infrastructure to host the data, tools and interfaces

5. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 Aus-VO: what we do Science content Science content –providing data –providing software –metadata –registry content Science services –source finding –coordinate transformations –catalogue matching –visualization –interfaces to legacy environments

6. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 APAC Grid: what we do Data grids – –data store – –data access protocols – –authentication & authorisation – –network facilities Compute grids – –workflow – –job configuration, submission & monitoring – –code-to-data paradigms

7. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 Aus-VO: current research Data mining – –catalogue matching – –searches for new empirical relationships & rare objects Legacy software integration – –using legacy code in VO services – –using VO services in legacy software environments – –transparent access to Web and Grid Services from the desktop Visualisation – –remote visualisation service (legacy s/w drawing to remote “canvases” via Corba, SOAP, …) – –VOTable to VRML service – –volume rendering

8. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 Aus-VO: projects 2004 Common format on-line archive projects: – –HIPASS catalog: HI Parkes All Sky Survey: neutral Hydrogen spectral line survey, 4,300 sources with ~30 parameters and 1024-channel radio spectra – –SUMSS catalog: Sydney University Molonglo Sky Survey: radio continuum survey at 843 MHz, 107,000 sources with ~15 parameters – –2dFGRS QSO catalog: 2-degree Field Galaxy Redshift Survey: optical spectra of >20,000 southern quasi-stellar objects – –ATCA archive: Australia Telescope Compact Array archive: all observations since 1988, circa 1.5 TB of more than 1,000 separate observing projects! Substantial exercise in describing data with metadata. – –MACHO, RAVE, Pulsar timing archives, Gemini archive and more in 2004!

9. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 Aus-VO: projects 2004 Server- and client-based visualisation tools: – –client canvas for legacy software package AIPS++ to draw upon from a remote server. – –3d visualisation of catalogues Data reduction pipelines for: – –Molonglo Observatory Synthesis Telescope – –AT Compact Array archive – –Gemini, for Australian-used instruments Portal for theoretical astrophysics jobs: – –configuration and execution – –monitoring – –result retrieval and initial analysis and display

10. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 The Australia Telescope Online Archive Consists of: Consists of: 1.an archive of all data observed with the Australia Telescope Compact Array 2.a data reduction pipeline to process the data for ‘quick and dirty’ images 3.a remote visualisation system to inspect the images on the client machine People: People: –Peter Lamb, Tara Murphy, Anil Chandra, Praveena Tokachichu, Chris Owen

11. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 1. The archive All ATCA data since 1990 All ATCA data since 1990 Total of 1.9 Tb covering over 1000 projects Total of 1.9 Tb covering over 1000 projects Hosted on a machine with 2GB memory and 6TB of disk Hosted on a machine with 2GB memory and 6TB of disk Back end is a 20 node Beowulf cluster Back end is a 20 node Beowulf cluster

12. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 1. The archive Challenges: Challenges: –Unifying metadata to fully describe data –Capturing the intentions of the observers determining the calibration strategy determining the calibration strategy –Dealing with the significant minority of non-standard observations

13. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 2. The pipeline Automatically runs complete data reduction process for ATCA data Automatically runs complete data reduction process for ATCA data Underlying work done by AIPS++ Underlying work done by AIPS++ Methods exposed as web services using Glish/Java Methods exposed as web services using Glish/Java Uses metadata from archive to make decisions about reduction Uses metadata from archive to make decisions about reduction

14. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 2. The pipeline Challenges: Challenges: –Dealing with legacy code –Dealing with legacy data (via the archive) –Getting the pipeline to make sensible decisions about how to process the data

15. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 3. Visualization system A software system that allows scientists to view and analyse astronomical data that are stored in remote locations. A software system that allows scientists to view and analyse astronomical data that are stored in remote locations. RVS Server User Data Centre User connects to view data Visualisation Data (Image) sent to user RVS response Dataset sent to RVS Browse data

16. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 3. Visualization system Challenges: Challenges: –Performance RVS Image data format RVS Image data format Connection speeds between services Connection speeds between services –Session Management How to coordinate data centres and clients? How to coordinate data centres and clients? Caching of images for future sessions Caching of images for future sessions

17. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 ATOA Walkthrough 1. Query the archive Query the archive Query the archive 2. Query results Query results Query results 3. Automatic data reduction Automatic data reduction Automatic data reduction 4. Visualise data Visualise data Visualise data

18. Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 Aus-VO: links www.aus-vo.org www.aus-vo.org www.aus-vo.org www.atnf.csiro.au/vo www.atnf.csiro.au/vo www.atnf.csiro.au/vo RVS will be public in ~2 months RVS will be public in ~2 months The ATOA will be public in ~2 months The ATOA will be public in ~2 months The prototype pipeline will be public in ~3 months The prototype pipeline will be public in ~3 months