Data Intensive Astronomy Group Talk II ICRAR Con 4 September 2015 Chen Wu

1 –MWA data system –MWA data usage –MWA storage modelling –GLEAM archive –GLEAM VO usage –In-archive processing Agenda

2 MWA Data System

3 MWA Archive

4

5 MWA ingest and retrieval 24 Jan Aug 2014

6 Data access by region 14 million successful retrievals

7 MWA usage

8 Distribution of file freshness and staleness

Data MWA LTA Pawsey 9 10 Gbps Stage Release Fast Disk Storage Tape Library DM-GET DM-PUT Archive Stage FE Nodes CXFS API POSIX MWA Pawsey Hierarchical Storage Management (DMF) Tape Library x2 (CSIRO - library + cables + optics) x32 Bulk Disk Storage Science DB M&C DB

10 Staging time  AGE_WEIGHT = constant + multiplier*

11 Storage performance modelling “simulation” using the MWA data access stream (25 million successful requests) (a, b, c, d, c, a)  3 is the reuse distance (a, b, c, d, c, a)  a is staged from the tape

12 GLEAM IVOA Interface GLEAM VO Server Web Interface GLEAM Archive Store 04 GLEAM Archive Store 06 NGAS Client Over 800,000 images 20,000 MeasurementSet 220 TB

13 GLEAM usage on all-sky view All sky view in Aladin Lite!

14 In archive processing Some “real” requirements from both MWA and GLEAM: –Interactive processing Cutout and regridding, NGAS Tasks –Batch (re-)processing - Process all files satisfying some conditions currently in the archive: e.g. Compress all visibility files that are (1) EoR project and (2) Observed on last Friday (MWA) Rescale flux of all snapshot images of GLEAM Phase 1 that are ingested in the past two weeks Make movies from images formed in DEC -26 strip scans Re-index all WCS headers of images ingested from last November –Incremental processing - Asynchronously, continuously, and selectively processing "newly" ingested files After a snapshot image tar is ingested, decompress it, and for each FITS image, compute its sky coverage, and update VO database indexes accordingly As soon as a 32MHz image is ingested, if its Robustness is 0, send a copy to RRI at India before transferring it to RDSI –NGAS Job Framework With the same spirit of MapReduce File  Object  Container or DROP

15 Data re-processing Web UI

16 –MWA data system –MWA data usage –MWA storage modelling –GLEAM archive –GLEAM VO usage –In-archive processing Conclusion

17 Thank you! Q & A