TeraGrid & XD approach to long-term archival Phil Andrews et al The Moving Finger writes; and having writ, Moves on; nor all your Piety nor Wit Shall lure it back to cancel half a Line, Nor all your Tears wash out a Word of it. -Omar Khayyam The output of our labour is now predominantly large datasets. There is currently no unified mechanism to ensure its continued safety and availability

2 Data Archival: no current long- term funding Earlier this year, TeraGrid (Andrews, PI) submitted a $12M proposal for data- replication within TG. No money was available this fiscal year. Plan to resubmit a more general proposal in late ’10. Short term SDSC and NCSA archival issues resolved. It’s Later Than You Think! – A Tale of Two Cities, Charles Dickens Science Advisory Board Meeting, December'09

3 Looking ahead … this will be a recurring and growing issue Science Advisory Board, December, 2009 Notional projections for illustration only! Estimated 3PB/yr/PF(peak) for T2 systems, 5PB/yr for Blue Waters. Current transition

4 What are the high-end users doing? Many users need data for 3-5 years Experimental data must be stored long-term, dual copies Climate data recently in the news: used to motivate $Trillions of policy decisions; should not be deleted COLA group currently generating ~1PB/year of data that needs to be saved and disseminated Single runs can generate 100+TB, 1 yr to generate, 1-2 years to analyze, need easy access during period Data Access profile: high speed I/O, then dissemination, then continued archival Science Advisory Board, December, 2009

5 No Concerted NSF plan for Archival Two NSF DataNet awards at ~$25M each, but no provision for actual data storage Need general NSF plan for data archival-we feel TG/XD is uniquely positioned to lead Will propose joint data responsibility: multiple copies across TG/XD, no individual RP dependence: “Lloyds of London” approach Science Advisory Board Meeting, December'09 I spent half of my money on women, booze, and gambling-the rest of it, I just frittered away! - George Best

6 How should it be done? Science Advisory Board Meeting, December'09 Many possible ways, expect linked global file systems for data transport and dissemination with integrated archival mechanisms over multiple sites. Ongoing development in response to user requirements and technological advances A merry road, a mazy road, and such as we did tread. The night we went to Birmingham by way of Beachy Head! – G.K Chesterton

7 Would like: The endorsement of the SAB for a TG/XD proposal that would provide general archival services, inc. dissemination and replication To submit new proposal to the NSF in late ’10 Recognition of data storage importance Science Advisory Board Meeting, December'09 “Rome wasn’t built in a day: but I wasn’t on that particular job! – Brian Clough, English Soccer Manager