Storage Services Collecting the pieces of the puzzle TNC 2008, Brugge, 21 May 2008 Jan Meijer

2 Something is brewing... Storage as (a) utility service(s) within the European higher educational and research community

3 What if...

4 Wait, get rid of storage on campus? Think about it: NAS, FC, iSCSI, backup, replication – as – a – service The network makes it possible Smaller institutions (NREN “long tail”) ‏ Moving towards a virtualized world NREN “long tail” will benefit Centralized? Decentralized? Federated!

5 Example: Høgskole i NoenBy

6 But, why? Economies of scale Allow sites to more easily reap benefits from new storage technology Technology enables us to consider this option these days Mixing different financial models: pay as you go, upfront, etc. limited pool of experts: institutional, national, within Europe need to deal with a growing array of ever more complex technologies increased reliability demands are more easily met in co-operation. Think disaster recovery Do more with less money Storage as an enabler Where do YOU want to focus your IT efforts on?

7 Example: NorStore Storage infrastructure for Norwegian research Main objective: Establish and maintain a broad and sustainable infrastructure for the curation, archiving and preservation of data from computational science and the natural sciences. integrated part of the national e-Infrastructure (HPC, national network, national AAI and national grid) ‏

8 A global storage cloud for research & education? Some time away....

9 Spot the differences high capacity network as a utility compute cycles as a utility data storage as a utility

10 Lucky us! Fast networks! 1 PetaByte = 8,000,000,000,000,000 bits 10 Gbit = 10,000,000,000 bits 1 day is sec And this matters why? NorStore has 2x600TB netto in phase 1 1 minute of JPEG2000 video = 6GB LHC > 1 PB/year raw data production And it just keeps growing!

11 Other challenges proof of data ownership single name spacing, tight integration between disparate storage elements what with storing data across borders? predictability (measurements) ‏ reliability of our storage archiving & data curation! AAI-enabling of storage access methods energy consumption

12 Some trends Improved understanding of storage reliability. :( Storing is cheaper than re-acquiring data. Reacquiring isn't always possible Enabling new applications with old data Increased pressure on data security stemming from incidents.... Storage virtualization, server virtualization iSCSI, pNFS SSDs! Fast, high throughput, reliable

13 Proof, pudding, tasting....hu: national storage service, iSCSI.pl: national storage service project national research data storage NREN storage consolidation LHC PetaFlops = PetaBytes...

14 To conclude People want to share their data People want to collaborate across organizational borders People want to have access to their data The technology has to support this Virtualizing storage into a utility service across institutional boundaries is needed, and a great opportunity for all

15 To close Interested in the storage challenge? Join the TF-Storage :) ‏ Next meeting 12 September, Riga, Latvia, we're inviting vendors :) ‏ Thanks jan dot meijer at uninett dot no feel free to contact me

16 File Exchange

17 File exchange + EduGain + NorStore

18 Digital notary experiment Investigating: Proof of data ownership Protecting data integrity Deployment options

19 Services? Building block services: sites doing end-user frontends, end-user support Block storage: iSCSI, FC (..), InfiniBand? File systems: CIFS, (p)NFS, GPFS,.... Archiving, data curation Full services: disaster recovery, off site backup, storage frontends,..? Different service providers in one common (federated) framework!