1. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Copyright Copyright Charles Augustine 2006. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

2. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Making the leap to disk-to-disk backup Charles Augustine Manager, System and Data Management Services Princeton University

3. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Princeton University 700 full-time faculty 700 full-time faculty 4,600 undergraduates 4,600 undergraduates 2,000 graduate students 2,000 graduate students 23,000 host computers 23,000 host computers

4. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Backup Service 8,000 nodes 160 TB of data 5 AIX servers Running Tivoli Storage Manager since 1994 Switched from mainframe to Unix in 2001

5. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Background – the first problem PROBLEM: Poor performance & inadequate budget SOLUTION: First fix performance, then start charging

6. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Timeline Fall 2002 – fix performance Fall 2003 – develop charging policy Spring 2004 – implement charging Fall/Spring 2004 – plan and implement disk-to-disk upgrade

7. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Fixing performance September 2002 - About 1/3 of campus clients backed up nightly OIT server backups were OK November 2003 - 98% of managed desktops 90% of all active systems

8. 10/11/2006 Office of Information Technology Princeton University Educause 2006 What Changed? Added resources -Added two servers (for a total of five) -Added memory to existing servers Reduced demand -Quit backing up Windows system files on weekdays

9. 10/11/2006 Office of Information Technology Princeton University Educause 2006 How could we pay for it? On average, the backup budget must balance. Occupancy continued to grow. Income was not growing. –Income was a fixed fee from every network host. –All hosts paid, but only 1/3 used the service.

10. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Growing demand Occupancy grew from 60 to 90 TB in 2003-04

11. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Fixing the budget problem Understand demand Understand costs Get stakeholder input Implement charging

12. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Understanding demand

13. 10/11/2006 Office of Information Technology Princeton University Educause 2006

14. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Understanding demand A few undergraduates used disproportionate amounts. Most undergraduates were not using the backup service. About 1% of all systems accounted for 30% of occupancy.

15. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Understanding costs Developed spreadsheet cost model –Hardware amortization –Hardware maintenance –Software licenses –Staff salaries –Consumables – tape cartridges Marginal annual cost for additional occupancy was between $8 - $9 / GB

16. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Backup Cost Breakdown

17. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Putting together a solution Get stakeholder input on approaches to charging large users. Work with CIO to develop final use and charging policy. Implement charging beginning in 2005 fiscal year.

18. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Stakeholder Input Consulted with constituent groups –Departmental computing support staff –Desktop Systems Council –Research Computing Advisory Group –Individual owners of large systems –OIT Support Services –OIT Finance group –OIT Cabinet

19. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Improved funding model Retain base level of funding from network fees Add two-tier charging policy (Personal and Complete options)

20. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Final Policy Default non-charged service is only for “personal” directories (e.g. My Documents), with no limit on occupancy “Complete” backup for $8/month/system Usage above 150 GB charged at $.80/GB/month/system No new undergraduate accounts –Undergraduates get space on a shared file server for their personal work.

21. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Controlling Demand Occupancy stayed constant from September 2004 to July 2005

22. 10/11/2006 Office of Information Technology Princeton University Educause 2006 The second problem Performance and manageability –Restore times were too long for the largest administrative databases –Insufficient tape processing time –Frequent maintenance calls for tape drives and libraries –TSM required constant administrator attention

23. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Alternative solutions Retain current tape hardware, add another person Upgrade to new tape drives Replace tapes by disk, smaller libraries and 1/3 as many tape drives.

24. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Pros and cons for tapes Faster, higher density tapes –Reduced costs for tape cartridges –Unchanged library maintenance costs –Tape management reduced somewhat –Questions about future of STK “access- centric” tape technology –Increasing capacity is cheaper if the library is not filled.

25. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Pros and cons for disk All disk backup –Improved restore times –Dramatically simplified administration –Improved reliability –Reduced maintenance costs –Purchase price less than tape upgrade –Less floor space

26. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Comparing costs Cost for disk option –100 TB for $500,000 - $750,000 –20% more over five years than doing nothing. –About 30% less than upgrading existing tape drives (purchase price). –Continued use of tape would require additional staff to handle operations adequately.

27. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Presentation to Management The need for change Alternative solutions Cost analysis over five years

28. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Final configuration December 2004 purchase –100 TB STK BladeStore ATA disk –10 STK 9940B (200 GB) tape drives –2 STK L700 (600 tape) libraries Backups go directly to ATA disk Disk pool backed up incrementally to tape

29. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Random access vs. virtual tape Random access –Eliminates reclamation –Minimizes unused space –Lets TSM distribute disk I/O –STK BladeStore ATA disk claimed better random access performance than standard SATA disk

30. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Random access vs. virtual tape Virtual tape –STK proposed virtual tape to control disk I/O distribution –Tivoli recommended virtual tape –Does not eliminate reclaim –Leaves unused areas on disk –Constrains maximum simultaneous streams

31. 10/11/2006 Office of Information Technology Princeton University Educause 2006 How it worked Improved performance Greatly reduced administrator effort. No more struggling to reclaim tapes. Tape copy allowed off-site storage without impacting restore time. Better reliability

32. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Improved transfer rates Disk vs. tape comparison for administrative servers

33. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Better reliability Failure of RAID disks are handled automatically. Repair call is placed automatically. Tape drives and robotics are more reliable because they are less heavily used.

34. 10/11/2006 Office of Information Technology Princeton University Educause 2006 The current situation Occupancy growth continues. We have added 50 TB of data in the last year. Fortunately, TSM allows us to easily put the overflow to tape. The funding model is working, so we will be able to purchase more disk. New SATA disk is less than half of 2004 prices.

35. 10/11/2006 Office of Information Technology Princeton University Educause 2006 Our current plans Three level storage hierarchy –Data from client computers initially stored on random access disk. –Data is migrated from random access disk to “virtual tape”. –When occupancy exceeds available disk, oldest data is migrated to physical tape.

36. 10/11/2006 Office of Information Technology Princeton University Educause 2006 For more information Charles Augustine –Senior Manager, Systems and Data Management Services –augustin@princeton.eduaugustin@princeton.edu Gretchen Thiele –TSM Administrator –tsm@princeton.edutsm@princeton.edu