Status: Central Storage Services CD/LSC/CSI/CSG June 26, 2007

Storage Services 1. File Based Storage NFS/CIFS (BlueArc) – Fast on-site access AFS – Global access, authenticated filesystem 2. Block Based Storage Fibre-Channel connect to SAN 3. Archival Storage Backups 1. File Based Storage NFS/CIFS (BlueArc) – Fast on-site access AFS – Global access, authenticated filesystem 2. Block Based Storage Fibre-Channel connect to SAN 3. Archival Storage Backups

NAS Status Newest Service 2 Production clusters 1. Fermi-Blue (1 st generation cluster) 2. RHEA (2 nd generation cluster) Newest Service 2 Production clusters 1. Fermi-Blue (1 st generation cluster) 2. RHEA (2 nd generation cluster)

NAS Status 3/06 – NAS heads ordered (Fermi-Blue) 5/06 – Pilot deployment SLF, DSG, KITS, PPD and FESS department servers Year 1 projection: 10TB deployed storage 3/06 – NAS heads ordered (Fermi-Blue) 5/06 – Pilot deployment SLF, DSG, KITS, PPD and FESS department servers Year 1 projection: 10TB deployed storage

NAS Status Phase 1 Phase 2 Phase 3 Year 2 Year 3 Year 1 Department Servers, Array Consolidation Rollout to Farms servers Rollout to Farms Workers Projected Rollout

NAS Status Phase 1 Phase 2 Phase 3 Year 2 Year 3 Year 1 Department Servers, Array Consolidation Rollout to Farms servers Rollout to Farms Workers Actual Rollout

NAS Status Actual Year 1 Deployment Q : Pilot Program Early adopters Timing CMS “home” area evaluation Fermigrid NFS issues Actual Year 1 Deployment Q : Pilot Program Early adopters Timing CMS “home” area evaluation Fermigrid NFS issues

NAS Status Actual Year 1 Deployment (cont) Q3 2006: Production Phase 1 in full production CMS + Fermigrid go production (Phase 2) Additional NAS heads purchase (RHEA) Year 1 projection revised to 200TB deployed storage Actual Year 1 Deployment (cont) Q3 2006: Production Phase 1 in full production CMS + Fermigrid go production (Phase 2) Additional NAS heads purchase (RHEA) Year 1 projection revised to 200TB deployed storage

NAS Status Actual Year 1 deployment (cont) Q4/2006 CMS and Fermigrid deploy to worker nodes (Phase-3) Q1/Q D0/CDF/Miniboone begin consolidation of servers into central NAS service Requests for space from LHC, ILC and SDSS Actual Year 1 deployment (cont) Q4/2006 CMS and Fermigrid deploy to worker nodes (Phase-3) Q1/Q D0/CDF/Miniboone begin consolidation of servers into central NAS service Requests for space from LHC, ILC and SDSS

NAS Status NAS Storage Growth Year 1

NAS Status Q storage TB

NAS Status Current customers Experiments CMS, CDF, D0, FermiGrid/OSG, Miniboone ILC, LHC, SDSS, Sciboone(?) Departments CD, Directorate, FESS, ES&H, PPD, VMS Services Scientific Linux (FERMI), CVS, KITS, Alphaflow, Enstore Current customers Experiments CMS, CDF, D0, FermiGrid/OSG, Miniboone ILC, LHC, SDSS, Sciboone(?) Departments CD, Directorate, FESS, ES&H, PPD, VMS Services Scientific Linux (FERMI), CVS, KITS, Alphaflow, Enstore

NAS Status Benefits Stability -- Savings multiplier Effort re-directed towards supporting application Reduced downtime Increased productivity Consolidation (30+ servers/storage arrays) Reduce equipment support costs Reduce power + cooling Benefits Stability -- Savings multiplier Effort re-directed towards supporting application Reduced downtime Increased productivity Consolidation (30+ servers/storage arrays) Reduce equipment support costs Reduce power + cooling

NAS Status Benefits (cont) Ease of use Familiar storage solution – minimal training Flexible Choice of storage tiers, price points Benefits (cont) Ease of use Familiar storage solution – minimal training Flexible Choice of storage tiers, price points

NAS Status Challenges Growth higher than expected – Lun limit Each cluster is limited to 256 luns Each lun limited to 2TB Upgrade to 64TB lun support expected EOY 2008 Criticality of service Central location Offsite DR required? Challenges Growth higher than expected – Lun limit Each cluster is limited to 256 luns Each lun limited to 2TB Upgrade to 64TB lun support expected EOY 2008 Criticality of service Central location Offsite DR required?

NAS Status Challenges (cont) Backup of large data an issue Large data areas >5TB Millions of files Logistics Power Floor space Challenges (cont) Backup of large data an issue Large data areas >5TB Millions of files Logistics Power Floor space

NAS Status FY08 Plans Expansion of service Participate in Tier 3 evaluation Development of better reporting tools FY08 Plans Expansion of service Participate in Tier 3 evaluation Development of better reporting tools

NAS Status More info: Questions? More info: Questions?

SAN Status 272 Fibre-Channel ports 128 ports added to fabric in ‘07 (CMS contribution) Qlogic switches 2Gb Fibre Channel Connections 272 Fibre-Channel ports 128 ports added to fabric in ‘07 (CMS contribution) Qlogic switches 2Gb Fibre Channel Connections

SAN Status 23 storage arrays 12 centrally managed Database Array (3PAR) purchased and tested D0ora2 deployment 7/2/2007 Start retiring 1 st Generation Tier 2 storage arrays (Infortrend) 11 externally managed 23 storage arrays 12 centrally managed Database Array (3PAR) purchased and tested D0ora2 deployment 7/2/2007 Start retiring 1 st Generation Tier 2 storage arrays (Infortrend) 11 externally managed

SAN Status 346TB 156TB centrally managed 190TB externally managed 346TB 156TB centrally managed 190TB externally managed

SAN Status SAN fabric opened up to external members CMS, CDF, D0, Miniboone Must retire LSI storage array End of support (year end 2007) Impacts IMAP/POP, AFS, DSG(CDF) SAN fabric opened up to external members CMS, CDF, D0, Miniboone Must retire LSI storage array End of support (year end 2007) Impacts IMAP/POP, AFS, DSG(CDF)

SAN Status FY07 Plans Additional HDS array NAS storage for SDSS, Windows Migration, DSG Block storage for LSI migration FY07 Plans Additional HDS array NAS storage for SDSS, Windows Migration, DSG Block storage for LSI migration

SAN Status FY07 Plans (Cont) Purchase 2 Nexsan SATAbeasts Replace 4 Infortrend arrays Backup cache disk, DSG RMAN disks Test as possible tier 3 candidates FY07 Plans (Cont) Purchase 2 Nexsan SATAbeasts Replace 4 Infortrend arrays Backup cache disk, DSG RMAN disks Test as possible tier 3 candidates

SAN Status FY08 Plans Additional capacity for 3PAR For sparing DSG migration Additional capacity for NAS Decommission remaining Infortrend arrays Other tier 3 alternatives (nexgen HDS, DDN) Virtualization across arrays FY08 Plans Additional capacity for 3PAR For sparing DSG migration Additional capacity for NAS Decommission remaining Infortrend arrays Other tier 3 alternatives (nexgen HDS, DDN) Virtualization across arrays

SAN Status Questions?

Site Backup Status Service entering 4 th year 10/07 2 Backup Servers Chasm (infrastructure and business) Canyon (experiment) 1 Library (600 slots) 8 SAIT-1 Tape Drives 2 Infortrend Storage arrays TiBS Backup Software Service entering 4 th year 10/07 2 Backup Servers Chasm (infrastructure and business) Canyon (experiment) 1 Library (600 slots) 8 SAIT-1 Tape Drives 2 Infortrend Storage arrays TiBS Backup Software

Site Backup Status 22TB+ data 12,700+ backup volumes 5,506 UNIX/Windows, 7171 AFS, 25 NDMP 452+ clients 18.5% increase in past 6 months (3.7TB) No single volume > 100GB 22TB+ data 12,700+ backup volumes 5,506 UNIX/Windows, 7171 AFS, 25 NDMP 452+ clients 18.5% increase in past 6 months (3.7TB) No single volume > 100GB

Site-Backup Status Typical Daily Backup Timeline (canyon) Incr/NetworkMergesRetry Debug 6:00PM2:00AM1:00PM1:40PM 24 hour window

Site-Backup Status Issues Resolving client backup issues High client volatility Reconfiguration/Renaming/Reinstalls Large delta in data Contacting admins Slow client network performance Issues Resolving client backup issues High client volatility Reconfiguration/Renaming/Reinstalls Large delta in data Contacting admins Slow client network performance

Site-Backup Status Issues (cont) Merge problems Can be difficult to debug Tape drive/Software or combination Cache disk Multiple disk failures Issues (cont) Merge problems Can be difficult to debug Tape drive/Software or combination Cache disk Multiple disk failures

Site-Backup Status Issues (cont) SAIT-1 drive performance issues Tapes written on one drive are slow to read on another Long debug time > 1 hour Usually requires multiple replacements Sony and Spectra investigating Too few Issues (cont) SAIT-1 drive performance issues Tapes written on one drive are slow to read on another Long debug time > 1 hour Usually requires multiple replacements Sony and Spectra investigating Too few

Site-Backup Status FY07 Plans Chasm Canyon IP Disk Cache SAIT-1 Drives SAN Migrate more backups to NDMP Relieve pressure on chasm Migrate clients from canyon to chasm Relieve pressure on canyon LTO-4 NDMP

Site-Backup Status FY07 Plans (cont) Upgrade cache disks Replace aging Infortrend disks Higher performing array RAID 6 FY07 Plans (cont) Upgrade cache disks Replace aging Infortrend disks Higher performing array RAID 6

Site-Backup Status Challenges Desire from users to expand backups Larger backup volumes Larger backup sets Challenges Desire from users to expand backups Larger backup volumes Larger backup sets

Site-Backup Status FY08 Plans Upgrade Servers to Solaris 10 Faster IP stack and Filesystem Upgrade server hardware Faster bus speed Utilize faster cache disk Take advantage of faster filesystem Feed faster tape drives Migrate canyon backups to LTO-4 FY08 Plans Upgrade Servers to Solaris 10 Faster IP stack and Filesystem Upgrade server hardware Faster bus speed Utilize faster cache disk Take advantage of faster filesystem Feed faster tape drives Migrate canyon backups to LTO-4

Site-Backup Status FY08 Plans (cont) Investigate Disk-based library TiBS specific implementation Use common disks as a disk library Synchronous copy to tape (also) Faster restores, possibly backups May increase overall backup system throughput FY08 Plans (cont) Investigate Disk-based library TiBS specific implementation Use common disks as a disk library Synchronous copy to tape (also) Faster restores, possibly backups May increase overall backup system throughput

Site-Backup Status FY08 Plans (cont) Investigate Virtual Tape Library Agnostic solution (not TiBS specific) Asynchronous copy to tape Emulate tape drives and libraries Faster restores and backups Will increase overall backup system throughput Some systems have data-deduplication Inline or post-process FY08 Plans (cont) Investigate Virtual Tape Library Agnostic solution (not TiBS specific) Asynchronous copy to tape Emulate tape drives and libraries Faster restores and backups Will increase overall backup system throughput Some systems have data-deduplication Inline or post-process

Site-Backup Status More information: Questions? More information: Questions?

AFS Status 12 AFS servers ~17TB storage Largest customers: Minos and Web Roughly 8-10% increase per year (Based off number of volumes) Must migrate servers off of LSI storage array and onto HDS Tier 2 storage. 12 AFS servers ~17TB storage Largest customers: Minos and Web Roughly 8-10% increase per year (Based off number of volumes) Must migrate servers off of LSI storage array and onto HDS Tier 2 storage.

AFS Status FY07 Plans Migrate data to HDS Tier 2 disks Migration partially complete (1.8TB installed) Tier 2 storage re-allocated to NAS due to high demand Test Solaris 10 AFS server with ZFS FY07 Plans Migrate data to HDS Tier 2 disks Migration partially complete (1.8TB installed) Tier 2 storage re-allocated to NAS due to high demand Test Solaris 10 AFS server with ZFS

AFS Status FY08 Plans Upgrade Servers to Solaris 10 Faster OS – filesystem and IP stack Newer CPUs – low power Dual Power Supply Upgrade OpenAFS Multi-domain support Support for > 2GB files Promote RO copies to RW copies FY08 Plans Upgrade Servers to Solaris 10 Faster OS – filesystem and IP stack Newer CPUs – low power Dual Power Supply Upgrade OpenAFS Multi-domain support Support for > 2GB files Promote RO copies to RW copies

AFS Status More information: Questions? More information: Questions?