MARFC Operational Backup A Case Study January 26, 2006

1/26/2006MARFC Operational Backup2 Outline ProblemProblem Proposed SolutionProposed Solution ResultsResults IssuesIssues

1/26/2006MARFC Operational Backup3 Problem MARFC moving to new facilityMARFC moving to new facility AWIPS unavailable for up to 6 days during moveAWIPS unavailable for up to 6 days during move How to conduct operations during AWIPS outageHow to conduct operations during AWIPS outage –Maintain full operational output Official ProductsOfficial Products All web informationAll web information –Worst case planning?

1/26/2006MARFC Operational Backup4 Proposed Solution Utilize in-house Linux serverUtilize in-house Linux server X-client LaptopsX-client Laptops ER WAN ConnectionER WAN Connection Data feed via LDMData feed via LDM Transmission via LDADTransmission via LDAD Additional seats for flooding and supportAdditional seats for flooding and support AWIPS OB4 BasisAWIPS OB4 Basis

1/26/2006MARFC Operational Backup5 Proposed Solution - Server Linux ServerLinux Server –Dell 4600 PowerEdge (late 2002) –Dual 2 GHz Xeon CPUs –1 GB Memory –Raid 1 SCSI 73GB HDs –Dual Power Supplies –RH 7.3

1/26/2006MARFC Operational Backup6 Proposed Solution - Access 4 Operational seats4 Operational seats X access via 3 client laptopsX access via 3 client laptops –Part of ER RFC Backup Project –1600x1050 resolution 1 ½ screens1 ½ screens –External mouse and keyboard –Additional monitor for “non-AWIPS” display Server console for HASServer console for HAS 2 add’l clients for support and flooding2 add’l clients for support and flooding

1/26/2006MARFC Operational Backup7 Proposed Solution - Comms ER WAN ConnectionER WAN Connection Data feed via LDMData feed via LDM –Redundancy via ERH and SRH Transmission via LDADTransmission via LDAD –Redundancy via PBZ with ERH backup

1/26/2006MARFC Operational Backup8 Proposed Solution - Apps Based on AWIPS OB4Based on AWIPS OB4 Synchronize local apps changes between AWIPS and serverSynchronize local apps changes between AWIPS and server –Crons as well Allow all auto processes to continueAllow all auto processes to continue –Shut off delivery via tokens Identify files to sync to go liveIdentify files to sync to go live –OFS, DB, control

1/26/2006MARFC Operational Backup9 Results Difference in capabilitiesDifference in capabilities –No D2D –No ArcView use FOPFOP Inundation mappingInundation mapping –No 12Planet (within AWIPS only) –1 ½ vs. 3 screens

1/26/2006MARFC Operational Backup10 Results OperationsOperations –Started with HAS AM shift 12/5/2005 –Product delivery easily shut off on AWIPS and initiated on backup via tokens –Continued in backup mode until Thursday, 12/8, morning hydro shift AWIPS available for testing Wednesday night, 12/7AWIPS available for testing Wednesday night, 12/7 –Token reset shut off delivery from backup and initiated on AWIPS

1/26/2006MARFC Operational Backup11 Results PerformancePerformance –No forecaster perceived slowness in any operational application –No missed delivery of any text or graphical product SUCCESS!SUCCESS!

1/26/2006MARFC Operational Backup12 Issues Keeping in-step with AWIPSKeeping in-step with AWIPS –Upgraded to OB5 on AWIPS prior to move –Stayed on OB4 due to DB table changes Not sure of best approach to incorporate changes of this natureNot sure of best approach to incorporate changes of this nature OB6?OB6? Help with problemsHelp with problems Maintenance overhead – 2 systemsMaintenance overhead – 2 systems Overall RFC solution?Overall RFC solution?

1/26/2006MARFC Operational Backup13 Issues Redundancy is critical!Redundancy is critical! 1 hour into backup operations one HD on server failed1 hour into backup operations one HD on server failed –Continued with one HD with no problems

1/26/2006MARFC Operational Backup14 Issues Off-site computing capabilitiesOff-site computing capabilities If incapable of full operations support, what do you cut outIf incapable of full operations support, what do you cut out –Customers expecting information HD failure forced laptop server implementationHD failure forced laptop server implementation

1/26/2006MARFC Operational Backup15 Issues Off-site computing capabilitiesOff-site computing capabilities –Part of ER RFC Backup project –Dell 5150 “desktop replacement” laptop –1 year old –Pentium GHz –1 GB Memory –1 100GB HD; 7200 RPM ATA

1/26/2006MARFC Operational Backup16 Issues Off-site computing capabilitiesOff-site computing capabilities –Operations test 3 client laptops3 client laptops Laptop displayLaptop display Full cron loadFull cron load Morning forecast operationsMorning forecast operations

1/26/2006MARFC Operational Backup17 Issues Off-site computing capabilitiesOff-site computing capabilities –Operations test – results SlowdownsSlowdowns –NMAP initiation –Heavy disk access – e.g. Informix extractions Slower than in-house serverSlower than in-house server Acceptable by forecast staff knowing limitationsAcceptable by forecast staff knowing limitations

1/26/2006MARFC Operational Backup18 Performance Comparison Identical cron instructionsIdentical cron instructions Handoff and execution of OFS jobs from AWIPSHandoff and execution of OFS jobs from AWIPS Laptop slowdown with heavy disk access activityLaptop slowdown with heavy disk access activity More analysis neededMore analysis needed

1/26/2006MARFC Operational Backup19 updatedb, db_purge, sys_clean db_purge ???

1/26/2006MARFC Operational Backup20 Off-site Capabilities “Shoebox” server, aka ShuttlePC“Shoebox” server, aka ShuttlePC “Server-like”; luggable“Server-like”; luggable Dual CPUsDual CPUs Multi-GB memory capabilitiesMulti-GB memory capabilities 7200 RPM SATA II7200 RPM SATA II –SCSI-like performance? –Larger, cheaper than SCSI –RAID 1

1/26/2006MARFC Operational Backup21 Off-site Capabilities Less than $3000Less than $3000 –Dual 64-bit AMD Opteron, 2.4 GHz –2 GB memory –Dual 120 GB 7200 RPM SATA II RAID 1RAID 1 –GB ethernet –19” LCD –External keyboard and mouse

The End