1. Paul Pellegrini, Director of IT Colin Watkins, DBA James Blankenship, Sr. QAD Consultant, QAD Certified Professional & present Disaster Recovery in Action

4. Who is American Trim? Lima, OH Wapakoneta, OH Sidney, OH Erie, PA Cullman, AL Monterrey, NL, MX Eight Manufacturing Facilities in four states, plus Mexico Acquired Angell-Demmel, NA with facilities in Lebanon, KY Dayton, OH Dec 2012 QAD customer since 1993

5. What do we do? www.amtrim.com

6. American Trim IT Staffing DBA Network Engineer Systems Analyst Programmer Analyst EDI Manager Programmer/Analyst Applications Integration / Support Manager PC Technician Help Desk Director of IT  reports to CFO Trusted Partners

7. Prior to 2008 No formal plan existed Acted on task oriented opportunities Natural Gas Generator installation Liebert Room Battery backup Servers were stand alone and many (26 physical machines) Backups to tape daily and carried off-site Failure Modes and Effects Analysis (FMEA) We acknowledged existing plan was nonexistent Began documenting in earnest Server Hardware Server Applications Iterative cycle of evaluation Starting in 2008

8. Document Servers / Applications

10. Document Options TOP 12 RISKS COMMON STRATEGIES PROS / CONS

11. Document Strategies 1.Immediate action opportunities 2.Add hardware capacity 3.Backup frequency with immediate off-site copies 4.Implement redundant systems 5.Application specific solutions

12. Confirm Assumptions

13. Quick Timeline Activities 2009 April 6, 2013 2011 2013 January 21, 2012 2012 May 7, 2012 2010 Standardize: Hardware Software Procedures Consistency is key! Document your plan: if you’re not there, how is someone else supposed to know where to start? Document your Trusted Partners January 26, 2013

14. Environment American Trim environment: HP ProLiant DL370 G6 VMware ESXi 5.1 Red Hat Linux 5.5 QAD 2012SE (346 named users) Progress 10.2B07 Lima– QADDR – QADTEST – QADDEV Sidney – QADPROD

15. QADPROD All Progress databases start with AI set to start a new file every 15 minutes, moving the rolled AI to /backup/ai/mfg folder CRON every 15 minutes All AI files – GZIP then copy to QADDR rsync selective folders i.e. apps, cups, data, home, etc QADDR CRON every 15 minutes apply the copied AI files to each DB on DR Synchronizing

16. OTHER QADDR CRON 23:40 daily take probackup, then copy to tape we suspend AI roll forward during backups There is additional logic that: Sends QADTEST & QADDEV files to QADDR and then to tape Sends QADDEV /apps to QADPROD for off-site copies Takes QADPROD local files backups to same server for human mistakes, such as an errant file deletes Refer to actual scripts for additional details and notification logic on the flash drive you have. flash drive Synchronizing

17. Mock Disaster Lessons: Shut off EDI jobs before turning DR box up as PROD Enhanced step-by-step to EDI process shut down & start up Most end clients and servers needed ipconfig /flushdns Found Loftware DR problems (since resolved) Found MS Terminal Service licensing problems (since resolved) Enhanced step-by-step Factivity client start up Our Progress environment was up and fully accessible in less than 30 minutes Mock DR Lessons Learned

18. May 2012 Lessons Learned: Communicate! Engage Trusted Partners immediately Inform IT staff, management, family Inform operations (customers); include how they should get updates Status updates (email blasts) as scope is understood Positive Attitude Take time to get your mental state under control Avoid quick/rash decisions Work as a team / sounding board Define contingencies & milestones Document first, prioritize, then act Health Don’t forget to eat Walk away to clear your head Humor is a good pressure release! May 2012 Lessons Learned This is not the time to “try” things!

19. Prevention An ounce of prevention… VMware hardware probes VMware virtual server probes VMware baselines (CPU, Memory) The Dude WAN/LAN link probes The Dude performance (WAN link utilization) The Dude critical services probes The Dude device heart beats (simple ping) Access Points, Batteries, ILO/DRAC, NAS, DVRs, Kronos clocks, Shift Bells, Room Alerts, etc Microsoft Performance Monitor / Free disk space monitors

21. Questions?

22. Vendors American Trim leverages: BravePoint ProStar Software TailorPro Factivity Eagle Consulting & Development Cyberscience Cyberquery ACOM Solutions Trubiquity