Installation and Maintenance of Health IT Systems Lecture b This material Comp8_Unit 8b was developed by Duke University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number IU24OC Troubleshooting; Maintenance and Upgrades; Interaction with Vendors, Developers, and Users

Troubleshooting; Maintenance and Upgrades; Interaction with Vendors, Developers, and Users Learning Objectives 1.Identify and implement an effective troubleshooting procedure for reporting, evaluating, fixing, deploying, and follow-up of errors, problems, or limitations for the system (lectures a & b) 2.Integrate downtime schedule for OS, network, database, and client application maintenance and updates (lecture b). 3.Develop a process for communicating requirements and supplying updates between vendors/developer and users (lecture b). 4.Create a baseline for system performance measurement and comparison for troubleshooting (lecture b). 2 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b

Troubleshooting: Diagnosing Complex EHR Infrastructure Issues Identify the problem Determine the scope of the issue Start Simple Seek Additional help during an Outage Remember to communicate (Halamka) 3 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b

Troubleshooting: Diagnosing Complex EHR Infrastructure Issues Use the logs Develop a timeline Stick to the Plan…Set deadlines for escalation Stay focused on resolving the problem. Don’t be too quick to declare a problem resolved. (Halamka; AHRQ, 2011) 4 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b

Troubleshooting: An Example User contacts the EHR Helpdesk about slow EHR… –Start with the local machine Recent Updates or Patches System and Application Logs Wireless/ Ethernet Connectivity 5 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b

Troubleshooting: An Example If the issue is more widespread… –Network or ISP Issues Misconfigured switch, router, ISP Connectivity –Erroneous OS Upgrade/ Misconfigured Server –Server Hardware Error/ Upgrade Needed –Application Upgrade Error EHR Design EHR Software Code 6 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b

Performance Baseline Generated after completion of performance baseline testing. Shows normal operating parameters of your system under normal load conditions. Used to gauge overall system health & assist with isolating performance problems. (Wunder) 7 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b

Performance Baseline: Testing “Benchmark testing” Compare performance of new server/system against standard (e.g., existing measurements or software specs). Test at regular intervals, when problem-free: –System resources (server & hardware) –Network architecture –Operating system –Database applications –Client applications (Sqlserver, 2009) 8 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b

Performance Baseline: Utilities Available Work with vendor to determine best method for your EHR. Many third-party utilities available. Windows built-in utilities: –System Monitor (collect & view real-time data on usage of memory, disk, processor) –Performance logs & alerts –Task Manager –Event Tracing for Windows (ETW; trace & log events raised by user-mode applications & kernel-mode drivers; export most SQL Server events.) (Microsoft, 2011; Technical, 2005) Databases often require special utilities. SQL Server built-in utilities: –SQL Trace –SQL Server Profiler –SQL Server Management Studio Activity Monitor –SQL Server Management Studio Graphical Showplan –Stored procedures –Database Console Commands (DBCC) –Built-in functions –Trace flags 9 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b

Performance Baseline: Measurements & Next Steps Measurements –Peak vs. off-peak hours –Production-query, batch-command response times –Database backup and restore completion times Next steps –Compare baseline statistics to current server/system performance. –Investigate if numbers far above or below baseline. (Wunder) 10 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b

EHR Maintenance & Upgrades Upgrades crucial for extending EHR functionality & overall system lifespan. Improperly planned/managed upgrade procedures can severely damage system, reducing performance. Use highly structured approach in upgrading any major, critical production system. 11 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b

EHR Maintenance & Upgrades: Structured Approach Upgrade team –Works with vendor. –Reads upgrade documentation. –Analyzes functional enhancements. –Works with vendor on unknown variables. –Makes plan to configure & test affected applications. Technical team develops alternative testing environment (test bed). Upgrade team compiles pre-installation & installation checklists, determines downtime. After first testing, logs reviewed & errors/discrepancies reported to vendor. 12 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b

EHR Maintenance & Upgrades: Structured Approach (cont’d) After vendor resolves issues, upgrade process retested, ensuring data integrity. Consider software to simulate workflow & users; track data for performance analysis. Unified spreadsheet to track issues. Once testing complete & upgrade approved, final documentation prepared & training scheduled. Installation team finalized & assigned roles. Upgrade scheduled during off-peak times. EHR “shadow copy” (read-only, without real-time interfaced results) accessible during upgrade. 13 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b

14 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b EHR Maintenance & Upgrades: Structured Approach (cont’d) Upgraded system tested immediately after completion. Once testing complete, help desk notified & systems brought back online. Dedicated team available for several business days to resolve issues. (Follow similar processes for upgrades to server OS, workstation OS, databases.)

Interacting with Users Stay up to date Set realistic expectations Value your user’s time Be friendly, but stay on task. Listen Closely, and acknowledge the user’s issue Use the opportunity to educate Be patient (Boyer, 2005) 15 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b

16 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b Essential not only for successful implementation, but also for continued success of product. –Issues will arise beyond scope of your IT workforce. –Routine upgrades needed. –Vendor plays pivotal role in these & other events. Successful partnerships built on mutual goals, trust. –“In this together”. –Success of vendor depends on success of product. Long-term commitment between organization & vendor. –Relationship becomes complex once organization dependent on fully embedded EHR. (Boyer, 2005) Client-Vendor Relationship

17 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b Client-Vendor Relationship: Before Purchase Understand vendor company & culture. –Visit headquarters, talk with as many as you can (including development & support). –Follow the money. What drives revenue stream? –Dependent on support fees? Understand support-fee structure. –Financially solvent enough to provide long-term support? Hosted and SaaS (Software as a Service) solutions: understand vendor’s upgrade strategy. –Will it conflict with your business practices? Impressions of vendor’s support staff (Boyer, 2005)

18 Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b Performance baseline –Helps gauge system health & isolate problems. Structured approach for upgrades –Separate environment for testing upgrades prior to rollout. Work closely with vendor to resolve issues. Rollout during non-peak hours. Client-vendor relationship –EHR purchase brings long-term commitment. Complexity once dependent on EHR. Understand vendor & culture before purchase. Troubleshooting; Maintenance and Upgrades; Interaction with Vendors, Developers, and Users Summary

Troubleshooting; Maintenance and Upgrades; Interaction with Vendors, Developers, and Users References – Lecture b References Boyer, E. and M. Soback (2005). Production Support. Implementing an Electronic Health Record System. J. M. Walker, E. J. Bieber and F. Richards, Springer London: “Event Tracing for Windows” (2011). Microsoft.com. [Internet]]. “Guide to Reducing Unintended Consequences of Electronic Health Records” (2011). AHRQ. [Internet]. Halamka, John. “10 tips for troubleshooting complex EHR infrastructure problems”. KevinMD.com. [Internet]. “SQL Server Security, Performance & Tuning.” (2009).SSQA.net. [Internet]. qa.net/blogs/perftune/archive/2009/07/26/5820.aspxhttp://sqlserver- qa.net/blogs/perftune/archive/2009/07/26/5820.aspx “Technical Comparison of Oracle Database 10g and SQL Server 2005: Focus on Manageability, May 2005” (2005, May). Oracle. [Internet]. Wunder, Bill “Benchmarking Techniques Using T-SQL Part 1 - System Statistical Functional”. [Internet] Health IT Workforce Curriculum Version 3.0/Spring 2012 Installation and Maintenance of Health IT Systems Troubleshooting, Maintenance & Upgrades & Interaction with Vendors, Developers and Users Lecture b