1 Track Owners: Zinc Instructions Please update the chart to the right before submitting the presentation into Active Following review and approval in Active, each presentation given by GE employees or customers at Centricity LIVE must be approved through Zinc Jessica Diniz will be uploading each presentation into Zinc following Active review Before submitting this presentation in Active for review, Track Owners must select the appropriate answer to each Legal and Regulatory question ALL QUESTIONS MUST BE ANSWERED If further substantiation is required, please the appropriate permissions, sources, or citations to so she can upload to the Zinc Once the presentation is approved in Zinc, please delete this slide from the final deck before presenting at Centricity LIVE Individual Page Use?:No Are all Uses and Applications Discussed or Referenced in the Proposed Material on Label for the Product?: Yes Relates to a Major new Product/Services Launch?: Yes Includes Product or Service Claims, Features, Benefits or similar information?: Yes Contains Competitive or Comparative Claims?:No Includes Return on investment (ROI), quantifiable cost of ownership, reimbursement, or meaningful use claim or reference?: No Include References to Market / Segment Share or Market / Segment Leadership?: No Include identifiable Patient Information / Data?:No If Customer Names are Used, are Necessary and Appropriate Permissions in Writing and on File?: No Include Customer Testimonials?: No If any non-GE Stock Images are Used, are Necessary and Appropriate Permissions in Writing and on File?: N/A Do the Proposed Materials Contain Images of GE Products?: No Do all Product Images Depict the Device(s) Being Promoted in the Piece?: N/A This is a Translation of a Previously Approved Piece in Which no Changes Have Been Made?: No This is a Revision of a Document Previously Approved for Ad / Promo Use?: No

Centricity™ Practice Solution / Centricity EMR Performance Troubleshooting Tools and Techniques Brian Divine, Staff Software Engineer May 1, 2015

©2015 General Electric Company – All rights reserved. The results expressed in this document may not be applicable to a particular site or installation and individual results may vary. This document and its contents are provided to you for informational purposes only and do not constitute a representation, warranty or performance guarantee. GE disclaims liability for any loss, which may arise from reliance on or use of information, contained in this document. All illustrations are provided as fictional examples only. Your product features and configuration may be different than those shown. Information contained herein is proprietary to GE. No part of this publication may be reproduced for any purpose without written permission of GE. DESCRIPTIONS OF FUTURE FUNCTIONALITY REFLECT CURRENT PRODUCT DIRECTION, ARE FOR INFORMATIONAL PURPOSES ONLY AND DO NOT CONSTITUTE A COMMITMENT TO PROVIDE SPECIFIC FUNCTIONALITY. TIMING AND AVAILABILITY REMAIN AT GE’S DISCRETION AND ARE SUBJECT TO CHANGE AND APPLICABLE REGULATORY CLEARANCE. GE, the GE Monogram, Centricity, and imagination at work are trademarks of General Electric Company. All other product names and logos are trademarks or registered trademarks of their respective companies. General Electric Company, by and through its GE Healthcare division. 3 Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Agenda Overview Troubleshooting Systematic Issues Troubleshooting Workflow Issues 4

Overview 5

Expectations Interactive Sharing is good, we all learn Take-aways for you Tools and techniques for troubleshooting Solve issues independently Better information when calling in issues Take-aways for GE Methods you use Tools that could be helpful Questions that need answering 6 You GE Healthcare

Types of Issues 7 Systematic Everything seems slow Often intermittent, slower at some times, faster at others Usually caused by an environmental issue, the trick is narrowing it down. Errors/freezing without pattern or workflow Workflow Encounters opening Clicking in form Signing order/encounter Problem search

Troubleshooting Systematic Issues 8

Performance Virtualization and oversubscription Server resources (Memory, CPU, Disk) Network Database JBoss 9

Virtualization and Oversubscription When it comes to your JBoss and database servers, just say no to oversubscription. Using EsxTop in VMWare CPU CPU % ready should be under 5% Load Average 1.00 means fully utilized, 0.5 half utilized, 2.00 very overloaded Memory Over commitment - 0 or less indicates no over commitment, no extra RAM needed, 0.5 is 50% overcommitted, add RAM to host, reduce RAM in VM’s, or reduce number of VM’s 10

Virtualization and Oversubscription (cont.) Other areas to look Ballooning/swapping Storage latency Network latency These articles are great resources for VM Performance &externalId= its-real-meaning/ 11

Server Resources Perfmon can tell you a lot: Network Memory Disk usage CPU Processes The hardware calculator does a good job, double check with Perfmon on-going Each site is unique in how the system is used 12

Network Low bandwidth Sometimes unavoidable, especially for remote sites, but need to watch Broadcast storms Bad NICs Avoid Jboss to DB over WAN 13

Database Oracle ® AWR and ASH reports are your friends here –Long running queries –Archive logs –Memory utilization SQL Developer can make life easier SQL Server CPNS query Page life expectancy shows how well SQL is utilizing memory. Should be very large 5k- 10k or more. Low numbers indicate need of memory. Very low number (double digits) can lead to instability 14

JBoss CQR feed Med to large sites should have CQR feeds on a separate Jboss to users Large Sites Consider horizontal scalability –Manual in CPS 12.0/EMR 9.8 –Coming in CPS 12.2/EMR 9.10 – easier to setup 15

Troubleshooting Workflow Issues 16

Workflow Performance Forms Opening Operating eRx 17

Form Performance Opening Which form(s) are the cause? –Open blank encounter –Add forms one at a time, find the culprit(s) If custom, take a look at initialization, watchers, etc. MEL Traces can help for individual forms (open that form only and see what is occurring) If opening individual form is fine, could be interaction via watchers Take opposite approach, remove one form at a time, see if performance improves. 18

Form Performance Operating slow Watchers Heavy MEL symbol use Interaction with multiple items on form 19

eRx (e-Prescribing) -Network latency -Registration information bad 20

Troubleshooting Instability Issues 21

Instability 22 Patterns: Help Us Help You Crash DumpErrorLog Table Workflows? Time of day? Single user / Multi-user? Located in /appdata/local/ centricity/logs Two files for each dump,.txt and.dmp Allows us to see where issue occurred Not all crash dump files indicate a user had the application exit Only available in CPS Shows fatal and error categories NOTE: Error category shows many information messages, fatal is where most useful information is.

ErrorLog Table Queries What error caused crash dump file? select a.LogTimestamp, b.LogTimestamp, a.WorkstationID, a.ProcessId, b.ProcessId, a.winloginname, a.LogMessage, b.LogMessage from errorlog a join errorlog b on b.WorkstationId = a.WorkstationId and b.WinLoginName = a.WinLoginName and b.LogTimestamp = (select top(1)c.logtimestamp from ErrorLog c where c.WorkstationId = a.WorkstationId and c.WinLoginName = a.WinLoginName and c.LogTimestamp < a.LogTimestamp order by LogTimestamp desc) where a.logmessage like '%crash dump%‘ and a.logtimestamp > ' ‘ order by a.processid, a.LogTimestamp What was the user doing at time of error? 23

Errorlog Queries (cont.) What was the user doing? (thanks to Derrick Bovenkamp, Family Care Network) varchar(30) date = '1/12/2015' ---whatever date you want to start with = '' ---Username select e.LogTimestamp as [timestamp],e.WinLoginName as [user],e.ComputerName as [computer],'ErrorLog' as [Source],e.LogLevel as type], e.LogMessage as [message1], e.ExceptionTrace as [message2],'' as [patientID] from ErrorLog e where ( ---filter out some PM-specific diagnostic logging e.LogMessage not like '%OnHSInitDialog%‘ and e.LogMessage not like '%CDlgAttachAcquirePatient%‘ and ---filter out diagnostic logging for SPR e.LogMessage not like '%_UseRxRefillFixAlgorithm%‘ And ---filter out known erroneous logging not known to contribute to performance issues e.LogMessage not like'%Multiple-step OLE DB operation generated errors%IDispatch error #3105%INSERT INTO DBO.AUDIT_EVENT%' And e.LogMessage not like '%DuplicateUILayoutException: UILayout record already exists%' AND e.LogMessage not like '%com.gehcit.cp.domain.model.client.UILayoutNotFoundException: UILayout record not found%‘ ) and e.WinLoginName and e.LogTimestamp UNION select ae.EVENT_TIMESTAMP as [timestamp], ap.LOGINNAME as [user],ap.OS_MACHINE_NAME as [computer],'AuditLog' as [Source], aet.EVENT_TYPE_DESCRIPTION as [type],aed.EVENT_VALUE_1 as [message1],aed.EVENT_VALUE_2 as [message2],p.PATIENTID as [patientID] from AUDIT_EVENT ae inner join AUDIT_EVENT_TYPE aet on aet.EVENT_TYPE_ID=ae.EVENT_TYPE_ID inner join AUDIT_PROFILE ap on ap.PROFILE_ID=ae.PROFILE_ID inner join AUDIT_EVENT_DETAIL aed on aed.EVENT_ID=ae.EVENT_ID left join PERSON p on p.PID=ae.PID where ap.LOGINNAME and ae.EVENT_TIMESTAMP ORDER BY [timestamp] desc 24

Summary

Key Takeaways There are many areas outside the application itself that can cause performance issues Tools and techniques that you can use for self help Information that can be gathered to better help GE Healthcare help you

Questions?