1. JoAnn Logue-O’Malley, Beaumont Health Roseann Ferrara, SCC
The Installation Wagon Trail
System Changes In Your Frontier
JoAnn Logue-O’Malley, Beaumont Health Roseann Ferrara, SCC

2. Workshop Objectives
Understanding the installation process for implementation of system changes
Determine improvements in your installation process
Improving your validation and general functionality testing leading to less risk at installation
Developing a downtime Run Book
8:30 to 10:00
10:00 to 10:15 Break
10:15 to 11:30

3. Client Validation Form
Your Validation Steps
Key Project Milestones
Milestone
Currently use
Modify
Add 3
Please complete the steps you take to validate new software, hot fixes, patches

4. Workshop Terminology SIF = Software Installation Form
SYNCH = Software Synchronization
SIF = Software Installation Form
SWI = Software Installation
SSS = Software Support Service
DM = Development Manager
SE = System Engineer
AE = Account Executive/Account Manager
CSR = Client Support Representative
VP OPS = Vice President of Operations
CAB = Change Advisory Board
CCB = Change Control Board
IC = Software Installation Coordinator
TSS = Technical Support Specialist (Post Live Support)

5. Client Requests Software Installation Implementation – Upgrades
Your AE will contact clients to schedule additional deliveries or HF’s available to discuss scheduling. Clients may also work with their AE’s as well
Via
Via Conference Call

6. Requests for Software Installation Post Live Patches and HF’s
Client works with TSS to determine installation scope and either Client or TSS creates SSS task
Once scope is finalized the AE is notified by the TSS that the client needs to have an SWI
AE notifies Post Live Load Coordinator Team of request for SWI
Post Live Installation Coordinator will work on SWI form creation for SIF or SYNCH and approvals and keeps AE’s posted
For afterhours or weekend supported installations the load coordinator contacts the AE for after hours quote who will work on quote/PO and DEDICATED resource assignments

7. TO SIF OR SYNCH? THAT IS THE QUESTION
For NEW ENVIRONMENTS in preparation for example for an upgrade or patch project the following occurs
SYNCH OF CURRENT LIVE SOFTWARE AND SETUPS IS DONE 1ST TO THE NEW ENVIRONMENT
SIF IS DONE TO THE NEW ENVIRONMENT AFTER SYNCH IS COMPLETED
AFTER NEW ENVIRONMENT IS COMPELTED SETUPS NEED TO BE MAINTAINED MANUALLY BETWEEN THE TWO ENVIRONMENTS

8. TO SIF OR SYNCH? THAT IS THE QUESTION
SIF = Software Installation and is used to Install/Uninstall Software ONLY (NO SETUPS)
SYNCH = Software Synchronization from a SOURCE environment to a TARGET environment
SYNCH’s may copy Software AND Setups OR EITHER ONE .
The above is applicable to New (non-Implementation) Environments, Upgrades and Patch Projects
It an SCC change management policy to never plan to install software directly into a LIVE production client environment. Therefore a SIF is used to install to an alternate non-production environment 1st then SYNCH to LIVE production.
The only exception to this policy is when there is an emergency system down or safety situation and an alternate and equivalent environment is not available at the time.
It is important to note that when NEW SOFTWARE is planned for a New Environment that is NOT a NEW IMPLEMENTATION where current software and setups are being used to create the environment that SETUP SYNCH is ALWAYS the 1st step in the process.
This is because once new software is loaded setups CANNOT be recopied again and clients MUST keep setups in synch between the source and target environment until go live. This is due to changes in the DB structure.

9. SCC Processes SWI Request and Creates Form
AE (Implementation/Upgrades or Load Coordinator (Post Live Patch and HF installations) Creates Software Installation Form
LIVE DEMO

10. SCC Approval of Software Installation Form(s) (Finalizes Scope)
Release Management Reviews SIF/SYNCH Form(s)
Form approval process for DM – SE – VP Approvals
LIVE DEMO

11. SCC Notification to Client of Final Approved Scope of Install by AE or Load Coordinator
Once notification of form approval is received by the AE or Load coordinator the following is done
Review Downtime Estimate
Attach Software Summary to SWI task for client review
Note: Software Summary contains the final scope of the installation including any recalled or mandatory HF’s
Places the SWI task in SW status with an action to the client about downtime estimate, attached HF summary and available installation dates
Requests permission to proceed from the client to install and request for confirmation of date and time. In of LIVE production contact information is also requested for contact 15 minutes prior to install for final permission to proceed
LIVE DEMO

12. Client Steps- Change Management Submission and Approval Process
Client selects installation date from dates AE provides in SWI task
Client prepares required paperwork for submission to their respective change advisory boards (CAB) for approval and scheduling of the installation to TESTx or UPGx etc. environments
Client processes through weekly Change Advisory Board
Client notifies SCC of selected and approved date/time for the installation

13. SCC Processes SWI Installation Request from Client
AE or Load Coordinator (Post Live) Books the Date of SWI Install on the SE Calendar
On the Date of Install the assigned SE performs the Installation and actions task to AE or Load Coordinator (Post Live) when done
In Case of LIVE production installs a call is made to client mins ahead of time for FINAL confirmation of the install.
For Implementation / Upgrade Projects SCC tests the install FIRST then places the task in TW or SW status for the client to perform validation before the SWI task is closed.
For Post Live Patch Installs SCC also tests the install FIRST then the Load Coordinator closes the SWI task and notifies the client to open up new SSS tasks for any issues found during testing.
For Post Live HF Installs the client is notified install is complete and SWI task is closed. Client will open SSS tasks during their testing for any issues created and support will address them
Additional Note: for Quoted Afterhours SWI events involving DEDICATED support sometimes an additional conference line is set up between client and assigned SCC personnel for additional communications. In some cases additional testing is requested to be performed by SCC and SCC stays on line through the client testing process
LIVE DEMO
SWI Calendar / Installation

14. SCC Technical Support Release Notices
Client can review the patch or upgrade items via SCC website/Technical Services/Release notes
LIVE DEMO
Release lists

15. Design your validation process
Validate each Hot Fix
Validate each patch release note item
Use SCC test plan
Validate client workbook
Validate using general functionality test plans
Cap Requirements

16. Client - Retrieve HF Summary List (from TMS SWI Task)

17. Release Notes Query

18. Client - List of Patch Items
Client retrieves list of patch items from SCC web site
Release notes
Version on to Version moving to

19. Client - Example List

20. Client - Export to Excel

21. Example Test Case

22. Client Workbook

23. Client Workbook Summary

24. General Functionality Test Plans
Basic test plans to validate system integrity
Orders/results through HIS

25. General Functionality Test Plans

26. CAP Requirements

27. CAP Requirements

28. CAP Requirements

29. CAP Requirement

30. CAP Requirements

31. Option 1: Small Scale Hot Fix Bundles
Handy Tools in the tool box
Folders or SharePoint site -all users have access
List of all delivered Hot Fix items
List of general functionality testing scenarios
List of downstream interface workflows (include customization on IE)

32. Option 1: Small Scale Hot Fix Bundles

33. Client - Processing Hot Fix List
Client Processes List
Recommended Steps
List items need to be managed for testing, validation and Ready for Live Status
Add hot fixes to list such as excel or SharePoint
Hot fix number
Module
Description
Assigned to
Status
Associated TMS if applicable
Attach test plan
Notes

34. Hot Fix Testing Template

35. Loads to Live List

36. General Functionality Testing

37. Client - Management Views

38. SharePoint List – Manage Views

39. Option 2: Large Scale Patch or Upgrade Projects

40. Option 2: Large Scale Patch or Upgrade Projects
Handy Tools in the tool box
Folders or SharePoint site -all users have access
Document Storage
PowerPoint update presentations to leadership
Agendas for meetings
Module processing workflows (Visio or PowerPoint)
Downstream interface workflows
Project Plan

41. Project Plan
Project Plan

42. Project Dashboard

43. Project Dashboard

44. Option 2: Large Scale Patch or Upgrade Projects
Business operations issues

45. Business Operations List

46. Option 2: Large Scale Patch or Upgrade Projects
Technical Issues list

47. Technical Issues List

48. Option 2: Large Scale Patch or Upgrade Projects
Build list

49. Build List

50. Option 2: Large Scale Patch or Upgrade Projects
Decision Document
Tool to escalate to leadership for an executive decision driving direction for issue resolution
Outlines issue and potential solutions/workarounds
Documented record of decision

51. Decision Document

54. Option 2: Large Scale Patch or Upgrade Projects
Validation Tools
List of all delivered Hot Fix items
List of general functionality testing scenarios
List of Unit tests (orderable test selection)
List of Module processing scenarios
List of Cross module processing scenarios
List of business unit operational Scenarios
Validation Defects List

55. Client - SharePoint List

56. Client Preparation for Validation
Client Additional Processes for Patch and Upgrade
For Each Module Unit Testing (per orderable)
For each Module Full Functional Scenarios
Cross Module Full Functional Scenarios
Interface and Interface Engine Customization Scenarios
Vallapalooza

57. Client Preparation for Vallapalooza Validation
Client Additional Processes for Patch and Upgrade
Schedule a workroom with PC, labels, report printers
Schedule analyst and lab super-users
Label workstations as to module or function
Execute all test plans
Vallapalooza

58. SCC - Vallapalooza Support
Develop communication plan outlining Key Contacts at SCC
Develop scheduling plan for Key contacts to be made available to respond to any immediate issues with client testing
Deliver additional fix and/or system and interface configurations in a timely manner
Assist with meeting clients Vallapalooza timelines on repeat testing

59. Vallapalooza Unit Testing

60. Vallapalooza Scenario Testing
Workflow scenarios
Within module – complex testing
Cross module
Operational Workflows

61. Scenario Script

63. Vallapalooza Scenario Testing

64. Option 2: Large Scale Patch or Upgrade Projects
Defect List – from Vallapalooza
List to work defects through to completion
Defects can be for various phases of project
Defects can be build, vendor, interface, etc. related
Once fixed pushes items back for retesting

65. Defect Vallapalooza List
Validation Phases
Issue Status
Error Code

66. SharePoint Vallapalooza Defects List

67. Client - Management Views

68. Release Note Progress

69. Client Workbook Progress

70. Leadership Status Updates

71. Scenario Execution Status By Functionality/Module

72. Vallapalooza Total Completion 95.4%

73. Validation Completion Status By Module for each OE System

74. Option 2: Large Scale Patch or Upgrade Projects
Education Development
Bulletin Master
Procedure Master
Training Master

75. Client to Call “Go for Load”
Review SharePoint List for Completion
All outstanding issues have been mediated from Vallapalooza

76. Break – Yee Haw
See ya’ll in 15

77. Client Downtime Run Book Steps
Your Downtime Run Book Steps
Key Project Milestones
Milestone
Currently use
Modify
Add
7777
Please complete the steps you take to conduct a downtime for implementation of new software, hot fixes, patches

78. SCC – Process SWI Scheduling Request for SYNCH Implementation – Upgrades
AE creates SWI task for SYNCH to x environment
Software Installation Coordinator adds/schedules SWI to SE Calendar ……………..
SE installs SWI and passes for internal testing by SCC
Note - for LIVE Production SWI’s client client contact is made for FINAL go forward permisision
AE notifies Client that testing is complete
Client validates and notifies AE when SWI task may be closed
Client opens New SSS task for any new issues
For Installs Requiring Dedicated Support
Client submits RFQ for dedicated support coverage.
Note - For SCC to proceed on a dedicated event a signed quote, PO and 50% dyment is required.
Once all the above items are received AE works with SE -DM to line up dedicated resources for Installation Manager to contact for issues that might occur
SE installs the SWI and passes task to SIC
AE Monitors with Software Installation Manager

79. SCC – Process SWI Scheduling Request for SYNCH (Post Live)
Release Management creates SWI task for SYNCH to x environment
Software Installation Coordinator adds/schedules SWI to SE Calendar ……………..
SE installs SWI and passes task back to SIC
Note - for LIVE Production SWI’s client client contact is made for FINAL go forward permisision
SIC (PL) closes SWI task.
Client opens New SSS task for any new issues
For Installs Requiring Dedicated Support
Client submits RFQ for dedicated support coverage.
Note - For SCC to proceed on a dedicated event a signed quote, PO and 50% down payment is required.
Once all the above items are received AE works with SE -DM to line up dedicated resources for Installation Manager to contact for issues that might occur
SE installs the SWI and passes task to SIC
AE Monitors with Escalation Manager

80. SCC Calendar Scheduling Process
SCC – Process Client Request for SYNCH to Production SWI Scheduling (Implementation/Upgrades)
LIVE DEMO
SCC Calendar Scheduling Process

81. SCC Processes SYNCH Form
AE (IMP/UPG) or PL synch support rep creates SYNCH FORM
(uses TMS Task # from SWI task)
LIVE DEMO

82. SCC Creates SWI task for SYNCH
SWI task is created with description of Software Load
LIVE DEMO

83. SCC Processes SYNCH Form Synch Approval Process
Release Management and DM Review SYNCH Form
AE/SCC marks Synch form as client validated and verifies
SE approves and provides final downtime estimate
VP of Operations approves
LIVE DEMO

84. Load coordinators will confirm if ok or not
SCC scheduling SYNCH
IMP/UPG -- AE Checks with Software Installation Coordinators (IC) for available installation dates to present to client for scheduling
PL -- client determines date/time of Go Live within given time frames provided by SCC.
Load coordinators will confirm if ok or not
LIVE DEMO
SWI Calendar

85. SCC – Process SWI: Scheduling of SYNCH TO PRODUCTION
Installation Manager requests breakout of downtime by module for presentation to client from SE –DBA teams
Installation manager attaches breakout in action to client for review with copy to AE
AE works with client to finalize date
Client actions task with permission for installation with date/time
AE and Installation Manager send notification to SE –DBA-Programming teams and 1800SOFTLAB Support
***Does not pertain to HF process

86. Things that can derail your date / possibly require more testing
Dependencies
Mandatories
Found by SCC or client base that have been rolled into patch
Based on hot fixes added these can have additional dependencies
Encountered if you have taken hot fixes and patches over a period of time without rolling to live
Intentional
Hot fixes to hot fixes

87. Client - Request Time Line for Installation Steps by SCC to LIVE
Discuss with SCC the Date and Time for Loading to Production Environment
Enter Change Management for installation
A Timeline and Module impact is critical to determining messaging and communication
Minimize downtime Internal to Lab
Orders
Results
Downtime Number assignment
Minimize downtime External to Lab
Delay of results
SOFT ID and collection throughout hospital facilities

88. Downtime Run Book –Prep Step
Place holder for Date
Create Change Management Request
Alert Lab Leadership for any additional staffing needs

89. Client - Change Management (example)

90. Downtime Run Book –Prep Step
Is there a need to educate
Determine target audience groups
Determine informational to deliver
Determine educational tools
Determine how long it will take to complete education
Determine verification of training completed

91. Laboratory Education Library

92. Education

95. Education

97. Downtime Run Book –Prep Step
Determine if you will use downtime labels
Prepare enough for length of downtime
Procedure to create downtime labels
Procedure to use downtime labels

101. Downtime Run Book –Prep Step
For Extended downtimes
Review Lab downtime procedures
Report delivery
Called Stats
Review Nursing downtime procedures
Use of white board order management

104. Downtime Run Book –Prep Step
Call Center
Determine if needed
Create schedule
Set up call center rooms with PCs, phone bank
Set up Phone triage
Set up call center issues list
Schedule daily hospital leadership update conference calls
Prepare PowerPoint template to include
Critical Issues
Stats on TAT

105. Call Center Schedule

106. Call Center Schedule

107. Call Center Floor Plan

108. Call Center Phone Triage

109. Call Center Phone Bank

110. Daily Leadership Updates

111. Call Center Metrics

112. Issue Tracker Metrics
Issues completed

113. Client and SCC Prior to Go Live Date
Client actions task with permission for installation with date/time at least one week prior to LIVE date
LIVE date should be discussed and planned much further in advance (2-4 weeks) to secure date and scc resources
AE and Installation Manager send notification to SE –DBA-Programming teams and 1800SOFTLAB Support (for Post Live Load Coordinator does this step)

114. Downtime Run Book Week Prior to Go Live Date
Communicate installation plans to Laboratory Leadership
Determine if there are any concerns
Make sure there is agreement for roll out process and timeline
Last planning conference call
Readable list of major hot fixes, system updates

115. Downtime Run Book Week Prior to Go Live Date
Notification to Service Desk and Operations Center
Please be advised SCC has given us a time frame of 3-4 hours. We are publishing the downtime to be from 1AM to 5AM. Hopefully we come back up on the shorter estimate.
I will page to let everyone know how we are doing at 4AM.
EPIC orders will be released at 11PM. Please prepare for an extended downtime to cover sweeps Just in case we need the full 4 hours.

116. Downtime Run Book Week Prior to Go Live Date
Service Request for early release of EPIC lab orders for sweeps

117. Downtime Run Book Week Prior to Go Live Date
Notifications sent to Infrastructure teams
Disable all server reboots for downtime
Citrix Team Lead
Typically no assistance is needed just FYI
SCC pushes via SOFT Updater to all Citrix servers
Monitoring processes will be interrupted
Whats Up Gold
CompuWare
Server Dashboards

118. Downtime Run Book Week Prior to Go Live Date
System Wide End User Communications Go Out
communication – IT Bulletin
Newsletter communication-Leader Links
Communication – Daily Dose
Hallway posters

119. IT Bulletin

120. Daily Dose

121. Leader Notifications

122. Downtime Run Book Week Prior to Go Live Date
Schedule IT or Lab Testers
Pager or contact information
Assigned tasks and general functionality templates to complete

123. Schedule for Testers

124. Downtime Run Book Week Prior to Go Live Date
Create communication tool for use after install for validation with testers
Web Ex
Lync
Recorded voice and IM

125. Downtime Run Book Week Prior to Go Live Date
Create communication forum for use during install between client and SCC
Create a Web Ex or Lync as back up if needed
Send to participants
between point persons (IT installation analyst and SCC installation analyst)
Send as placeholder

126. Downtime Run Book Week Prior to Go Live Date
Reports
Do not schedule any reports for kick off during downtime
AR jobs reschedule
Move its

127. Downtime Run Book Week Prior to Go Live Date
Microbiology Downtime Server
Server and report printer which capture prelims every 12 hours.
Converted to PDF files
Can be printed on demand
Verify last file back up was successful

128. Downtime Run Book Week Prior to Go Live Date
Update Test Patients with new Billing number
Send Lab Team reminder to review procedure for testing on test patients in Production
ward/physician use
minimize leak to a real client or report physician
JoAnn- I went ahead this morning and updated the visits in SOFT so that they route to our dummy wards ROCG, TRCG, GPCG.  They’re all ready for tomorrow.  Visits are:

129. Downtime Run Book Load to Production
Confirm Look ahead time is set for greater than 3 hours to cover if delay in system being returned

130. Downtime Run Book Load to Production
IT analyst
Verify EPIC Orders were released by 11PM
Procedure is for EPIC analyst to page

131. Downtime Run Book Load to Production
IT analyst
Reminder call to Service desk
Update Hot Line message
Assume On Call pager
Open Line of Communication between Client and SCC at 20 mins prior to installation
Record milestones, issues
Respond to any additional permissions as needed

132. Downtime Run Book Load to Production
Production is taken down
SCC installs installation items as outline to plan
SCC and IT Analyst periodically touch base to verify milestone updates and timeline on track

134. Downtime Run Book Load to Production
SCC completes installation
SCC performs internal testing if applicable
SCC releases system

135. Downtime Run Book Load to Production
Page testers to begin General Functionality testing
Call key lab areas such as Stat Labs to verify no immediate concerns
Page Lab Leader pager group with update
Open Lync for Testers – start recording
Testers note progress
Testers note issues and TMS numbers
with SCC still open
Provide updates to SCC implementer on progress
Report issues and TMS numbers
Work to resolve all open issues

137. Downtime Run Book Load to Production
System not up as expected
Make immediate assessment and decision regarding notifications and incident command execution
System up with issues
Determine impact of issue on end users
Work with SCC to resolve
Document TMS numbers, Issue and Resolution steps
Include time started and resolved

138. Use of Incident Command

139. Dedicated Support or Large Go Lives Command Center or Conference Line
IT Staff and SCC join a Technical Session
Customers join the Customer Session
Hosted by an IT Leader
Incident Commander engaged to work communications

140. Open a Customer Communication
This is an opportunity to listen to the customer/user to obtain additional information about the incident request to help the investigation, as well as provide confidence to the user that their incident issue has been received, logged, has an owner and is actively being worked to restore service as quickly as possible.
Start a Microsoft Lync session for incident customer/user communication, if needed for more effective user communication than sending a routine incident notification message. 
Use the pre-established number and Conference ID for an incident customer/user update MS-Lync session.  Use this direct link to

141. Create Hot Line Message
The IT Systems Downtime Hotline Message is a recorded message that resides at the following phone number
This message is used to communicate statuses/updates when IT Systems are unavailable or are functioning in less than optimal manner and it is necessary to communicate statuses to many users. This service is used for Scheduled and Unscheduled Downtimes

142. Load is Completed Close Lync Session with Testers
Store recorded file on Change Management Entry
Send out all clear page to lab leadership
Return pager assignment to on-call person
Return SOFT ID roles back to look ahead 3 hours
Close Change Management
Attach session recordings/summaries
Close with SCC
Compose and send a Summary of the Load and send to IT and Lab Leadership

143. Leadership Notification
Provide Summary of downtime to include milestones and issues to IT and Laboratory Leadership

144. Completed Validation Run Book

145. Completed Validation Run Book

146. Completed Downtime Run Book

147. Completed Downtime Run Book

148. QUESTIONS Our Software Installation Workshop is now over
Hope you found this presentation informative and enjoyable 
JoAnn and Roseann