1. Installation and Maintenance of Health IT Systems
Creating Fault-Tolerant Systems, Backups, and Decommissioning
Welcome to Installation and Maintenance of Health IT Systems, Creating Fault Tolerant Systems, Backups, and Decommissioning, This is lecture c.
This component, Installation and Maintenance of Health IT Systems covers fundamentals of selection, installation, and maintenance of typical Electronic Health Records (EHR) systems.
This unit, Creating Fault Tolerant Systems, Backups, and Decommissioning, will discuss ensuring availability and resiliency through fault tolerance, data reliability through backup, and secure decommissioning of EHR systems
Lecture c
This material (Comp 8 Unit 9) was developed by Duke University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number IU24OC This material was updated in 2016 by The University of Texas Health Science Center at Houston under Award Number 90WT0006.
This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. To view a copy of this license, visit
Health IT Workforce Curriculum Version 4.0

2. Creating Fault-Tolerant Systems, Backups, and Decommissioning Learning Objectives
Define availability, reliability, redundancy, and fault tolerance (Lecture a)
Explain areas and outline rules for implementing fault tolerant systems (Lecture a)
Perform risk assessment (Lecture a)
Follow best practice guidelines for common implementations (Lecture b)
Develop strategies for backup and restore of operating systems, applications, configuration settings, and databases (Lecture c)
Decommission systems and data (Lecture c)
The objectives for this unit, Creating Fault-Tolerant Systems, Backups, and Decommissioning are to:
Define availability, reliability, redundancy, and fault tolerance
Explain areas and outline rules for implementing fault tolerant systems
Perform risk assessment
Follow best practice guidelines for common implementations
Develop strategies for backup and restore of operating systems, applications, configuration settings, and databases and
Decommission systems and data
As healthcare organizations adopt new technology to improve their efficiency, their dependence on that technology increases exponentially. However, what happens to all of these critical applications if a failure were to occur? What about the integrity of the caregiver’s data in the event of a disaster?
In lecture c, we will finalize our discussion and outline some backup strategies. And since we are on the subject of backing up, we will finish with some tips on archiving and decommissioning data and hardware.
Health IT Workforce Curriculum Version 4.0

3. Importance of Backup
Volume of data: hospital can generate 12 terabytes/yr in radiology alone.
HIPAA (Health Information Portability & Accountability Act) Security Rule requires exact backup copies of all healthcare data, easily retrievable
Should be called “Importance of Restore”
(Surviving a Bottleneck, 2009)
Healthcare institutions must now cope with the need to retain vast and ever-increasing quantities of medical data for protracted periods of time to safeguard themselves and their patients.
The average 100-bed hospital generates anywhere between 40,000 to 45,000 radiological examinations yearly, equating to approximately 12 Terabytes of space needed for storage of these documents alone. This figure doesn’t even count the vast amounts of data generated yearly from billing, staffing and administration, and other typical needs of a healthcare setting.
The HIPAA , or Health Information Portability & Accountability Act, Security Rule establishes the requirement to keep exact backup copies of all healthcare data that can be retrieved in a timely manner to restore documentation, should data be corrupted or lost.
Think of a backup as a first step – the real reason to make a backup is not the backup, but the restoration of what was lost.
Health IT Workforce Curriculum Version 4.0

4. Backup Strategies: Requirements
Laws regarding length of time health information data must be retained depend on the jurisdiction (usually state), and can involve:
Flat length of time (X years)
Age of patient
Time since age of majority, or of discharge, or of death
Length of statute of limitations for malpractice
What constitutes best practices for a backup?
Exact, verified copy of the material – Multiple copies!
Stored off-site location in case of natural disaster, fires, flooding, etc.
Easily retrievable for timely restoration
Security via encryption and storage in secure location
Fault tolerant storage protection (like RAID) is not enough
Deleted files
Old versions
(AHIMA Practice Brief, n.d.)
Besides federal regulation such as HIPAA, state laws often describe retention requirements for health information. They may look at a fixed amount of time, or the age of the patient (especially for minors) or of the health record, time since discharge or death, or malpractice suit statute of limitations regulations.
Let’s take a look at best practices for general backups. The copy of the information should be verified to ensure its correctness. A backup that cannot be restored is not a backup at all. Additionally, multiple copies, with a copy of the data at a location off-site geographically to protect it from natural disasters, fires, flooding, and such. The data must be easily retrievable so data can be restored in a timely fashion. The data must be encrypted for security, especially if stored off-site or transported.
Note that RAID or other fault-tolerant systems (as discussed in lecture b) are NOT a substitute for backup. RAID does not protect against file deletion, or help in recovering older versions of the data. Backups can.
Health IT Workforce Curriculum Version 4.0

5. Backup Strategies: Backup Window
Time required to complete a given backup.
Determined by amount of data to be backed up divided by speed of network infrastructure
Backups that occur during production hours may be inconsistent (bad)
Problems when backup window reaches peak operation cycles, potentially straining resources and slowing down the system
What to do when system must be available 24/7?
(Harwood, 2003)
Another issue you need to consider when developing your backup strategy is how often and when you will complete your backups. Backups, which can sap network bandwidth and hinder access to resources should be conducted, whenever possible, in a manner that reduces performance issues during peak cycles.
A ‘backup window’ is “… the time it takes to complete a given backup. This backup window is determined by both the amount of data that must be backed up and by the speed of the network infrastructure that handles the data.” In a small organization, backups can complete in a small window, outside of production hours. An off-hours backup is advantageous because you may assume no changes to the data will be made during the backup. Any change to a backup mid-stream may lead to an inconsistent (and therefore flawed and possibly useless) backup.
However, as the amount of data increases, backups may extend into production hours, introducing that possibility and impacting system speed as well.
Finally, many systems are expected to run 24/7, and have only production time in which to backup.
Health IT Workforce Curriculum Version 4.0

6. Backup Strategies: Which Files?
Full backups
All files
Pro: Ultimate protection, simple restoration
Con: Requires more time & lots of storage to keep multiple file versions
Incremental backups
Only files that have changed since last backup (full or incremental)
Pro: Much faster
Con: Restoration from multiple files
(Harwood, 2003)
There are different types of backups which can be run, depending on your specific needs. Each has its advantages and disadvantages:
Full backups save all files, and provide a convenient restore because all of the programs and data needed are in the same backup. The largest drawback to a full backup is the size of the backup and the time required. Retaining multiple versions of backup data can make the size requirements grow quickly.
Incremental backup is faster because it saves only the copies of files which have changed since the last backup. For systems where a relatively few number of files change each day, this can save tremendously on storage. The drawback is that restoration requires access to multiple backups, increasing the time and effort for a restoration. Well-tended data libraries are recommended for incremental backups to allow easy identification of the proper backup from which to restore.
Health IT Workforce Curriculum Version 4.0

7. Backup Strategies: Which Files? (cont’d)
Differential backup
Middle ground: all the files that have changed since the last full backup
Pro: easier restoration
Synthetic full backup
Compensates for small/nonexistent backup window
Data from last full backup + differential / incremental backup combined to create new full backup tape
(Harwood, 2003)
Differential backups reduce the restoration problems from the incremental backup. Now only two backups are needed: the last full backup, and the last differential. This works by copying all data that has changed since the last full backup. It does mean, however, that the size of a differential backup will grow over time, eventually nearly reaching the size of the full backup.
Synthetic full backups are generated by merging a full backup with an incremental to allow for on-stop restoration. While it combines the speed of an incremental with the easy restore of a full, it does take some post-processing to merge the data. This increases the complexity of the backup, and the potential for error.
If the majority of your data files change frequently, then full backups are likely the best option
Health IT Workforce Curriculum Version 4.0

8. Backup Strategies: Which Files? (cont’d - 1)
File system snapshots
Available through VM environments and later UNIX versions
Backups at several times through the day without needing large amounts of additional storage media
Reliable backups without shutting down applications
(Harwood, 2003)
Snapshots are a recommended backup method for systems that have no downtime or off-hours. This feature, often implemented in a platform specific environment, allows the data to be “frozen” so a backup is taken of guaranteed consistent data, no matter how long the backup takes. It does this by writing all changes to data in a temporary area during the snapshot, which is used transparently for all other system access. Then once the backup is complete, the data in the temporary area is merged back with the primary system.
The biggest benefit of file system snapshots is that they allow backups on live data without disabling application access.
Health IT Workforce Curriculum Version 4.0

9. Backup Strategies: Where to store data
Direct backup
Tape drive / autoloader / library directly connected to every server to directly backup and restore data
Does not scale well
Centralized network backup
One server backs up data of all servers
SAN (Storage Area Network) backup
Storage network to which all servers & backup device connect
(Gordon, n.d.)
The most straightforward backup is to copy the data to an attached tape drive, optical drive, or other file storage system directly connected to the server. This is fine for small environments, but for multiple servers, tracking the storage media may be difficult.
Having a single backup server that connects to other networked servers is the next step up. This allows for extensive configured tracking of backed-up data in the backup server, and scales well. Also, it allows flexibility in restoration – anything that is on the network may be given a copy of freshly restored data.
At still larger scales, the Storage Area Network (or SAN) is a separate system of interfaces and connections between data servers. A SAN will provide bulk data storage for the network, and have provision for backup of that data on the same network, usually integrated by the storage vendor.
Health IT Workforce Curriculum Version 4.0

10. Backup Strategies: Backup Media
Storage of media should cycle off-site
Tapes
Robust but expensive
Optical
Inexpensive, relatively small, convenient
Flash
Robust, convenient, fast
Hard disk
Cheap, fast but fragile, bulky
Network or Cloud backup
Dependent on network throughout
Backups will start on-site, but any media that are created should periodically be stored off-site. The media to use for a backup are available in several types. Tapes are historically the most robust, but also relatively slow and the media is expensive. Storing large amount of data on tapes can create a physical inventory problem. For several years, the capacity and cost of hard drives have greatly outdistanced that of tapes.
Optical media is familiar and relatively inexpensive, and is readable for easy restoration without specialized equipment, as optical drives are standard equipment on most computers. They hold relatively little data though, and can be fragile. Their small space is attractive when considering physical size requirements of storage.
Flash media is mentioned because of its ubiquity. USB memory sticks are available in fairly large capacity at increasingly attractive prices, and the media is robust. Their maximum capacity can mean difficulty scaling to very large data sizes.
Backup to hard disk may sound oxymoronic, but because of advances in hard drive technology, huge amounts of data can be stored inexpensively and quickly. Because of their bulk and relative fragility, hard disks need careful physical storage.
Finally, backup to network or cloud locations, while not technically media, depend on the network throughput to the backup site. Processes that continually run in the background, updating any changes to an off-site backup, are an ideal solution as long as the cost and network bandwidth are available to support it.
Health IT Workforce Curriculum Version 4.0

11. Backup Strategies: Databases
Databases require extra considerations, depending on the database infrastructure used
Consult with database or EHR vendor to ensure backup strategy is compatible with database infrastructure.
Database backup is usually through specialize tools or applications, often provided with the database.
Database backup requires extra considerations. Before embarking on a backup strategy for your EHR databases, consult with your EHR vendor to ensure your backup strategy is compatible with your database infrastructure. They should have outlined best practices for your system. Often the database or application vendor will provide specialized tools or additional applications to backup.
Health IT Workforce Curriculum Version 4.0

12. Decommissioning
Goals in retiring old systems / applications / datasets
Active data properly retained
Inactive data archived or disposed of securely
Tips
Complete full data audit; note redundancies.
Determine data owners and stakeholders.
Identify active vs. inactive data.
Consider reporting / retrieval requirements for compliance.
Legacy systems are often maintained simply to reference historical data, sometimes at great cost to the organization. At some point, systems or applications past their prime, or datasets which must be retired, must be evaluated and dealt with in a manner that ensures that active data is properly retained and inactive data is archived or disposed of securely, consistent with the organizational needs.
Here are some tips for identifying and decommissioning legacy systems:
Complete a full data audit and identify the data you are collecting and retaining and note redundancies.
Complete a full inventory of EHR systems and determine what kinds of data you collect and retain, and require. This is a complex task, since managing the many overlaps and redundancies will make it confusing as to which application is being used for which purpose.
Determine who owns the data so you can work with them to resolve any compliance or archiving requirements.
Identify which data is inactive and which data is still active. This is done through both using logs and conferring with data owners and stakeholders.
Remember, archiving and retrieval of data, particularly in the healthcare arena, is a lifelong commitment. Be sure to plan adequately for archiving and tracking the data for compliance even once it’s off the servers.
Health IT Workforce Curriculum Version 4.0

13. Decommissioning (cont’d)
Tips (cont’d)
Document retention policies well & ensure consistency with government guidelines.
Standardize on single, well-navigable archival system.
Develop decommissioning plan & schedule.
Ensure integrity of archived data and destruction of decommissioned data.
Be sure your retention policies are well documented and are consistent with federal and state guidelines.
Be sure to standardize on a single, well-navigable archival system. This makes locating archived data easier and faster.
Develop a plan and a schedule for decommissioning. Be sure to notify your data owners and stakeholders of the event, what will happen to their data once the application is decommissioned, and any potential impacts or replacement applications brought online.
Once the server or applications are decommissioned, ensure the integrity of any archived data. Remember, simply erasing data from decommissioned hardware using conventional means is not enough. Data erased in this fashion can be retrieved using simple utilities. Be sure to render storage media useless or ensure it has been erased according to industry standards.
Health IT Workforce Curriculum Version 4.0

14. Creating Fault-Tolerant Systems, Backups, and Decommissioning Summary – Lecture c
Regulatory requirements for backups are stringent
An effective backup strategy minimizes the backup window while ensuring data integrity.
Backup considerations:
Onsite vs Off-site
Full vs Partial
Media
Verification
Decommissioning
Active data properly retained
Inactive data archived or disposed of securely
This concludes Creating Fault-Tolerant Systems, Backups, and Decommissioning.
Let’s take a quick moment to summarize the important points presented in this unit:
Regulations require healthcare institutions to keep exact backup copies of all healthcare data. That data should be protected, encrypted and stored in multiple locations to protect it from foreseeable harm for the duration of its retention period.
Backups often will occur during a limited timeframe, or backup window. As the amount of the data needing to be backed up increases, generally, so does the backup window. It is important to develop a backup strategy that minimizes the backup window while ensuring data integrity.
Consider on versus off-site and full versus partial backups, as well as the type of media to use. However, always remember that any backup without a restore is useless – verification of stored data is critical to ensure availability.
Lastly, decommissioning obsolete data or data storage devices require extra considerations to ensure that active data is properly retained, that inactive data is properly and safely discarded or archived and that regulatory compliance is maintained.
Health IT Workforce Curriculum Version 4.0

15. Creating Fault-Tolerant Systems, Backups, and Decommissioning Summary
Regulatory requirements for backups are stringent
An effective backup strategy minimizes the backup window while ensuring data integrity.
Backup considerations:
Onsite vs Off-site
Full vs Partial
Media
Verification
Decommissioning
Active data properly retained
Inactive data archived or disposed of securely
This concludes Creating Fault-Tolerant Systems, Backups, and Decommissioning.
Let’s take a quick moment to summarize the important points presented in this unit:
Regulations require healthcare institutions to keep exact backup copies of all healthcare data. That data should be protected, encrypted and stored in multiple locations to protect it from foreseeable harm for the duration of its retention period.
Backups often will occur during a limited timeframe, or backup window. As the amount of the data needing to be backed up increases, generally, so does the backup window. It is important to develop a backup strategy that minimizes the backup window while ensuring data integrity.
Consider on versus off-site and full versus partial backups, as well as the type of media to use. However, always remember that any backup without a restore is useless – verification of stored data is critical to ensure availability.
Lastly, decommissioning obsolete data or data storage devices require extra considerations to ensure that active data is properly retained, that inactive data is properly and safely discarded or archived and that regulatory compliance is maintained.
Health IT Workforce Curriculum Version 4.0

16. Creating Fault-Tolerant Systems, Backups, and Decommissioning References – Lecture c
Surviving a Bottleneck - Insights into Managing Exponential Growth of Digitized Medical Images. (2009, March 16). Retrieved from Scicasts website: asts.com/specialreports/86-healthcare-it/2442-surviving-a-bottleneck-insights-into-managing-exponential-growth-of-digitized-medical-images
Gordon, S. (n.d.). Comparing different backup strategies. Retrieved February 8, 2007, from SearchStorage website:
Retention and Destruction of Health Information. State Laws or Regulations Pertaining to Retention of Health Information. (updated 2013). Retrieved June 24, 2016, from AHIMA website:
No Audio. Ten seconds of silence.
Health IT Workforce Curriculum Version 4.0

17. Installation and Maintenance of Health IT Systems Creating Fault-Tolerant Systems, Backups, and Decommissioning Lecture c
This material was developed by Duke University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number IU24OC This material was updated in 2016 by The University of Texas Health Science Center at Houston under Award Number 90WT0006.
No Audio.
Health IT Workforce Curriculum Version 4.0