1. Usability and Human Factors
Ubiquitous Computing in Healthcare
Welcome to Usability and Human Factors, Ubiquitous Computing in Healthcare.
This material (Comp 15 Unit 9) was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1U24OC This material was updated by The University of Texas Health Center at Houston under Award Number 90WT0006.
This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. To view a copy of this license, visit
Health IT Workforce Curriculum Version 4.0

2. Ubiquitous Computing in Healthcare Lecture – Learning Objectives
Describe the history of ubiquitous computing
Discuss the basic principles of ubiquitous computing
Discuss applications of ubiquitous computing in healthcare
Describe technical challenges
After completing this lecture, students will be able to:
Describe the history of ubiquitous computing
Discuss the basic principles of ubiquitous computing
Discuss applications of ubiquitous computing in healthcare in both clinical settings and patient-centric technologies
Describe technical challenges posed by ubiquitous computing applications

3. History of Ubiquitous Computing
Three waves of computing
Mainframes (one main processor, many terminals)
Personal computing (one computer for one person)
Ubiquitous computing = many computers on and around each individual
Mobile computing (PDA, cell phone, etc.)
Pervasively embedded in the environment (smart environments)
Wearable computing
Computing technologies we use today have undergone a number of transformations over the years. The three main ones are usually described as three waves of computing. In the early years of computing technologies, all the processing was concentrated in one big mainframe, with many terminals attached to it. In this model, many users had to share one computer, and the design of applications was specifically tailored to this mode of interaction. In the early 1980s, this model was largely replaced by the personal computing paradigm, where each user would interact with their own computer. To a large degree this remains a dominant computing paradigm. However, in the early 1990s a new paradigm emerged that slowly became as widespread as personal computing. It did not quite replace it, but rather established its own space. In this paradigm, each individual was envisioned as interacting with many computers positioned on their body as well as in the environment around them.
Health IT Workforce Curriculum Version 4.0

4. Original Vision Xerox PARC, Mark Weiser Ubicomp project began in 1988
“the most profound technologies are those that disappear”
Paper
Writing
Vision for computing technologies that are so commonplace that they disappear from people’s conscious attention
This vision, became known as ubiquitous or sometimes pervasive computing, originated at Xerox Palo-Alto Research Center and is usually attributed to Mark Weiser, a senior research scientist at PARC. The Ubicomp project led by him began in 1988, and their most influential and highly cited article on it came out in 1991 in Scientific American. Weiser’s observation was that the most profound technologies are those that disappear. In the paper he discusses such technologies as paper or writing, which became so universally accepted that millions of individuals rely on them without ever questioning their existence. His vision for computing technology was that it will achieve similar level of penetration into the very fabric of human life and will become so commonplace that it will disappear from people’s conscious attention.
Health IT Workforce Curriculum Version 4.0

5. Original Vision (Cont’d – 1)
Computing by the inch (PDA, smart phone, mobile phone, voice recorder, etc.)
Image of iPhone touch screen first generation taken from Apple.com
To achieve this vision, Weiser proposed that computing power becomes available in different shapes, forms and ergonomic factors. For example, on the smaller scale he talked about computing by the inch – small personal devices that could either be used by themselves, or embedded into physical objects people use.
Nowadays, in realization of his vision, mobile phones have become the most popular computing devices and have the widest penetration across geographic, social, cultural, and economic boundaries. This computing can take a much smaller form of sensors, such as Radio Frequency Identification Device tags, that are becoming very wide-spread in retail as well as in healthcare.
Smart watch Image from Pixabay.com
RFID chip next to a grain of rice taken from Wikipedia

6. Original Vision (Cont’d – 2) Computing by the foot
Computing by the foot for a long time had a relatively stable form that most closely resembled traditional personal computing, namely a laptop. Laptops gave their users greater mobility while providing most of the same functionality they had with their desktop systems. Over the years laptops varied in style and form factors, including those with touch sensitive screens that could allow for more direct interaction, touchpads. Tablets such as the Apple iPad are pushing the boundaries of ubiquitous computing by the foot even further.
Dell, 2012
Apple, 2010

7. Original Vision (Cont’d – 3) Computing by the yard
The final form of ubiquitous computing, computing by the yard, involves large-scale displays that could be used by many individuals. They could be passive, as is the case with large public Liquid Crystal Display (LCD) displays, or they could be interactive, as is the case with electronic white boards. In recent years, many digital artists experimented with large public LCD displays making them more interactive and allowing people around those displays to control their state and cause changes in their appearance.
TelepresenceOptions.com, 2011

8. Version of Ubicomp Computing on the body (wearable computing)
Computing in the environment (ubicomp)
We already discussed that ubiquitous computing can take many different forms. One common approach is wearable computing, with computing systems positioned on (or within) human bodies, such as Google’s Glass, an optical head-mounted display shown in the top image.
An alternative approach is to distribute computing devices in the environment or even embed it in everyday objects. Both of these approaches have their strengths and weaknesses, and both present unique challenges on a technical level, and on the level of human-computer interaction. In this lecture we will primarily focus on ubiquitous rather than wearable computing.
(Ubicomp, 2008)
(Ubicomp, 2008)
(Google Glass, Wikipedia)

9. Challenges Natural input Context-aware computing
Automated capture and access
The vision of ubiquitous computing implies many computing devices distributed or embedded in the environment. This presents a number of challenges as well as new opportunities to assist individuals in their activities. The three we will discuss in this lecture include their reliance on natural input, their opportunity to enable context-aware computing and automated capture and access.
Health IT Workforce Curriculum Version 4.0

10. Natural Input
Computing distributed in environment requires naturalistic interaction patterns
Voice recognition
Handwriting
Gesture
Tangible computing (interact with computing as with physical environment)
Because ubiquitous computing is distributed in the environment, the more traditional interaction modalities may not be available. For example, it is hard to imagine using keyboards or mice with smart everyday objects. These devices require different interactions. At the same time, humans are used to interacting with our social and physical environments without relying on keyboards or mice either. There is an opportunity to adapt more natural input techniques to ubiquitous computing. Such methods as voice recognition, handwriting, and gestures have all been explored as potential methods for interaction between humans and ubiquitous computing. However, much remains to be done; accuracy and efficiency of these methods are still suboptimal. An interesting approach to this problem is suggested by tangible computing. In this paradigm, individuals interact with computing the way they do with physical environment, through touch and other forms of physical manipulation.
Health IT Workforce Curriculum Version 4.0

11. Context-Aware Computing
Delivery of information and services based on the automatically sensed context
Multiple definitions of context
Who, when, where, etc.
Success often depends on three factors
Accuracy of context information sensed by the system
Correctness of interpretation or knowing what action to take in a particular context
Consequence of performing this action
Because ubiquitous computing is continuously available in the environment around people, it presents a unique opportunity to deliver information and services opportunistically, based on the automatically sensed context. For example, context-aware computing could recommend cooking recipes based on what was purchased during a recent shopping trip, notify users of sales, when they are in a supermarket, warn them about a traffic jams down the road they are driving, or tell them about a delay in a bus schedule when they are standing at a bus station.
However, determining what context is appropriate is not trivial. There are different definitions for what constitutes context. Some of these definitions focus more on the identities of individuals and objects, others pay more attention to temporality of events, and yet others are more concerned with their spatial and geographic position and orientation. Each of these calls for a particular set of technologies capable of capturing this context. For example, RFID tags can assist with identification of objects, while Global Positioning Systems (GPS) capability in smart phones helps with geographic positioning. Many smartphone weather apps provide weather alerts based on a user’s present location, and the app Foursquare can take into account user preferences, recommendations from people the user trusts, and GPS location to make recommendations of places to go near the user. The success of context-aware computing often depends on three factors: the accuracy of context information sensed by the system, the correctness of interpretation or knowing what action to take in that particular context, and the consequence of performing the action.
Health IT Workforce Curriculum Version 4.0

12. Capture and Access
Using technology to capture the flow of activities in real time and provide access to the records on demand
Often used for capturing meetings, or brainstorming sessions
In healthcare: documenting patient-doctor encounters
The final aspect of ubiquitous computing we will discuss is a family of applications called capture and access. The basic goal of these applications is to use technology to capture the flow of activities in real time and provide access to the captured records on demand. Many of these applications focused on capture of meetings. They usually involved some form of video and audio capture, combined with access to digital artifacts reviewed during the meeting, such as documents and slides. One of the challenges of these applications is to devise an indexing approach so that the desired information can be retrieved efficiently.
Health IT Workforce Curriculum Version 4.0

13. Pervasive Healthcare
Application of ubiquitous computing technologies for healthcare
Making healthcare available everywhere, anytime, and to anyone
So far we have discussed some basic concepts related to ubiquitous computing.
In the second half of this lecture we will specifically look at applications of ubiquitous computing in healthcare. The term pervasive healthcare has two related but distinct meanings. On one hand, it is used to refer to the application of ubiquitous computing technologies for healthcare domain. On the other hand, and somewhat more generally, it is used to refer to the way of healthcare delivery that makes it available everywhere, anytime and to anyone. It is the hope of many researchers working in this area that introducing ubiquitous computing technologies can help to achieve the vision of universally accessible healthcare.
Health IT Workforce Curriculum Version 4.0

14. Pervasive Healthcare (Cont’d – 1)
Acute care settings – digital hospital
RFID for patient tracking
Context-aware clinical environments
Patient-centric technologies – health and wellness
Telemedicine (IDEATell)
Digital Family Portrait
MAHI
UbiFit Garden
We will focus on two broad classes of applications. One includes those that support acute care settings, such as hospitals. Specifically, we will look at some examples of projects that use RFID tags for tracking patients, and some attempts to introduce context-aware computing in clinical environments. The other family includes technologies that are designed to assist individuals with management of their health and wellness outside of the clinical setting. We will look at a number of applications in this family.
Health IT Workforce Curriculum Version 4.0

15. Improving Patient Safety
RFID patient tracking systems
Prevent errors (wrong medication to wrong patient)
Streamline billing (automatic tracking of procedures)
In recent years, patient safety became an issue that attracted much attention. There is a focused effort to reduce the number of preventable errors, such as delivering medication to a wrong patient, or for a wrong dose, or operating on the wrong patient. One solution to this problem is introduction of automated systems for patient tracking using, for example, RFID tags. These tags, when positioned on a patient, for example in their wrist band, can uniquely identify a patient. And when the unique identification is read by a portable computer, only the information pertaining to this patient can be made available. This can help to prevent some medical errors. This can also help to automatically track procedures, thus simplifying billing.
Real time location system (RTLS) technology can incorporate radio frequency to track patients and equipment in a hospital, and actively transmit messages in HL7 format.
RTLS has even been designed to improve handwashing by healthcare staff.
Siemens.com, 2010
Health IT Workforce Curriculum Version 4.0

16. Context-Aware Surgery Room
Main focus: improving patient safety
Providing the right information at the right time in the right place (pertinent patient data)
Drawing attention to information of concern (warnings of drug allergies, etc.)
utilizing surgical context (physical and clinical)
A similar concern is present in many diverse clinical environments. Here we see an example of a context-aware computing at work in the surgical ward. In this application, the schedule of procedures, the clinical personnel present in the room and the identity of a patient are all used to gather information pertaining to the case. Large screen displays enable hands-free lookup of data, as well as encourage communication among clinicians.
(Doryab, Afsaneh and Bardram, Jakob E., 2011)
Health IT Workforce Curriculum Version 4.0

17. Patient-Centric Technologies
Remote monitoring
Case manager
Education, recommendations, adjustments to care-plan
While ubiquitous computing is slowly becoming more common in acute care settings, its primary target remains health and wellness management outside of clinical environments. The applications, such as such as Care Innovations remote care management product, often focus on remote monitoring of individuals’ health, often with the assistance of a clinical case manager, who could use the gathered data to generate appropriate education, recommendations, and make necessary adjustments to care plan. In the examples on the screen, a patient at home is using a combination of devices to measure vital signs, such as blood pressure, blood glucose, among others. This data is transmitted to a nurse case –manager who can assess the data and recommend adjustments to the care.
(Mynatt, E.D., Rowan, J., Craighill, S., and Jacobs, A., 2001)
Image from CareInnovations.com
Health IT Workforce Curriculum Version 4.0

18. Health and Wellness Digital Family Portrait
Georgia Institute of Technology
Helping adult children maintain awareness of well-being of their parents
Activity of parents is sensed by motion detection sensors
Abstract visualization creates pattern without violating privacy
Other applications take a further step away from clinical settings and put majority of care in the hands of individuals and their family members. One of the early examples of such applications is Digital Family Portrait (DFP) developed by researchers at Georgia Institute of Technology. DFP is designed to help adult children maintain a level of awareness of well-being of their aging parents. It relies on a set of motion detection sensors positioned in places of common activity at the house of an aging parent. The readings of the sensors are aggregated and are condensed to one iconic presentation, a butterfly in the frame of a parent digital picture. The frame includes a number of these icons each representing a day in a life of the parent, the size of the butterfly show how much activity was sensed during the day. The very coarse presentation of data does not allow for any focused inquiry, however, it allows the child to notice changes in activity levels and draws their attention to sudden increases or decreases they might want to follow up on. This type of interaction also helps to avoid privacy concerns.
(Mynatt, E.D., Rowan, J., Craighill, S., and Jacobs, A., 2001)
Health IT Workforce Curriculum Version 4.0

19. Health and Wellness (Cont’d – 1)
MAHI (Georgia Institute of Technology)
Assistance with diabetes management
Mobile phone for capture of experiences (pictures of meals, voice records)
Integration with glucose monitor
Website for review with diabetes educator
MAHI, or Mobile Access to Health Information is another application from the Georgia Institute of Technology developed in collaboration with Siemens Corporation. MAHI is designed for individuals with diabetes and is meant to help them learn from their daily experiences. MAHI is a distributed application that includes several components: a mobile phone with a custom application allows individuals to take pictures, for example of their meals and to record voice notes with their concerns and questions. A custom-built Bluetooth attachment to a conventional glucose meter helps them to transfer all the readings from the meter onto their phone. These records combined are then sent to a remote server and posted on individuals’ websites where they can view them and discuss them with a diabetes educator. In deployment studies, MAHI was shown to help individuals to gain better appreciation of how their choices affect their heath.
(Mamykina, L., Mynatt, E.D., Davidson, P., and Greenblatt, D., 2008)
(Mamykina, L. and Mynatt, E.D., 2007)
Health IT Workforce Curriculum Version 4.0

20. Health and Wellness (Cont’d – 2)
UbiFit Garden
Intel
Monitoring physical activity
On the body monitoring
A variety of physical activity types
Ambient display on a mobile phone
Aesthetically pleasing visualization
Monitoring accomplishments
Reward for achievement of goals
The final applications we will review, a UbiFit garden developed by researchers at Intel, takes advantage of many of the same technologies we discussed before. It utilizes a relatively sophisticated sensing platform that allows individuals to capture various types of physical activity, such as swimming or biking. Individuals use a custom application on their mobile phones to set up their weekly activity goals. The application then automatically monitors their progress towards their goals and visualizes their activity levels using a flower garden as a metaphor. This display has several important properties: it is aesthetically pleasing, making it appropriate for a device that often becomes an expression of an individual’s identity; it allows them to easily monitor their accomplishments, without having to login to a website or calculate daily progress manually; finally, it rewards individuals for achieving their goals. These rewards are not big and certainly not monetary; individuals get butterflies in their garden. But they might just be enough to motivate somebody get off a couch and walk for 30 minutes to refresh their garden.
(Consolvo,C., McDonald, D.W., Toscos, T., Chen, M.Y., Froehlich, J., Harrison, B., et al. 2008)
Health IT Workforce Curriculum Version 4.0

21. Health and Wellness (Cont’d – 3)
FitBit
Monitors physical activity
On the body monitoring
Heart rate, sleep, activity, calories burned
Syncs data wirelessly to smart phone
Monitoring accomplishments
This is another device for tracking physical activity, the FitBit. It can track a variety of parameters and wirelessly transmit data to a smart phone.
(Consolvo,C., McDonald, D.W., Toscos, T., Chen, M.Y., Froehlich, J., Harrison, B., et al. 2008)
("Shop Fitbit Charge HR", n.d.)
Health IT Workforce Curriculum Version 4.0

22. Health and Wellness (Cont’d – 4)
Fall prevention and detection for elderly patients
Sensor-equipped floor mats can transmit data indicating falls
Data from smartphone combined with clinical data can help clinician assess and treat frail patients who are at risk of falls
(Fontecha, J. et al., 2013)
Elderly patients can be at high risk of morbidity from falls. Sensor-equipped floor mats can transmit data indicating a patient has fallen. Physical activity data from accelerometer-equipped smartphones, combined with clinical data, can help clinicians assess and treat frail patients who are at risk of falls.

23. Keeping Patients at Home
Patients want to be at home
Privacy issues
Non-intrusive technologies such as:
Pill box sensors
Kitchen motion sensors
Bathroom door sensors
 (Heather Kelly, 2014)
Finally, patients want to remain in their own homes as long as possible, and not be in a hospital or nursing home, and they also want to maintain their privacy. A variety of technologies and sensors can allow monitoring a person’s activity without video cameras. For example, sensors can be incorporated into pill boxes, the kitchen, and even the bathroom door.

24. Ubiquitous Computing in Healthcare Lecture – Summary
Ubiquitous computing is an important and rapidly growing area that has potential to make significant impact on how healthcare is delivered within and outside of clinical settings
However, many challenges remain to its successful penetration:
Many of these technologies rely on natural input such as voice or gesture, which is not always accurate and may be hindered by fragmented internet connectivity
As new enabling technologies become available, however, better applications of ubiquitous and pervasive computing will be invented
This concludes Usability and Human Factors, Ubiquitous Computing in Healthcare.
In conclusion, we hoped to demonstrate that ubiquitous computing is an important and also rapidly growing area that has a potential to make significant impact on how healthcare is delivered within and outside of clinical settings. However, many challenges remain to its successful penetration. For example, many of these technologies rely on natural input such as voice or gesture. Both of these modalities show continuing improvements in accuracy, however, it is still below optimal. They also often require constant connectivity and may not work well in an area with fragmented coverage. However, as new enabling technologies become available, so will inventive new applications of ubiquitous and pervasive computing.

25. Ubiquitous Computing in Healthcare References
Baslyman, M. (2014). Real-time, location-based hand hygiene monitoring and notification system, retrieved from
Fontecha, J., Navarro, F. J., Hervás, R., & Bravo, J. (2013). Elderly frailty detection by using accelerometer-enabled smartphones and clinical information records. Personal and ubiquitous computing, 17(6),
Heather Kelly, C. (2014). Sensors let Alzheimer's patients stay at home, safely. CNN. Retrieved 23 June 2016, from
Weiser, M. (1993). Some computer science issues in ubiquitous computing. Communication of The ACM, vol.36(7), p
Wallace, M. How RTLS can streamline physician and patient flow. Healthcare IT News. Retreived December 15, 2014 from
Images
Slide 5: Image of iPhone touch screen first generation. Retrieved on September 10th, 2010 from
Slide 5: Smart watch Image. Retrieved 23 June 2016, from
Slide 5: RFID chip next to a grain of rice. Retrieved 23 June 2016, from
Slide 6: Image of Dell computer. Retrieved on September 10th, 2012 from
Slide 6: iPad image. Retrieved on September 10th, 2010 from
Slide 7: Original vision image retrieved September 10th, 2010 from
No Audio.

26. Ubiquitous Computing in Healthcare References (Cont’d – 1)
Images
Slide 8: Version of Ubicomp image from Science Daily, April 28, 2008 (The Prototype Wearable Eye Tracker, image courtesy of ETH Zurich).
Slide 8: Google Glass image. Retrieved 23 June 2016, from
Slide 15: Improving Patient Safety. Retrieved March 22, 2010 from
Slide 16: Doryab, Afsaneh and Bardram, Jakob E. (2011). In Proceedings of the 2011 Workshop on Context-awareness in Retrieval and Recommendation, pages 43-46, New York, NY, USA, Designing activity-aware recommender systems for operating rooms.
Slide 17 & 18: Mynatt, E.D., Rowan, J., Craighill, S., and Jacobs, A. (2001). Digital family portraits: supporting peace of mind for extended family members. In Proceedings of the SIGCHI conference on Human factors in computing systems (CHI '01). ACM, New York, NY, USA,
Slide 17: Retrieved 23 June 2016, from
Slide 19: Mamykina, L., Mynatt, E.D., Davidson, P., and Greenblatt, D. (2008). MAHI: investigation of social scaffolding for reflective thinking in diabetes management. In the Proceedings of the 2008 ACM Conference on Human Factors in Computing (CHI 2008), p
Slide 19: Mamykina, L. and Mynatt, E.D. (2007). Investigating and supporting health management practices of individuals with diabetes. In Proceedings of the 1st ACM SIGMOBILE international workshop on Systems and networking support for healthcare and assisted living environments (HealthNet 2007)
Slide 20 and 21: Consolvo,C., McDonald, D.W., Toscos, T., Chen, M.Y., Froehlich, J., Harrison, B., et al. (2008). Activity sensing in the wild: a field trial of ubifit garden. In Proceedings of the twenty-sixth annual SIGCHI conference on Human factors in computing systems (CHI '08). ACM, New York, NY, USA,  
Slide 21: Shop Fitbit Charge HR. Fitbit.com. Retrieved 23 June 2016, from
No Audio.

27. Usability and Human Factors Ubiquitous Computing in Healthcare
This material was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1U24OC This material was updated by The University of Texas Health Center at Houston under Award Number 90WT0006.
No Audio.
Health IT Workforce Curriculum Version 4.0