1. CS 594: Empirical Methods in HCC Sensor Data Streams in HCI Research
Dr. Debaleena Chattopadhyay
Department of Computer Science
debaleena.com
hci.cs.uic.edu

2. Agenda
Sensor data streams
Eye tracking

3. Seismo

4. What are sensor data streams?
Continuously collect streams of low-level data about people and their environments.
E.g., locations, physiological states, contact with other people, situated uses of devices, and other digital traces can potentially be recorded and analyzed (quantitatively).
A large number of samples can be gathered quickly and with relatively low overhead.
Considerations:
Careful experimental design
Appropriate sensor design
Deployment
Participant training
data storage requirements

5. Brief history of sensor data streams
Experience sampling method (ESM), pioneered by Larson and Csikszentmihalyi (1983). Also called ecological momentary assessment or EMA.
Technology-oriented adaptations of ESM (circa 2000)
Still, ESM relies on active responses from research participants during data collection.
Sociological data collection technique

6. What questions can this method answer
Sensor data stream collection may be used to understand people’s activities, behaviors, or practices.
What can be instrumented?
People
Environment, such as home or office
Record actions and their context more accurately than self-reported responses.
Low-level data collected by various sensing devices can be used to determine higher-level behavior, such as activity, usage, or communication roles.

7. Various Units of Analysis
Egocentric Sensor Data Streams
focused on monitoring the movements, activities, and interactions of a single individual (similar to using log data)
Group-Centric Sensor Data Streams
capture data across a group
Space-Centric Sensor Data Streams
capture data in semipublic or public spaces
alternative: Infrastructure Mediated Sensing (IMS), using existing home infrastructure to detect activity within a home
IMS: using existing home infrastructure to detect activity within a home

8. Phoneprioception

9. Hydrosense

10. Examples of data that can be captured
Logs = “virtual sensor” data streams
Data streams might also come from log files or usage statistics from instrumented user interfaces
Physical sensors as a way of studying people and their behaviors
Sensing toolkits often facilitate data collection using a variety electrical switches, motion sensors, pressure sensors, voltmeters, photometers, thermometers, moisture sensors, proximity sensors, RFID tags and beacons, microphones, or cameras.
Time-stamped data flow

11. Vibrosight

12. Sensor Data Streams and Context-Aware Computing
Context-aware computing is a form of interactive computing where a user’s implicit behavior or the environment of the system can serve as alternative or auxiliary inputs to the system
E.g., people’s location, their physical activity, or their interactions with other people
Context- aware computing leverages streams of data as inputs
context- aware computing leverages streams of data as inputs

13. Collecting vs. leveraging sensor data streams
research goals (studying user behaviors versus developing interactive systems),
when the collected data are processed an analyzed (as part of the analysis vs. in real time), e.g., edge computing
whether the supporting technologies are primarily intended to generate a user model vs. predict user behavior based on a preexisting model.
In context-aware computing, data collection and analysis often
need to be tightly coupled so that the results of the data stream analysis are available as soon as possible following the physical interactions that generated them.

14. Creepy Tracker Toolkit for Context-aware Interfaces

15. Limitations of sensor data streams
Sensor data generally does a poor job of answering questions of why things have happened in the real world.
Senor readings provide little insight about the intention behind the actions or the broader aims, goals, or internal, mental states of a participant.
Possible remedy triangulation or data fusion
Phenomena to be measured or observed must be well understood
Sensors have limitations in the quality of data they can collect
Care must be taken to select sensors that can effectively capture the right kind of data at the right fidelity and minimize intrusiveness and discomfort
Sensors introduce a level of technical complexity and create large datasets quickly.

16. How to do sensor data streams research?
Generating the research questions and planning how to analyze the data streams
Building, acquiring, or provisioning the sensors
Determining how frequently and at what level of fidelity to collect data samples
Installing the sensors
Storing the data representation
Making sense of the collected corpus of data
User modeling and event detection
Validating user models

17. Take away Quantitative analysis
Large amount of data; little user burden
Data fidelity depends on sensor characteristics.
Low-level data need to be processed to high-level metrics for answering research questions.

18. Basic definitions for eye tracking
We perceive the environment all-around us effortlessly because of rapid eye movements. These sudden jumps in eye position that occur through fast eye movements are known as saccades.
Information processing is thought to occur during fixations, when the eye position is relatively static.

19. What is Eye Tracking?
Eye tracking is the process of measuring either the point of gaze (“where we are looking”) or the movement of the eye relative to the head.
Used as a research method since the 1800s; but the technology has evolved.
An eye tracker is a device for measuring eye positions and eye movement.
There are a variety of ways to do eye tracking, e.g., using surface electrodes, infrared corneal reflections, or video-based pupil monitoring

20. Tobii Eye Tracking

21. What Questions the Method Can Answer?
Eye fixation reflect the current focus of the user’s attention and the amount of cognitive processing on the fixated object(s).
Heatmaps visualize the distribution and intensity of user attention on the display
Other research explorations:
Vision Science (Neuroscience/Psychology)
Computer Vision: Perceptual Models of
Eye Gaze
Psychology: Reading Behavior
Market Research

22. Different uses of eye tracking in HCI
Understanding the perceptual aspects of user attention on displays (what do users notice)
Cognitive aspects of attention (what do users focus on, or spend time processing)
Social aspects of attention (e.g., mutual gaze in human–human interaction)
As an input method, using gaze as an alternative to the keyboard and mouse

23. Orbits

24. Gaze-Shifting

25. Aspects of a good eye tracking study
Hypothesis: Formulate clear null and alternate hypotheses, motivate them, and state the underlying assumptions.
Design: Determine whether it is an observational study or an experimental study.
Task description: What task is assigned to the participants? Are they freely viewing the displays, or are they performing a task such as searching for a particular object in the display.
Analysis: Eye tracking data can be analyzed in qualitative ways (e.g., heatmap visualization, observing where people look) and using rigorous quantitative methods.
Example metrics: number of eye fixations, duration of eye fixations, number of saccades, time to first fixation

26. Eye tracking as a research or for research

27. Happy Thanksgiving!