Supporting Multi-Fidelity Computations in Mobile Interactive Applications Dushyanth Narayanan

2 Mobile Interactive Applications Wearable computing = mobility + interactivity A mobile system has  limited battery energy  unpredictable, turbulent network environment  other resources are scarce An interactive system needs  low latency (good interactive response)  minimal user distraction Goal: build a system to support mobile interactive apps

3 Some examples Augmented reality  an architect wants to remodel a warehouse.  an aircraft technician wants to see hidden wiring.  a soldier needs to be aware of enemy positions. Search  find restaurants that I can get to within five minutes.  find images that look like this building’s façade.  plan an alternate driving route before I miss my exit. Speech recognition Image processing  surveillance: filter out uninteresting images, report interesting ones.

4 Why are interactive apps special? The challenge: provide low latency, adequate battery lifetimes in a turbulent, resource-poor environment. The opportunity: degrading the result is acceptable since users are tolerant sinks of output.  render 3-D objects at lower resolutions for augmented reality.  simplify search query, or search a subset of the entire space.  use a smaller language model for speech recognition.

5 Multi-fidelity computation Traditional algorithms have a fixed output specification. Resource consumption is the dependent variable. Multi-fidelity computations can have a range of outputs. Along with the output, the fidelity is indicated. The computation provides knobs to set the fidelity. Lets us bound resource usage by varying fidelity. Dependence of variables is reversed.

6 Thesis statement Multi-fidelity computation is a viable approach to supporting mobile interactive applications.  There are interesting multi-fidelity, mobile,interactive apps.  Multi-fidelity computation improves their usability. I plan to build multi-fidelity support in Odyssey.  Design a programming model for multi-fidelity apps.  Support the programming model in Odyssey.  Demonstrate a small number of proof-of-concept apps.  Evaluate the system.

7 Outline  Thesis statement  Fidelity and resources  Programming model  Research plan  Related work  Contributions

8 What is fidelity? Fidelity of a computational result: extent to which it matches a reference result. Multi-dimensional fidelity: A result might have multiple fidelity metrics. A multi-fidelity computation provides knobs to set the fidelity metrics. At each execution, the system has to set these knobs.

9 Example of fidelity metrics Radiosity colors a 3-D model prior to rendering (might use in an augmented reality app). Two fidelity metrics:  resolution (fraction of polygons retained) continuous, from 0 to 1  choice of algorithm (hierarchical/progressive) discrete, with two values At each execution, system picks the fidelity (e.g., resolution=0.4, method=progessive).

10 Radiosity application (unmodified)

11 Multi-fidelity radiosity app

12 Multi-fidelity radiosity app

13 Multi-fidelity radiosity app

14 Multi-fidelity radiosity app

15 Fidelity and resources Setting the knobs affects the fidelity of the result; it also affects the resource consumption. The resources of interest to the user are:  Latency (user wait time)  Battery energy  User attention  Money

16 What is a resource? “Resource” is an abstraction for anything required by a computation from its environment. Fidelity Latency User attention Battery energy Money CPU Memory Network Disk bw Cache state Resources Quality metrics User visible User invisible

17 Cache state as a resource Caching remote objects key to avoiding network access. Cache (or cache state) is a critical resource:  the computation makes demands on it.  the system provides it at varying levels. but not a resource in the traditional sense:  value is not necessarily a number (set of objects).  not shared or allocated in the usual way.  not the same resource as cache space. Example: user wants to search a set of documents. “Better” cache state means better latency (or fidelity).

18 User attention as a resource A wearable computer mediates user’s reality. Interactive app make demands on user’s attention. A multi-fidelity system has an impact on user attention:  Different fidelities demand different levels of attention.  System solicits user feedback to make good tradeoffs.

19 Resource usage dependencies Cache state Memory Network transmit receive CPU local remote Disk bw Fidelity Data size Money Latency Battery User attention CPU usage is a function of fidelity, input data size Battery energy and latency are functions of CPU, network, disk usage.

20 Outline  Thesis statement  Fidelity and resources  Programming model  Research plan  Related work  Contributions

21 Interactive computations Interactive computation: smallest unit of computation visible to the user. We care about multi-fidelity interactive computations. Each execution has a set of quality values, and a utility. For each execution of such a computation, we need to set the fidelity to maximize the utility.

22 Programming model The application programmer  identifies the multi-fidelity interactive computations.  inserts hooks for system to set the fidelity.  inserts hooks for system to monitor the execution. The run-time system:  selects the fidelity.  monitors each execution and its resource consumption.  uses the information to improve fidelity selection.

23 Research plan Build a system to support this scenario:  User asks radiosity app to color some scene.  Radiosity app queries system for fidelity.  System makes a fidelity selection.  monitors/predicts resources available to app.  predict’s app’s resource consumption (using dependency graph + resource availability).  maps user-visible resources + fidelity to user happiness (using some utility function).  searches over the fidelity space to maximize utility.

24 Mapping fidelity to utility Computation Fidelity 1 Fidelity 2 … Input data System state CPU availability Memory availability Network bandwidth Disk bandwidth Cache state $ cost of network Energy cost of CPU Energy cost of network Energy cost of disk User’s distraction level Quality Fidelity 1 Fidelity 2 …. Latency User attention Money Battery drain Utility Resource consumption CPU Memory Network Disk bw Cache state User attention Resource consumption functions Utility function

25 Research plan Develop an API  Extend API to support cache state, user attention Implement support in Odyssey  Predicting resource availability  Predicting resource consumption  Determining the utility function  Maximizing the utility function Demonstrate proof-of-concept applications  Radiosity application  Augmented or virtual reality  Search  Engineering computation and/or 3-D visualization Evaluate the system

26 Multi-fidelity API On initialization register_fidelity({f 1 :continuous[0..1], f 2 :discrete[hier,prog]}); hint_resource(CPU, (f 2 == hier) ? 1.4 nf 1 log(nf 1 ) : 2 n 2 f 1 2 ); hint_constraint(LATENCY, 0.05 seconds ); When the user requests a coloring operation begin_fidelity_op(1000 input data size in polygons ); returns {f1=0.4, f2=hier} values of fidelity metrics … run the radiosity computation … end_fidelity_op(); signals end of interactive computation

27 Cache state and user attention To use cache state in making fidelity decisions:  monitor cache state, staleness of objects.  extend API to allow disclosure of accesses.  predict #objects accessed, hit rate from past history. To measure user attention:  assign cost to each aspect of user interface (windows, menu bars, audio messages).  extend interface to allow app hints on user attention.  do cost/benefit when soliciting user feedback on utility.

28 Predicting resource consumption (1) Resource consumption is a function of  input data  fidelity How can we know the resource consumption function? Solution: a combination of  application programmer hints  learning from a history database We represent input data by a single feature (typically size).

29 Predicting resource consumption (2) Cache state Memory Network transmit receive CPU local remote Disk bw Fidelity Data size Money Latency Battery User attention Split complex function into multiple simpler functions. E.g. battery is an application-independent linear function of CPU, network, and disk usage.

30 Learning the utility function Two ways of learning utility or user happiness:  Implicit (outside the scope of this thesis)  Explicit  application programmer hints  feedback from user Hints will be in the form of constraints. programmer-specified bounds on important quality metric(s) Constraints are simplified utility functions. assume utility is a step-function on constrained quality metric

31 Using user feedback User feedback:  slider bars/buttons/preference files to set user constraints  query the user when we have a difficult tradeoff decision  allow user to interrupt incremental-fidelity operations More user feedback leads to better tradeoffs. This is not free: we are consuming user attention. There is a meta-tradeoff: between goodness of tradeoff and user attention.

32 Finding sweet spots Constraints are only an approximation: assumption that utility is a step-function. For non-real time apps, we can fudge the constraint. Example: a search app has a 1s latency constraint. In 1.2s we can search all documents in the cache. After that, we need to go over the network. There is a knee, or sweet spot at 1.2s. I plan to automatically identify sweet spots.

33 Evaluation Evaluating the idea:  How much can we impact resource consumption by adapting the fidelity?  Does this adaptation lead to better usability?  How easy/difficult is it to program to the multi-fidelity API? Evaluating the system:  How accurately do we predict resource consumption?  How well do fidelity selections maximize utility?  What is the overhead of computing fidelity selections?  How accurately do we find sweet spots?

34 Timeline

35 Outline  Thesis statement  Fidelity and resources  Programming model  Research plan  Related work  Contributions

36 Related work  Data fidelity  Odyssey [Noble97], Dynamic Distillation [Fox96]  Real-time QoS  Multi-resource QoS [Lee99]  Rialto [Jones95], Darwin [Chandra98].  Multi-fidelity computations in the real world  Approximation algorithms, probabilistic algorithms  Online aggregation [Hellerstein97]  Odyssey speech recognition  User interfaces  Coda user patience model [Mummert95]  Mixed-initiative user interfaces [Horvitz99]

37 Contributions Improving the usability of a mobile interactive system  Broad abstraction of multi-fidelity computation  Multi-fidelity API and system support  Support for new resources (user attention, cache state)  Applying learning to fidelity/resource tradeoff decisions  Demonstration of mobile interactive multi-fidelity applications