1. Input & Output CS 422: UI Design and Programming
HCI is the study of how people uses computer technologies and in turn how we can design user-friendly computer technologies. HCI drives innovation at the intersection of people and computers
Input & Output

2. Housekeeping
GR2 is due on 2/26. Rubric and examples are posted on Piazza.
You should already be working on PS2.
We will have our last lab next class on 2/26. That will conclude the UI prototyping module. Next up, visual design basics module.
GR1 grades will be released tomorrow, 2/22.

3. Lab or Lecture: What would you prefer?
I’d like a hands-on lab focusing on output
I’d like a lecture on graphic design and animation

4. Things we will review today
Raw vs. translated events
Event dispatch and propagation
State machines
Output representations
Animation principles and implementation (next class)

5. DOM is an example of ___________ in GUI implementation.
View Tree
Listener Pattern
Learnability
Efficiency

6. DOM is an example of ___________ in GUI implementation.
View Tree
Listener Pattern
Learnability
Efficiency

7. What style of GUI implementation is HTML?
Direct Manipulation
Declarative programming
Procedural programming
None of the above

8. What style of GUI implementation is HTML?
Direct Manipulation
Declarative programming
Procedural programming
None of the above

9. What style of GUI implementation is JavaScript?
Direct Manipulation
Declarative programming
Procedural programming
None of the above

10. What style of GUI implementation is JavaScript?
Direct Manipulation
Declarative programming
Procedural programming
None of the above

11. Raw vs. translated events
Two major categories of input events: raw and translated.
A raw event comes from a state transition in the input hardware.
E.g., mouse movements, mouse button down and up, and keyboard key down and up are the raw events seen in almost every capable GUI system.

12. A note on Keyboard events
In theory, the keydown and keyup events represent keys being pressed or released, while the keypress event represents a character being typed. The implementation of the theory is not same in all browsers.
Most programmers consider keypress as legacy now.
Note if a key is being pressed for a long enough time, it starts to repeat: the keydown triggers again and again, and then when it’s released we finally get keyup. For all repeating keys the event object has event.repeat property set to true.
Precedence: The *Down happens first, the *Press happens second (when text is entered), and the *Up happens last (when text input is complete).

13. Translated events Raw events are translated into higher-level events
For many GUI components, the raw events are too low-level, and must be translated into higher-level events.
E.g., a mouse button press and release is translated into a mouse click event—assuming the mouse didn’t move much between press and release - if it did, these events would be interpreted as a drag rather than a click, so a click event isn’t produced.

14. Translated events
Key down and up events are translated into character-typed events, which take modifiers (Shift/Ctrl/Alt)
If you hold a key down, multiple character-typed events may be generated by an auto repeat mechanism (usually built into the operating system or GUI toolkit).

15. State Machines Translate Events

16. Properties of an Input Event
Mouse position (X,Y)
Mouse button state
Modifier key state (Ctrl, Shift, Alt, Meta)
Timestamp
Keyboard key, character, or mouse button that changed
jQuery event, which overloads this for mouse events and key events
In jQuery, do not treat keydown/keyup and keypress as interchangeable; their names may be similar, but the parameters of the events are different.

17. Which of the following will never be the output?
keydown
keyup
keypres
None of the above

18. Which of the following will never be the output?
keydown
keyup
keypres
None of the above
Note that keydown and keyup provide a code indicating which key is pressed, while keypress indicates which character was entered. For example, a lowercase "a" will be reported as 65 by keydown and keyup, but as 97 by keypress. An uppercase "A" is reported as 65 by all events. Because of this distinction, when catching special keystrokes such as arrow keys, .keydown() or .keyup() is a better choice.

19. Event dispatch and propagation
Events are stored in a queue
User input tends to be in burss
Queue saves application from hard real time constraints (i.e., having to finish handling each event before next one might occur)
Edge events (button down and up events) are all kept in the queue unchanged. But multiple events that describe a continuing state - in particular, mouse movements - may be coalesced into a single event with the latest known state.
Is mouse coalescing always good?

20. Where in the following instances coalescing mouse movements is a good idea?
You’re sketching a freehand stroke with the mouse.
You’re dragging a big object across the screen, and the application can’t repaint the object fast enough to keep up with your mouse.
Both of the above
None of the above

21. Where in the following instances coalescing mouse movements is a good idea?
You’re sketching a freehand stroke with the mouse.
You’re dragging a big object across the screen, and the application can’t repaint the object fast enough to keep up with your mouse.
Both of the above
None of the above

22. Where in the following instances coalescing mouse movements is a good idea?
If you’re dragging a big object across the screen, and the application can’t repaint the object fast enough to keep up with your mouse, you don’t want the mouse movements to accumulate in the queue, because then the object will lag behind the mouse pointer, diligently (and foolishly) following the same path your mouse did. What interaction style is violated then?
If you’re sketching a freehand stroke with the mouse, and some of the mouse movements are coalesced, then the stroke may have straight segments at places where there should be a smooth curve.

23. Dispatch, propagation, & consumption
Dispatch: choose target component for event
Key event: component with keyboard focus
Mouse event: component under mouse (hit testing)
Mouse capture: any component can grab mouse temporarily so that it receives all mouse events (e.g. for drag & drop)
Propagation: event bubbles up hierarchy
If target component doesn’t handle event, the event passes up to its parent, and so on up the tree
Consumption: event stops propagating
May be automatic (because some component finally handles it) or manual (keeps going unless explicitly stopped)

24. What concept did we learn last class most related to event dispatch and propagation?
Mouse input events
Drag and Drop
Bubbling and Capturing
Keyboard input events

25. What concept did we learn last class most related to event dispatch and propagation?
Mouse input events
Drag and Drop
Bubbling and Capturing
Keyboard input events

26. Multitouch Dispatch (iPhone)
Multitouch input events have more than one (x, y) point (fingers on screen)
Touch-down event dispatches to the component containing it (which also captures future touch-moves and touch-up for that finger)
Touch events carry information about all fingers currently touching
A component can turn on “exclusive touch” to receive all touch-down events even if they fall outside it

27. Palm Rejection in Multitouch Interaction

28. State machines
A controller in a direct manipulation interface is a state machine.
Here’s an example of the state machine for a push button’s controller.
Transitions between states occur when a certain input event arrives, or sometimes when a timer times out.
Each state may need different feedback displayed by the view.
Changes to the model or the view occur on transitions, not states: e.g., a push button is actually invoked by the release of the mouse button.

29. State machines
A controller in a direct manipulation interface is a state machine.
Here’s an example of the state machine for a push button’s controller.
Transitions between states occur when a certain input event arrives, or sometimes when a timer times out.
Each state may need different feedback displayed by the view.
Changes to the model or the view occur on transitions, not states: e.g., a push button is actually invoked by the release of the mouse button.

30. Drag and Drop

31. Which state/transitions are not being captured by the model here?

32. test is _________ element
in-built element like <p></p>
class ID
the current document

33. Three Output representations
Objects
Graphical objects arranged in a tree with automatic redraw
Example: Label object, Line object
Also called: views, interactors, widgets, controls, elements
Strokes
High-level drawing primitives: lines, shapes, curves, text
Example: drawText() method, drawLine() method
Also called: vector graphics, structured graphics
Pixels
2D array of pixels
Also called: raster, image, bitmap

34. Output representations
All three output representations appear in virtually every modern GUI application.
The object representation always appears at the very top level, for windows, and often for graphical objects within the windows as well.
At some point, we reach the leaves of the view hierarchy, and the leaf views draw themselves with stroke calls.
A graphics package then converts those strokes into pixels displayed on the screen.
For performance reasons, an object may short-circuit the stroke package and draw pixels on the screen directly. On Windows, for example, video players do this using the DirectX interface to have direct control over a particular screen rectangle.

35. Output representations
At the highest level, a typical program presents itself in a window, which is an object.
At the lowest level, the window appears on the screen as a rectangle of pixels.
So a series of steps has to occur that translates that window object (and all its descendants in the view tree) into pixels.
The step from objects down to strokes is usually called drawing.
The step from strokes down to pixels is called rasterization (or scan conversion).

36. An example Object representation Stroke representation
Pixel representation
Since virtually every GUI uses all three representations, the design question becomes: at which points in your application do you want to step down into a lower-level kind of output?

37. Pure object representation
Each node and edge is an object in the view tree
A node object might have two child objects: circle and text label
Eventually, you’ll get down to the primitive objects available in your GUI toolkit.
Most GUI toolkits provide a text label;
most don’t provide a primitive circle. (One notable exception is SVG, which has object equivalents for all the common drawing primitives.)

38. Pure stroke representation
Alternatively, the top-level window might have no child objects.
Instead, it would draw the entire graph by a sequence of stroke calls:
drawCircle for the node outlines,
drawText for the labels,
drawLine for the edges.

39. Pure pixel representation
Your graph view might bypass stroke drawing and set pixels in the window directly.
The text labels might be assembled by copying character images to the screen.
This is rarely used nowadays, because it’s the most work for the programmer, but it used to be the only way to program graphics.

40. Hybrid representations
Hybrid representations for the graph view are certainly possible, in which some parts of the output use one representation, and others use another.
The graph view might use objects for nodes, but draw the edges itself as strokes.
It might draw all the lines itself, but use label objects for the text.

41. Next class
Lab 2