1. UI Issues, Neural Nets, RTS
CSE 490RA
January 27, 2005

2. Lecture outline
Leftovers from UI Lecture
Neural Networks
RTS

3. Uses of the stylus Direct writing Abstract writing Pointing Selecting
Gesture
Direct manipulation
Control

4. Mode issues in pen computing
Adding modes to the pen
Barrel button
Secondary button with non-dominant hand
Eraser tip
Pressure
Explicit mode buttons
Area based modes (writing area, gesture area, control area, etc.)
Cursor feedback
Importance of visual cues for informing user
Errors in crossing mode boundaries

5. Control
Widgets design for stylus use

6. Flow Menu Use movement through octants for control information Move
Item
100%
100%
Text
Item
Highlight
Zoom
66.6%
200%
66.6%
200%
Shape
50%
75.0
400%
50%
100.0
400%
25%
800%
25%
800%
Custom
Custom

7. Interaction with direct manipulation
Move
Item
Highlight
Zoom

8. CrossY: Crossing based UI
Specify operations by drawing through

9. Gestures Commands issued with a single stroke
May be drawn or invisible
Support from SDK
Register gestures to be recognized
UI Issues
Similar to keyboard short cuts
Speed up for experts
Hard to learn / remember

10. Gestures Ambiguity Robustness Distinction between gestures
Distinction between gesture and other ink
Robustness
Handling misrecognized gestures
False positive
False negative
Gesture initiated actions should be undoable

11. Neural Networks
Fundamentals for Handwriting Reco Lecture (Jay Pittman)
Recognition algorithm
Learning based recognition algorithm

12. General considerations for learning algorithms
Training sets
Collection
Evaluation
Training cost
Time and space
Algorithm cost
Robustness to error

13. Neural networks Perceptrons
Motivated by considerations of the brain

14. Single layer neural networks
Bias weights
Threshold activation function
Step function
Sigmoid function: 1/(1 + e-x)

15. What you can do with single layer networks
Any linearly separable dataset can be recognized with a single layer neural network

16. Gradient descent algorithm
Choose initial weights
While not at optimum
Compute derivative
Move along derivative
It can be proved this converges

17. However, single layer networks are very limited

18. Multilayer networks with hidden nodes
Can recognized much wider range of data set
The gradient descent algorithm generalizes to this case

19. Real Time Stylus
Allow for user computation on the ink thread

20. Architecture (Overview)
RealTime Event Sink
RealTime Event Sink
RealTime Event Sink
RealTimeStylus
Ink Collecting
Object
queue
storage
Pen
Service
InkCollector
“Inking” Thread
UI Thread

21. Substroke operations Examples Rendering Custom Inking Multiple Ink
Distributed Ink
Rendering
Dynamic – draw on ink packet
Static – draw on paint event

22. Custom Inking Create plugin to listen for packets Registor for Packets
Draw triangle on each packet

23. public void Packets(RealTimeStylus sender, PacketsData data){
for (int i = 0; i < data.Count; i += data.PacketPropertyCount){
Point point = new Point(data[i], data[i+1]); // Packet data always has x, y
// followed by the rest
Point convertedPoint = new Point(); // We need to convert to Pixels...
convertedPoint.X = (int)Math.Round((float)point.X * (float)myGraphics.DpiX/2540.0F);
convertedPoint.Y = (int)Math.Round((float)point.Y * (float)myGraphics.DpiY/2540.0F);
if (this.pointCount == 0)
this.firstPoint = convertedPoint;
if (this.pointCount > 1){
Color color = pointCount % 2 == 0 ? Color.Red : Color.Yellow;
Point[] triangle = new Point[3];
triangle[0] = this.firstPoint;
triangle[1] = this.previousPoint;
triangle[2] = convertedPoint;
myGraphics.FillPolygon(new SolidBrush(color), triangle); }
this.pointCount++;
this.previousPoint = convertedPoint;

24. Distributed Ink Capture ink packets on machine 1
Send packets to machine 2
Reconstruct ink stroke by inserting packets

25. Sample application

26. Distributed Ink Sender: Receiver:
Collect packets in packet, when count is above threshold, send message
Receiver:
Insert packets as custom data
Render custom data as it is received
Assemble custom data as an ink stroke