1. 1 User Interaction Includes material by Ed Angel Jeff Parker © 2013

2. 2 Objectives Review pipeline architecture Software components for an interactive graphics system Animation Events Handling events Buttons, menu, mouse clicks Transformations

3. 3 Pipeline Process objects as they are generated by the application Can consider only local lighting Pipeline architecture All steps can be implemented in hardware on GPU application program display

4. 4 Vertex Processing Much of the work in the pipeline is in converting object representations from one coordinate system to another Object coordinates Camera (eye) coordinates Screen coordinates Every change of coordinates is equivalent to a matrix transformation (next week) Vertex processor also computes vertex colors application program display

5. 5 Clipping Just as a real camera cannot “see” the whole world, the virtual camera can only see part of the world or object space Objects that are not within this volume are said to be clipped out of the scene

6. 6 Rasterization If object remains after clipping, appropriate pixels in frame buffer must be colored Rasterizer produces a set of fragments for each object Fragments are “potential pixels” - Each has Location in frame buffer Color and depth attributes Vertex attributes are interpolated over objects by the rasterizer application program display

7. 7 Fragment Processing Each fragment is processed to determine color of pixel in frame buffer Colors can be define by texture mapping or interpolation of vertex colors Fragments may be blocked by other fragments closer to the camera Hidden-surface removal application program display

8. 8 User Interaction Provided via callbacks Mouse Keyboard Reshape Learn to build more sophisticated interactive programs Select objects from the display Interactive drawing of rectangles and polygons

9. 9 Rotating Circle

10. 10 Rotating Circle

11. 11 rotatingSquare1.html attribute vec4 vPosition; uniform float theta; void main() { float s = sin( theta ); float c = cos( theta ); gl_Position.x = -s * vPosition.x + c * vPosition.y; gl_Position.y = s * vPosition.y + c * vPosition.x; gl_Position.z = 0.0; gl_Position.w = 1.0; }

12. 12 rotatingSquare1.html... void main() { float s = sin( theta ); float c = cos( theta ); gl_Position.x = -s * vPosition.x + c * vPosition.y; gl_Position.y = s * vPosition.y + c * vPosition.x; gl_Position.z = 0.0; gl_Position.w = 1.0; } var vertices = [ vec2( 0, 1 ), vec2( 1, 0 ), vec2( -1, 0 ), vec2( 0, -1 ) ];

13. 13 Animation We can trick the brain into seeing movement Change the image about 30 times a second We could set a timer for 1/30 of second, and update our image Today we call an API window.requestAnimFrame(render) This allows the browser to pick a good time to redraw Allows browser to call all updates at the same time Allows browser to ignore a page that is hidden

14. 14 Passing Theta to GPU var theta = 0.0; var thetaLoc; window.onload = function init() {.... thetaLoc = gl.getUniformLocation( program, "theta" ); render(); }; function render() { gl.clear( gl.COLOR_BUFFER_BIT ); theta += 0.1;

15. 15 rotatingSquare1.js var theta = 0.0; var thetaLoc; window.onload = function init() {... thetaLoc = gl.getUniformLocation( program, "theta" );... }; function render() { gl.clear( gl.COLOR_BUFFER_BIT ); theta += 0.1; gl.uniform1f( thetaLoc, theta ); gl.drawArrays( gl.TRIANGLE_STRIP, 0, 4 ); window.requestAnimFrame(render); }

16. 16 rotatingSquare1.js function render() // Rotating Square 1 { gl.clear( gl.COLOR_BUFFER_BIT ); theta += 0.1; gl.uniform1f( thetaLoc, theta ); gl.drawArrays( gl.TRIANGLE_STRIP, 0, 4 ); window.requestAnimFrame(render); } function render() // Rotating Square 2 { gl.clear( gl.COLOR_BUFFER_BIT ); theta += (direction ? 0.1 : -0.1); gl.uniform1f(thetaLoc, theta); gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4); setTimeout( function (){requestAnimFrame(render);}, delay ); }

17. 17 Rotating Square v2 Change Rotation Direction Toggle Rotation Direction Spin Faster Spin Slower Oops... your browser doesn't support HTML5 canvas element

18. 18 rotatingSquare2.html Change Rotation Direction... // In the JavaScript File //... // Initialize event handlers document.getElementById("Direction").onclick = function () { direction = !direction; };

19. 19 rotatingSquare2.js // Initialize event handlers document.getElementById("Direction").onclick = function () { direction = !direction; }; document.getElementById("Controls" ).onclick = function(event) { switch( event.srcElement.index ) { case 0: direction = !direction; break; case 1: delay /= 2.0; break; case 2: delay *= 2.0; break; } };

20. 20 rotatingSquare2.js window.onkeydown = function(event) { var key = String.fromCharCode(event.keyCode); switch(key) { case '1': direction = !direction; break; case '2': delay /= 2.0; break; case '3': delay *= 2.0; break; } };

21. 21 rotatingSquare2.js function render() { gl.clear( gl.COLOR_BUFFER_BIT ); theta += (direction ? 0.1 : -0.1); gl.uniform1f(thetaLoc, theta); gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4); setTimeout( function (){requestAnimFrame(render);}, delay ); }

22. 22 cad1.js Pick a color or use default Pick two corners in grey area Need two types of interaction Selection color Picking corner

23. 23 cad1.js We can register for callbacks All mouse moves Mouse down Mouse up

24. 24 Mouse Event attributes developer.mozilla.org/en-US/docs/Web/API/MouseEvent screenX long The X in global (screen) coords. screenY long The Y in global (screen) coords. clientX long The X in local (DOM) coords clientY long The Y in local (DOM) coords ctrlKey boolean true if the control key was down shiftKey boolean true if the shift key was down altKey boolean true if the alt key was down metaKey boolean true if the meta key was button unsigned short The button number pressed when the mouse event was fired. Left button=0, middle button=1 right button=2....

25. 25 cad1.js Register Listeners m.addEventListener("click", function() { cIndex = m.selectedIndex; }); canvas.addEventListener("mousedown", function(){ gl.bindBuffer( gl.ARRAY_BUFFER, vBuffer); if(first) { first = false; gl.bindBuffer( gl.ARRAY_BUFFER, vBuffer) t1 = vec2(2*event.clientX/canvas.width-1, 2*(canvas.height-event.clientY)/canvas.height- 1); } else {

26. 26 Mapping x = 2*event.clientX/canvas.width-1 What is going on? Event has (x, y) values Range of output X is clipping window [-1, 1] Range of clientX is [0, canvas.width) 2*clientX / canvas.width : [0, 2] Subtract one to get range: [-1, 1]

27. 27 Positioning What about y? The position in the screen window is usually measured in pixels with the origin at the top-left corner Historical consequence of refresh done from top to bottom OpenGL uses a world coordinate system with origin at the bottom left Must invert y coordinate by height of windowL y = h – y; t1 = vec2(2*event.clientX/canvas.width-1, 2*(canvas.height-event.clientY)/canvas.height-1); (0,0) h w

28. 28 Details for rectangle canvas.addEventListener("mousedown", function(){ gl.bindBuffer( gl.ARRAY_BUFFER, vBuffer); if(first) { first = false; gl.bindBuffer( gl.ARRAY_BUFFER, vBuffer) t1 = vec2(2*event.clientX/canvas.width-1, 2*(canvas.height-event.clientY)/canvas.height- 1); } else { first = true; t2 = vec2(2*event.clientX/canvas.width-1, 2*(canvas.height-event.clientY)/canvas.height- 1); t3 = vec2(t1[0], t2[1]); t4 = vec2(t2[0], t1[1]); t1 t2 t3 t4

29. Prototypes canvas.addEventListener("mousedown", function() {... gl.bufferSubData(gl.ARRAY_BUFFER, 8*index, flatten(t1)); gl.bufferSubData(gl.ARRAY_BUFFER, 8*(index+1), flatten(t3)); gl.bufferSubData(gl.ARRAY_BUFFER, 8*(index+2), flatten(t2)); gl.bufferSubData(gl.ARRAY_BUFFER, 8*(index+3), flatten(t4)); gl.bindBuffer( gl.ARRAY_BUFFER, cBuffer); index += 4; t = vec4(colors[cIndex]); gl.bufferSubData(gl.ARRAY_BUFFER, 16*(index-4), flatten(t)); gl.bufferSubData(gl.ARRAY_BUFFER, 16*(index-3), flatten(t)); gl.bufferSubData(gl.ARRAY_BUFFER, 16*(index-2), flatten(t)); gl.bufferSubData(gl.ARRAY_BUFFER, 16*(index-1), flatten(t)); } } );

30. 30 Prototypes canvas.addEventListener("mousedown", function() {... gl.bufferSubData(gl.ARRAY_BUFFER, 8*index, flatten(t1)); gl.bufferSubData(gl.ARRAY_BUFFER, 8*(index+1), flatten(t3)); gl.bufferSubData(gl.ARRAY_BUFFER, 8*(index+2), flatten(t2)); gl.bufferSubData(gl.ARRAY_BUFFER, 8*(index+3), flatten(t4)); gl.bindBuffer( gl.ARRAY_BUFFER, cBuffer); index += 4; [..x1, y1, x3, y3, x2, y2, x4, y4, … ] t1 t2 t3 t4 gl.vertexAttribPointer(vPosition, 2, gl.FLOAT, false, 0, 0); gl.drawArrays(gl.TRIANGLE_FAN, i, 4);

31. 31 Prototypes canvas.addEventListener("mousedown", function() {... gl.bufferSubData(gl.ARRAY_BUFFER, 8*index, flatten(t1)); gl.bufferSubData(gl.ARRAY_BUFFER, 8*(index+1), flatten(t3)); gl.bufferSubData(gl.ARRAY_BUFFER, 8*(index+2), flatten(t2)); gl.bufferSubData(gl.ARRAY_BUFFER, 8*(index+3), flatten(t4)); gl.bindBuffer( gl.ARRAY_BUFFER, cBuffer); index += 4; t = vec4(colors[cIndex]); gl.bufferSubData(gl.ARRAY_BUFFER, 16*(index-4), flatten(t)); gl.bufferSubData(gl.ARRAY_BUFFER, 16*(index-3), flatten(t)); gl.bufferSubData(gl.ARRAY_BUFFER, 16*(index-2), flatten(t)); gl.bufferSubData(gl.ARRAY_BUFFER, 16*(index-1), flatten(t)); } } ); [..x1, y1, x3, y3, x2, y2, x4, y4, … ] t1 t2 t3 t4 [ … r, g, b, a, r, g, b, a, r, g, b, a, r, g, b, a, …] gl.vertexAttribPointer(vColor, 4, gl.FLOAT, false, 0, 0);

32. Rendering function render() { gl.clear( gl.COLOR_BUFFER_BIT ); for(var i = 0; i<index; i+=4) gl.drawArrays( gl.TRIANGLE_FAN, i, 4 ); window.requestAnimFrame(render); }

33. Passing Data to GPU canvas.addEventListener("mousedown", function(){ gl.bindBuffer( gl.ARRAY_BUFFER, vBuffer); if(first) { first = false; gl.bindBuffer( gl.ARRAY_BUFFER, vBuffer) t1 = vec2(2*event.clientX/canvas.width-1, 2*(canvas.height-event.clientY)/canvas.height-1); } else { first = true; t2 = vec2(2*event.clientX/canvas.width-1, 2*(canvas.height-event.clientY)/canvas.height-1); t3 = vec2(t1[0], t2[1]); t4 = vec2(t2[0], t1[1]); gl.bufferSubData(gl.ARRAY_BUFFER, 8*index, flatten(t1)); gl.bufferSubData(gl.ARRAY_BUFFER, 8*(index+1), flatten(t3)); gl.bufferSubData(gl.ARRAY_BUFFER, 8*(index+2), flatten(t2)); gl.bufferSubData(gl.ARRAY_BUFFER, 8*(index+3), flatten(t4)); gl.bindBuffer( gl.ARRAY_BUFFER, cBuffer); index += 4; t = vec4(colors[cIndex]); gl.bufferSubData(gl.ARRAY_BUFFER, 16*(index-4), flatten(t)); gl.bufferSubData(gl.ARRAY_BUFFER, 16*(index-3), flatten(t)); gl.bufferSubData(gl.ARRAY_BUFFER, 16*(index-2), flatten(t)); gl.bufferSubData(gl.ARRAY_BUFFER, 16*(index-1), flatten(t)); } } ); render(); }

34. Passing Data to GPU canvas.addEventListener("mousedown", function(){ gl.bindBuffer( gl.ARRAY_BUFFER, vBuffer); if(first) { first = false; gl.bindBuffer( gl.ARRAY_BUFFER, vBuffer) t1 = vec2(2*event.clientX/canvas.width-1, 2*(canvas.height-event.clientY)/canvas.height-1); } else { first = true; t2 = vec2(2*event.clientX/canvas.width-1, 2*(canvas.height-event.clientY)/canvas.height-1); t3 = vec2(t1[0], t2[1]); t4 = vec2(t2[0], t1[1]); gl.bufferSubData(gl.ARRAY_BUFFER, 8*index, flatten(t1)); gl.bufferSubData(gl.ARRAY_BUFFER, 8*(index+1), flatten(t3)); gl.bufferSubData(gl.ARRAY_BUFFER, 8*(index+2), flatten(t2)); gl.bufferSubData(gl.ARRAY_BUFFER, 8*(index+3), flatten(t4)); gl.bindBuffer( gl.ARRAY_BUFFER, cBuffer); index += 4; t = vec4(colors[cIndex]); gl.bufferSubData(gl.ARRAY_BUFFER, 16*(index-4), flatten(t)); gl.bufferSubData(gl.ARRAY_BUFFER, 16*(index-3), flatten(t)); gl.bufferSubData(gl.ARRAY_BUFFER, 16*(index-2), flatten(t)); gl.bufferSubData(gl.ARRAY_BUFFER, 16*(index-1), flatten(t)); } } ); render(); } Mouse event does not call render Thus we are rendering on the animation cycle Wasted effort

35. 35 Reshaping the window We can reshape and resize the OpenGL display window by pulling the corner of the window What happens to the display? Must redraw from application Some applications (square.c) do not keep state: start over Two possibilities for reshape Display part of world Display whole world but force to fit in new window Can alter aspect ratio

36. 36 Reshape possiblities original reshaped

37. 37 Equation of Line Given two points, we can define a line using the two-point form Consider the example given the points (1, 2) and (4, 3) Note the relationship to the normal vector, (-1, 3)

38. 38 Other points Given pair (x, y), we can see if it is on line by plugging into the equation (7, 4) is on the line, as: 3 x 4 – 7 – 5 = 0 (1, 1) is not, as: 3 x 1 – 1 – 5 = -1 (2, 4) is not, as: 3 x 4 – 2 – 5 = 5 We have additional information: Sign tells us which side of line we are on

39. 39 Computation In fact, you don't need to create the equation: you can simply take the dot product of the point of interest with the normal vector. The endpoints of the line were (1,2) and (4,3) The dot products with the normal vector, n = (-1, 3), and the endpoints (1, 2) (-1, 3) = -1 + 6 = 5 (4, 3) (-1, 3) = -4 + 9 = 5 We return to the examples from the last slide (7, 4) is on the line, as: (7, 4) (-1, 3) = -7 + 12 = 5 (1, 1) is not, as: (1, 1)(-1, 3) = -1 + 3 = 2 (2, 4) is not, as: (2, 4)(-1, 3) = (-2, + 12 = 10 We can tell which side they are on by comparing to the "right" value: 5 But why is this valid? These are not vectors, they are points!?!

40. 40 What is going on? In problem 2, we needed the distance from a point to a line. What we need now is simpler: Which side of the line is the point? The construction is similar. Again the solution is to compute the dot product of the vector from origin to the point with the normal vector. In this case, we only need to know relative distance We are computing the relative distance from the line 3y – x = 0 Points on our line lie 5 units above the line through the origin If we needed the true distance, we would take the absolute value and divide by the length of the normal vector. In illustration: the point (3, 2) is below our line (-1, 3)(3, 2) = -3 + 6 = 3 < 5

41. 41 Rubberband initial display draw line with mouse in XOR mode mouse moved to new position first point second point Draw linesCould erase lines: hides background

42. 42 Writing Modes frame buffer application ‘ bitwise logical operation

43. 43 XOR write Usual (default) mode: source replaces destination (d’ = s) Cannot write temporary lines this way because we cannot recover what was “under” the line in a fast simple way Exclusive OR mode (XOR) (d’ = d  s) x  y  x =y Hence, if we use XOR mode to write a line, we can draw it a second time and line is erased!

44. 44 Rubberbanding Switch to XOR write mode Draw object For line can use first mouse click to fix one endpoint and then use motion callback to continuously update the second endpoint Each time mouse is moved, redraw line which erases it and then draw line from fixed first position to to new second position At end, switch back to normal drawing mode and draw line Works for other objects: rectangles, circles

45. 45 Rubberband Lines initial display draw line with mouse in XOR mode mouse moved to new position first point second point original line redrawn with XOR new line drawn with XOR

46. 46 Objectives Introduce standard transformations Rotation Translation Scaling Shear Learn to build arbitrary transformation matrices from simple transformations Look at some 2 dimensional examples, with an excursion to 3D We start with a simple example to motivate this

47. 47 General Transformations Transformation maps points to other points and/or vectors to other vectors Q=T(P) v=T(u)

48. 48 Non-Rigid Transformations

49. 49 Ocean Sunfish The Ocean Sunfish is the world's heaviest known bony fish Adapted from Thomas D'Arcy's On Growth and Form

50. 50 How many ways? Although we can move a point to a new location in infinite ways, when we move many points there is usually only one way objecttranslation: every point displaced by same vector

51. 51 Pipeline Implementation transformationrasterizer u v u v T T(u) T(v) T(u) T(v) vertices pixels frame buffer

52. 52 Affine Transformations We want our transformations to be Line Preserving Characteristic of many physically important transformations Rigid body transformations: rotation, translation Scaling, shear Importance in graphics is that we need only transform endpoints of line segments Let implementation draw line segment between the transformed endpoints

53. 53 Translation Move (translate, displace) a point to a new location Displacement determined by a vector d P’=P+d P P’ d

54. 54 Not Commutative While often A x B = B x A, transformations are not usually commutative If I take a step left, and then turn left, not the same as Turn left, take a step left This is fundamental, and cannot be patched or fixed.

55. 55 Rotations in 2D We wish to take triplets (x, y) and map them to new points (x', y') While we will want to introduce operations that change scale, we will start with rigid body translations Translation (x, y)  (x + deltaX, y) Translation (x, y)  (x + deltaX, y + deltaY) Rotation (x, y)  ? Insight: fix origin, and track (1, 0) and (0, 1) as we rotate through angle a

56. 56 Rotations Any point (x, y) can be expressed in terms of (1, 0), (0, 1) e.g. (12, 15) = 12*(1,0) + 15*(0, 1) These unit vectors form a basis The coordinates of the rotation of T(1, 0) = (cos(a), sin(a)) The coordinates of the rotation of T(0, 1) = (-sin(a), cos(a)) The coordinates of T (x, y) = (x cos(a) + y sin(a), -x sin(a) + y cos(a)) Each term of the result is a dot product (x, y) ( cos(a), sin(a)) = (x cos(a) + y sin(a)) (x, y) (-sin(a), cos(a)) = (x sin(a) - y cos(a))

57. 57 Matrices Matrices provide a compact representation for rotations, and many other transformation T (x, y) = (x cos(a) - y sin(a), x sin(a) + y cos(a)) To multiply matrices, multiply the rows of first by the columns of second

58. 58 Determinant If the length of each column is 1, the matrix preserves the length of vectors (1, 0) and (0, 1) We also will look at the Determinant. Determinant of a rotation is 1.

59. 59 3D Matrices Can act on 3 space T (x, y, z) = (x cos(a) + y sin(a), -x sin(a) + y cos(a), z) This is called a "Rotation about the z axis" – z values are unchanged

60. 60 3D Matrices Can rotate about other axes Can also rotate about other lines through the origin… Can perform rotations in order Any rotation is the produce of three of these rotations Euler Angles Not unique Euler Angles Wikipedia

61. 61 Euler Angles The Euler Angles for this rotation are not unique Euler Angles Wikipedia

62. 62 Scaling S = S(s x, s y, s z ) = x’=s x x y’=s y x z’=s z x p’=Sp Expand or contract along each axis (fixed point of origin)

63. 63 Reflection Reflection corresponds to negative scale factors Example below sends (x, y, z)  (-x, y, z) Note that the product of two reflections is a rotation original s x = -1 s y = 1 s x = -1 s y = -1 s x = 1 s y = -1

64. 64 Limitations We cannot define a translation in 2D space with a 2x2 matrix There are no choices for a, b, c, and d that will move the origin, (0, 0), to some other point, such as (5, 3) in the equation above Further, we will see that perspective can not be handled by a matrix operation alone We will find ways to get around each of these problems

65. 65 Order of Transformations Note that matrix on the right is the first applied to the point p Mathematically, the following are equivalent p’ = ABCp = A(B(Cp)) We use column matrices to represent points. In terms of row matrices p ’T = p T C T B T A T That is, the "last" transformation is applied first. We will see that the implicit transformations have the same order property

66. 66 Rotation About a Fixed Point other than the Origin Move fixed point to origin Rotate Move fixed point back M = T(p f ) R(  ) T(-p f )

67. 67 Instancing In modeling, we often start with a simple object centered at the origin, oriented with the axis, and at a standard size We apply an instance transformation to its vertices to Scale Orient (rotate) Locate (translate)

68. Example Be sure to play with Nate Robin's Transformation example

69. Matrix Stack It is useful to be able to save the current transformation We can push the current state on a stack, and then Make new scale, translations, rotations transformations Then pop the stack and return to status quo ante

70. Example

71. Image is made up of subimages

72. Tree This is harder to do from scratch Jon Squire's fractalgl uses the Matrix Stack

73. 73 Shear Helpful to add one more basic transformation Equivalent to pulling faces in opposite directions

74. 74 Shear Matrix Consider simple shear along x axis x’ = x + y cot  y’ = y z’ = z H(  ) =

75. 75 Pen And Paper Write routines to decide if a circle intersects a line or a line segment How much harder would it be to find the point of intersection?

76. 76 Summary We know enough to make a realistic application Task – Write a program that displays an image, and uses user input to update the image. Don't worry about logic behind the display

77. 77 From Previous class Alex Chou's Pac Man

78. 78 Sample Projects From last class

79. 79 Summary We have played with transformations Spend today looking at movement in 2D In OpenGL, transformations are defined by matrix operations In new version, glTranslate, glRotate, and glScale are deprecated We have seen some 2D matrices for rotation and scaling You cannot define a 2x2 matrix that performs translation A puzzle to solve The most recently applied transformation works first You can push the current matrix state and restore later Turtle Graphics provides an alternative