Media Computation: Introducing Computing Contextualized in Video and Audio Processing Mark Guzdial and Barbara Ericson

2 Story Computing is important for everyone –Our track record isn’t so good. Media Computation: Making computing relevant and creative. –In Python and Java, CS1 and Data Structures –Implementing video and audio effects: Echoes and reversing Background subtraction and chromakey Connecting to other approaches –Alice + Media Computation –Robotics + Media Computation Assessment Results

3 The challenge of 1999 Fall 1999: All students at Georgia Tech must take a course in computer science. Why? –Computing was a College. –The tools of learning for computational scientists and engineers brought into the classroom. –Making competitive distinctions for Liberal Arts We have spent nearly 10 years trying to make this work!

4 Richard Dawkins on Fresh Aire, April 2007 GROSS: You close your book saying, "I am thrilled to be alive at a time when humanity is pushing against the limits of understanding." How do you think that's happening in your field of evolutionary biology? Mr. DAWKINS: Well, it's the most exciting time to be a biologist…Since Watson and Crick in 1953, biology has become a sort of branch of computer science. I mean, genes are just long computer tapes, and they use a code which is just another kind of computer code. It's quaternary rather than binary, but it's read in a sequential way just like a computer tape. It's transcribed. It's copied and pasted. All the familiar metaphors from computer science fit.

5 Only one class: Pass (A, B, or C) vs. WDF (Withdrawal, D or F) PassWDF 02 FallTotal74.01%26.74% Female62.99%36.65% Male77.00%22.90% 02 SpringTotal65.03%34.87% Female65.56%34.44% Male64.81%35.04% 01 FallTotal70.98%29.02% Female59.55%40.45% Male73.63%26.37% At first, only one class met the requirement

6 Contextualized Computing Education Since Spring 2003, we now teach 3 intro CS courses. –Responding to research results about CS being “irrelevant” –Based on Margolis and Fisher “alternative paths” Each course introduces computing using a context (examples, homework assignments, lecture discussion) relevant to majors.

7 Our Three CS1’s CS1301 Introduction to Computing Traditional CS1 for our CS majors, Science majors (math, physics, psychology, etc.). Now: All robotics. CS1371 Computing for Engineers CS1 for Engineers. Same topics, but using MATLAB with Engineering problems in homework and examples. CS1315 Introduction to Media Computation

8 Average 400/term –Overall, CS1315 has been 51% female –Required in Architecture, Management, Ivan Allen College of Liberal Arts, and Biology Focus: Learning programming and CS concepts within the context of media manipulation and creation –Converting images to grayscale and negatives, splicing and reversing sounds, writing programs to generate HTML, creating movies out of Web-accessed content. –Computing for communications, not calculation

9 Media Computation: Teaching in a Relevant Context Presenting CS topics with media projects and examples –Iteration as creating negative and grayscale images –Indexing in a range as removing redeye –Algorithms for blending both images and sounds –Linked lists as song fragments woven to make music –Information encodings as sound visualizations

10 def negative(picture): for px in getPixels(picture): red=getRed(px) green=getGreen(px) blue=getBlue(px) negColor=makeColor(255-red,255-green,255-blue) setColor(px,negColor) def clearRed(picture): for pixel in getPixels(picture): setRed(pixel,0) def greyscale(picture): for p in getPixels(picture): redness=getRed(p) greenness=getGreen(p) blueness=getBlue(p) luminance=(redness+blueness+greenness)/3 setColor(p, makeColor(luminance,luminance,luminance))

11 Examples of Student Work Soup- Audio Collage Canon- LinkedList of (MIDI) Music

12 Effects Examples from Media Computation Simple audio effects From simple image manipulations, to video special effects.

13 Reversing a sound def backwards(filename): source = makeSound(filename) target = makeSound(filename) sourceIndex = getLength(source) for targetIndex in range(1,getLength(target)+1): sourceValue = getSampleValueAt(source,sourceIndex) setSampleValueAt(target,targetIndex,sourceValue) sourceIndex = sourceIndex - 1 return target Simply reversing values in an array – but when it’s a sound, it’s fun!

14 Adding sounds

15 A function for adding two sounds def addSounds(sound1,sound2): for index in range(1,getLength(sound1)): s1Sample = getSampleValueAt(sound1,index) s2Sample = getSampleValueAt(sound2,index) setSampleValueAt(sound2,index,s1Sample+s2Sample)

16 Uses for adding sounds We can mix sounds –We even know how to change the volumes of the two sounds, even over time (e.g., fading in or fading out) We can create echoes We can add sine (or other) waves together to create kinds of instruments/sounds that never existed in nature, but sound complex

17 A Sunset Effect How do we turn this beach scene into a sunset? What happens at sunset? –Theory: As the sun sets, less blue and green is visible, which makes things look more red.

18 A Sunset-generation Function def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.7) value=getGreen(p) setGreen(p,value*0.7)

19 SlowSunset as a movie def slowsunset(directory): canvas = makePicture(getMediaPath("beach-smaller.jpg")) #outside the loop! for frame in range(0,100): #99 frames printNow("Frame number: "+str(frame)) makeSunset(canvas) # Now, write out the frame writeFrame(frame,directory,canvas) def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.99) #Just 1% decrease! value=getGreen(p) setGreen(p,value*0.99) Just one canvas repeatedly being manipulated

20 SlowSunset frames

21 Example Movies

22 Fading by background subtraction Background subtraction: def swapbg(person, bg, newbg,threshold): for x in range(1,getWidth(person)): for y in range(1,getHeight(person)): personPixel = getPixel(person,x,y) bgpx = getPixel(bg,x,y) personColor= getColor(personPixel) bgColor = getColor(bgpx) if distance(personColor,bgColor) < threshold: bgcolor = getColor(getPixel(newbg,x,y)) setColor(personPixel, bgcolor) Threshold as an input. Using the frame number

23 SlowFadeout

24 Example Movies

25 Chromakey Method public void chromakey(Picture newBg, Color color double dist) { Pixel currPixel = null; Pixel newPixel = null; // loop through the columns for (int x=0; x<getWidth(); x++) { // loop through the rows for (int y=0; y<getHeight(); y++) {

26 Chromakey Method - Cont // get the current pixel currPixel = this.getPixel(x,y); /* if the color at the current pixel is mostly blue * (blue value is greater than red and green combined), * then use the new background color */ double currDist = currPixel.colorDistance(color); if (currDist <= dist) { newPixel = newBg.getPixel(x,y); currPixel.setColor(newPixel.getColor()); }

27 Code for Kids on Moon Movie public void makeKidsOnMoonMovie(String dir) { String kidsDir = FileChooser.getMediaPath("kids-blue/"); String moonF = FileChooser.getMediaPath("moon-surface.jpg"); Picture moonP = new Picture(moonF); FrameSequencer frameSequencer = new FrameSequencer(dir); Picture currP = null; // get the array of files in the directory File dirObj = new File(kidsDir); String[] fileArray = dirObj.list();

28 Code for Kids on Moon Movie - Cont // loop through the array of files for (int i = 0; i < fileArray.length; i++) { if (fileArray[i].indexOf(".jpg") >= 0) { currP = new Picture(kidsDir + fileArray[i]); currP.chromakey(moonP,Color.black,100.0); frameSequencer.addFrame(currP); } // play the movie frameSequencer.play(30); }

29 Example Barb, I don’t have this one

30 Using Media Computation with Other Approaches Alice + Media Computation Robotics + Media Computation

31 Alice + Media Computation Barb, please put a couple slides here, plus movie?

32 A Context for CS1 for CS majors: Robotics Microsoft Research has funded the Institute for Personal Robotics in Education –Tucker Balch, Directing Joint between Bryn Mawr and Georgia Tech – Goal is to develop a CS1 and CS2 with robotics as the context. –Added a camera and media computation abilities

33 Robot Movies Robots have cameras, and Myro has media computation primitives. Wonderful project: Creative and Collaborative –Robots are characters. –One robot is camera How do you zoom? Aim and go forward! –Post-processing media computation for eerie disappearing effects.

34 Example movie

35 A Media Computation Data Structures Course Driving question: “How did the wildebeests stampede in The Lion King?”

36 Connecting to the Wildebeests It’s all about data structures

37 Similar Assignments, but with Objects and Agents

38 Results of four years of evaluation MediaComp students are more motivated and engaged (retention data, interviews), and find the course social, creative, and “relevant.” –Replicated at several institutions now. Students in the contextualized courses program outside of class. –Immediately (engineers) and even a year later (MediaComp) Students in MediaComp classes (both, and new Architecture course) spend extra time on homework “because it’s cool.”

39 Results: CS1315 “Media Computation” PassWDF 04 FallTotal72.55%27.45% Female82.90%17.10% Male77.46%22.54% 04 SpringTotal89.87%9.37% Female91.94%7.58% Male87.50%11.41% 03 FallTotal86.47%12.54% Female88.36%10.27% Male84.71%14.65%

40 Success Rates for Specific Majors Success rates in traditional CS1 for students in various majors average Fall ’99 to Fall ’02, compared to Spring ’03 to Fall ’05 in Media Computation.

41 “Did the class change how you interact with computers?” Results from a survey a year later: –“Definitely makes me think of what is going on behind the scenes of such programs like Photoshop and Illustrator.” –'I understand technological concepts more easily now; I am more willing and able to experience new things with computers now ’ –'I have learned more about the big picture behind computer science and programming. This has helped me to figure out how to use programs that I've never used before, troubleshoot problems on my own computer, use programs that I was already familiar with in a more sophisticated way, and given me more confidence to try to problem solve, explore, and fix my computer.’

42 The Other Results We don’t know if they learn the same. –The challenge of comparative studies when there is no common reality. While engaging, majority of students do not find the MediaComp courses relevant to their degrees or professions. –Many do find it relevant to their lives. Students distinguish between “more MediaComp classes” and “more CS classes”

43 Summary Media Computation is an approach to teaching computing about image, audio, and video effects. –Teaching computing through developing the techniques behind gaming and other digital media. Ties easily to other approaches. –Alice, Robotics…gaming? Assessment results are promising. –Retention result is very strong