Mark Guzdial College of Computing Georgia Institute of Technology -Please grab a CD. There are USB drives to borrow – copy the folder “MediaComp-Cambridge- June2010” - Grab a seat in the back with the flip-up tables. Power is in the floor. - We have a coffee break at 10:15 with food. (The catering folks assumed you’d have breakfast at the hotel. We will have breakfast at Marriott Thurs.) - For WiFi, join “Cambridge” network, and use event code sw616.

Introductions Who are you? Where are you from? What do you teach? What do you want to get out of this workshop?

Workshop Plan: Python on Wednesday 9-9:30: Welcome and Introductions. 9:30: Introduction to manipulation of images with Media Computation in Python.Python 10:30-10:45: Coffee Break 11: Cropping and Scaling of Images 12-1: Lunch break (and play with images) 1-2: Introducing sound, sound manipulations, splicing sounds. 2-3: Integrating Media Computation with Alice 3-3:15: Coffee Break. 3:15-4:30. Try manipulating images and sounds, and carrying them in or out of Alice.

Workshop Plan: Java on Thursday 9-10: Research results – what do we know about use of Media Computation? 10-11:00: Mapping this all to Java. Manipulating images and sounds in Java. 10:30-10:45: Coffee Break 11-12: Introducing objects with turtles and MIDI 12-1: Lunch 1-2: Making movies and doing digital video special effects. 2-3:15. Try movies and MIDI yourself. 3:15-3:30: Coffee Break. 3:30-4:30: Group Discussion: Which of these approaches could you use in your classes?

How could I use any of this in my courses? Trying to use MIDI even if can’t record their own voices. Can also bring in recordings on USB stick. Can use Python instead of JavaScript for simple introduction to variables and looping and similar ideas. Using Media to motivate learning. Great way to use nested for-loops (for pictures and movies). Using for intro-to-programming for non-majors. May also for art department’s Photoshop “How to manipulate images” class. Can we do gened real art and real computing. You don’t have to have 1:1 art:computing students. Teaching in a design studio setting, so students are working and seeing one-another’s work.

More using it in my classroom Can use a Wiki to share ideas, when don’t have a physical design studio. Feeling much less depressed teaching a more fun version of intro to Java. Use pictures and sounds as data in more traditional assignments. Adding one or two projects can be too much overhead (learning the libraries) for too little benefit. With Python, enough details are hidden so that you could do a LITTLE bit of media. Can do a little Alice, as an extension – even without a book, even not “officially.” Not enough exercises, not enough of the little bitty ones. “What does this for loop do?” and “Trace this code…”

More on use of classroom Yeah, but am I going to give them “real” examples, too? What can’t we cover here that we might want to cover? Sorting. Engineering computations (e.g., do the area under the curve computation). Queen’s problem, chessboard setups. “Real” arrays. Moving things around in arrays. Writing complete applications, start-to-finish without turtles or multimedia. File processing is there, but hard to adapt to a CS0 class. Strings, lists, file I/O, networking etc. – radically different from the media stuff. Can students deal with the rest of the class after starting with media? The goal of CS1 may be to improve the “survival” rate. Getting them excited is what keeps them going. May result in better grasp of what they do cover.

Workshop Plan: Data structures on Friday 9-10: Introducing linked lists with MIDI 10-11: Linked lists and trees of images : Try making a MIDI song or a linked list animation. 12-1: Lunch. 1-2: Making simulations in GreenfootGreenfoot 2-3: Making our own simulation package. 3-3:15: Coffee break. 3:15-4:30: Choice: Discrete event simulations (to introduce stacks and queues), or Exercise: Make simulations in Greenfoot or the MediaComp Data Structures simulation package.

What’s on my CD (or USB stick)?

Introduction to Computing and Programming in Python: A Multimedia Approach, 2ed Introduction to Computing and Programming in Java: A Multimedia Approach Exploring Wonderland: Java with Alice and Media Computation Problem Solving with Data Structures: A Multimedia Approach Overview of the books

Detailed Overview Follows…

Average 300/term Overall, CS1315 has been 51% female Required in Architecture, Management, Ivan Allen College of Liberal Arts, and Biology Focus: Learning programming and CS concepts within the context of media manipulation and creation Converting images to grayscale and negatives, splicing and reversing sounds, writing programs to generate HTML, creating movies out of Web-accessed content. Computing for communications, not calculation Multi-media CS 1 in Python

Using Python Huge issue Use in commercial contexts authenticates the choice IL&M, Google, Nextel, etc. Minimal syntax Looks like other programming languages Potential for transfer

Rough overview of Syllabus Defining and executing functions Pictures Psychophysics, data structures, defining functions, for loops, if conditionals Bitmap vs. vector notations Sounds Psychophysics, data structures, defining functions, for loops, if conditionals Sampled sounds vs. synthesized, MP3 vs. MIDI Text Converting between media, generating HTML, database, and networking A little trees (directories) and hash tables (database) Movies Then, Computer Science topics (last 1/3 class)

def negative(picture): for px in getPixels(picture): red=getRed(px) green=getGreen(px) blue=getBlue(px) negColor=makeColor(255-red,255-green,255-blue) setColor(px,negColor) def clearRed(picture): for pixel in getPixels(picture): setRed(pixel,0) def greyscale(picture): for p in getPixels(picture): redness=getRed(p) greenness=getGreen(p) blueness=getBlue(p) luminance=(redness+blueness+greenness)/3 setColor(p, makeColor(luminance,luminance,luminance))

16 Open-ended, contextualized homework in Media Computation CS1 Sound collage

Some Computer Science Topics inter-mixed We talk about algorithms across media Sampling a picture (to scale it) is the same algorithm as sampling a sound (to shift frequency) Blending two pictures (fading one into the other) and two sounds is the same algorithm. We talk about representations and mappings (Goedel) From samples to numbers (and into Excel), through a mapping to pixel colors We talk about design and debugging But they mostly don’t hear us

Computer Science Topics as solutions to their problems “Why is PhotoShop so much faster?” Compiling vs. interpreting Machine language and how the computer works “Writing programs is hard! Are there ways to make it easier? Or at least shorter?” Object-oriented programming Functional programming and recursion “Movie-manipulating programs take a long time to execute. Why? How fast/slow can programs be?” Algorithmic complexity

Originally developed for AP CS teachers in Georgia. State of Georgia wanted to train Business teachers to become computer science teachers. Needed an approach to learning CS1 that was successful with business and liberal arts students Multimedia CS1 in Java

Syllabus Introduction to Java Math operators, printing results, data types, casting, relational operators, Strings, variables Introduction to Programming Creating and naming objects Using a turtle and a world Creating new Turtle methods Draw simple shapes Using parameters

Syllabus - Continued Modifying Pictures using Loops One-dimensional arrays Use for-each, while, and for loops to Increase/decrease colors, fake a sunset, lighten and darken, create a negative, and grayscale

Syllabus - Continued Modifying Pixels in a Matrix Two-dimensional arrays Nested loops Copying, mirroring, blending, rotation, scaling

Syllabus - Continued Conditionally Modifying Pixels Boolean expressions Using && and || Replacing a color, reducing red-eye, edge detection, sepia-toned, posterize, highlight extremes, blurring, background subtractions, chromakey

Syllabus - Continued Drawing on Pictures Using existing Java classes Inheritance Interfaces Drawing simple shapes, drawing text, general copy, general scale, shearing, gradient paint, general blending, clipping

Syllabus - Continued Modifying all Samples in a Sound 1D arrays Loops Conditional execution Change volume, normalizing a sound (make it as loud as possible), force to extremes

Syllabus - Continued Modifying Samples using Ranges Loops Clipping, splicing, reversing, mirroring

Syllabus - Continued Combining and Creating Sounds Class and private methods Composing sounds, blending sounds, changing frequencies, and creating echoes Creating sounds Sine Waves, Square Waves, Triangle Waves MP3 and MIDI

Syllabus - Continued Creating Classes Identifying objects and classes Defining a class Overloading constructors Creating and initializing an array Creating getters and setters Creating a main method Javadoc comments Reusing a class via inheritance ConfusedTurtle Turtle

Syllabus - Continued Creating and Modifying Text String methods Reading from and writing to files Handling Exceptions Creating a form letter Modifying programs Getting text from networks Creating random sentences Using text to shift between media

Syllabus - Continued Making Text for the Web Throwing exceptions, “unnamed” package, HashMap, Generics, and Iterators Generating HTML Create a web page from a directory Create a web page from other web pages Databases Creating a web page from a database

Syllabus - Continued Encoding, Manipulating, and Creating Movies Frame-based animations with simple shapes and text Special effects – fade out, fake sunset, and chromakey

Syllabus - Continued Speed What makes programs fast? Compilers and Interpreters Writing a graphics interpreter and compiler Searching Algorithms that can’t be written What makes computers fast? Clock rates, Storage, Display

Syllabus - Continued Javascript Syntax User Interfaces Multimedia

Originally suggested by Randy Pausch In going from Alice to Java, “I’d use Media Computation.” Java Media Computation as the “special effects studio” for Alice Alice and Media Comp in Java

Alice and Media Computati on Intro 35 Intro to Objects, Classes, and Methods Create objects in Alice and invoke methods on them Create a new method in Alice And pass in a parameter Create objects in Java and invoke methods on them Create a new method in Java And pass in a parameter

Alice and Media Computation Intro 36 Sound Projects Create sound clips or splice sounds together to create a soundtrack for your Alice movie Normalize all sounds recorded for character dialog. Manipulate the sounds for an underwater movie to sound as if they were produced under water. Blend two sounds together to add sound effects to your sound track. Change the frequency of a sound and use the new sounds for different Alice objects

Alice and Media Computati on Intro 37 Sound Frequency Project Create an Alice world with 3 frogs Resize one to be bigger and one to be smaller Using croak.wav Create a higher sound Take every other sample Create a lower sound Take every sample twice Import the sound files and have the frogs play their sounds and tap their foot

Alice and Media Computation Intro 38 Picture Projects Negate an Alice picture Darken or lighten an Alice picture Fade out a picture Change the colors on an Alice picture Make it look like sunset Mirror an Alice picture Make it look like a reflection in the water Do chromakey To put Alice objects on a normal photo Or a person in an Alice world

Alice and Media Computati on Intro 39 Using Chromakey Create objects in Alice and position them as desired Delete the ground set the color of the background to green in the world properties Use a conditional to replace the green color You may also need to scale one of the pictures

Alice and Media Computati on Intro 40 Drawing Projects Draw something on an Alice picture Like a hat, glasses, or a scarf Fireworks Create images in Media Computation that you use as textures in Alice like a gradient for the ground Clip an Alice picture To look like a porthole Create a poster advertisement for your Alice movie. With your name on it Create a comic panel with Alice pictures

Alice and Media Computation Intro 41 Comic Strip Project Create several scenes in Alice and take pictures of the scenes Use Media Computation to create a Comic Panel and/or Comic Strips Draw different types of text balloons Speech balloon, thought balloon, scream balloon

Alice and Media Computation Intro 42 Movie Projects Blend two Alice pictures together with different amounts to have one scene fade out and the other fade in Change the background of an Alice movie using chromakey Make an Alice movie look as if it was pencil-drawn by running edge detection on every frame of the movie. Add yourself to an Alice movie using chromakey Add an Alice character to a normal movie using chromakey

Alice and Media Computation Intro 43 Add Yourself to an Alice Movie Use MovieCapturePanel to save out jpg frames from an alice movie or use Alice 2.2 Use chromakey to position a picture of yourself in an Alice frame Change where you place the picture of you to make it look like you are moving

Alice and Media Computation Intro 44 Add Alice Characters to a Real Movie Split out frames from a movie of people Capture frames from an Alice movie using MovieCapturePanel Use chromakey to copy the Alice items to the real movie Use MovieWriter to create a merged movie

Media Computation Data Structures Developed because we finally had Liberal Arts, Architecture, and Management students who wanted to go on to a second course

Driving question: “How did the wildebeests stampede in The Lion King?” Spring 2005: 31 students, 75% female, 91% success rate. Multimedia CS2 in Java

Connecting to the Wildebeests It’s all about data structures

Syllabus Introduction to Java and Media Computation Manipulating turtles, images, MIDI, sampled sounds. Insertion and deletion (with shifting) of sampled sounds (arrays). Structuring Music Goal: A structure for flexible music composition Put MIDI phrases into linked list nodes. Use Weave and Repeat to create repeating motifs as found in Western Music At very end, create a two-branched list to start on trees. CanonSwan BellsFur Elise

HW2: Create a collage, but must use turtles

Syllabus (Continued) Structuring Images Using linearity in linked list to represent ordering (e.g., left to right) Using linearity in linked list to represent layering (as in PowerPoint) Mixing positioned and layered in one structure, using abstract super classes. Structuring a scene in terms of branches—introducing a scene graph (first tree) (We’ll see these slides as an example later.)

Syllabus (Cont’d) Structuring Sound Collecting sampled sounds into linked lists and trees, as with images. But all traversals are recursive. Use different traversals of same tree to generate different sounds. Replace a sound in- place Original Scale the children Scale the next

Syllabus (cont’d) Generalizing lists and trees Create an abstract class “Linked List Node” (LLNode) on top of the sound and image class hierarchies Make all image and sound examples work the same abstract LLNode Knows next Knows how to do all basic list operations

Syllabus (Cont’d) GUIs as trees We introduce construction of a Swing frame as construction of a tree. Different layout managers are then different renderers of the same tree. Traditional (binary) trees as a specialization Can we make trees that are faster to search? JFrame JPanel JLabel “This is panel1!” JPanel JButton “Make a sound” JButton “Make a picture”

Syllabus (cont’d) Lists that Loop Introduce circular linked lists as a way of create Mario- Brothers’ style cel animations. Introduce trees that loop as a way of introducing graphs. jSide.jpg jWalk.jpg jSide.jpg jWalk2.jpg

Syllabus (cont’d) Introducing Simulations Introduce continuous and discrete event simulations, and Normal and uniform probability distributions We do wolves and deer, disease propagation, political influence. Create a set of classes for simulation, then re-write our simulations for those classes. Writing results to a file for later analysis Finally, Making the Wildebeests and Villagers Mapping from positions of our turtles to an animation frame. Creating an animation from a simulation.

HW7: Simulate European emigration to America Students are required to try several different scenarios, aiming for historical accuracy. Counts of Europeans, Americans, and in- transit per year are written to a file for graphing in Excel

Syllabus (cont’d) Introduction to Discrete Event Simulations Create a discrete event simulation of trucks, factories, salespeople, and markets. Use turtles to create an animated display. Now, the real focus is the simulation, and the animation is just a mapping from the simulation. Animation becomes yet another medium in which we can review results, like data in an Excel spreadsheet, music, or sound.