Introducing Computer Programming with Multimedia Barb Ericson Institute for Computing Education Georgia Institute of Technology

Computer Science Crisis 40% drop in majors since 2001 Nationwide –Up to 60% drop in some places Percentage of women is down to about 12% –From a high of 30% in the 1980s Never attracted many African Americans, Hispanics, Native Americans

Why the Decline? Dot com crash in 2001 –Loss of jobs AP Exam switched from C++ to Java –Many schools quit teaching AP Computer Science Visual Basic changed –In.net to be object-oriented Perceptions of Computer Science –boring, antisocial, not creative, and just programming

Looming Shortage Only a few jobs have gone overseas –About 2-3% We have recovered from the dot com crash –There are now more jobs in this area than in 2001 Computing jobs are some of the fastest growing –US Dept of Labor predicts 45.93% growth

Reversing the Trend We need to get more students interested in computer science –Not drive them away We need to appeal to a diverse group of students –Not just geeks All students need to have some understanding of computer science –Our society relies on technology

Our Solution Use multimedia projects to introduce computing concepts –Pictures, sounds, movies, text Use open-ended projects to encourage creativity –Create an image collage, create a sound collage, create a movie Make computing social –Encourage students to show off their work

Choice of Languages Python –Used by many companies: IL&M, Google, Nextel, etc. –Easier syntax than many languages –Can use to teach functional, procedural, or object- oriented programming –Used as a scripting or extension language for many applications –Growing in popularity Java –Popular language used for AP CS

Digitizing Pictures We digitize pictures into lots of little dots Enough dots and it looks like a continuous whole to our eye Each picture element is referred to as a pixel –Each pixel knows its color –It also knows where it is in its picture

Encoding Color: RGB In RGB, each color has three component colors: –Red, Green, and Blue In most computer-based models of RGB, a single byte (8 bits) is used for each –So a complete RGB color is 24 bits, 8 bits of each –0,0,0 is black –255,255,255 is white –255,0,0 is red

Make and Show a Picture Use pickAFile function To select a picture file (ends in.jpg) Use makePicture(file) to create the picture Use show(picture) to display the picture

Use MediaTools Click on MediaTools Select the picture to explore Move the cursor while holding down the left mouse button –To see x,y,r,g,b Click on zoom –To zoom in or out

Creating a Python Function Define a function Load the function Try the function

Rules for Python Functions Use the keyword def to define a function –Followed by the function name Use parentheses after the name –Name parameters End with a colon Indent the body of the function def pickAndShow(): myFile = pickAFile() myPicture = makePicture(myFile) show(myPicture)

Using a Loop def decreaseRed(picture): for p in getPixels(picture): value = getRed(p) setRed(p,value * 0.5)

More Picture Functions def greyscale(picture): for p in getPixels(picture): redness=getRed(p) greenness=getGreen(p) blueness=getBlue(p) luminance=(redness+blueness+greenness)/3 setColor(p, makeColor(luminance,luminance,luminance)) def negate(picture): for px in getPixels(picture): red=getRed(px) green=getGreen(px) blue=getBlue(px) negColor=makeColor(255-red,255-green,255- blue) setColor(px,negColor)

How Sound Works: Sounds are waves of air pressure –Sound comes in cycles –The frequency of a wave is the number of cycles per second (cps), or Hertz –The amplitude is the maximum height of the wave

Volume and Pitch Our perception of volume is related (logarithmically) to changes in amplitude –If the amplitude doubles, it’s about a 3 decibel (dB) change. Our perception of pitch is related (logarithmically) to changes in frequency –Higher frequencies are perceived as higher pitches –We can hear between 5 Hz and 20,000 Hz (20 kHz) –A above middle C is 440 Hz

Digitizing Sound We can estimate the sound curve –Analog-to-digital conversion (ADC) will give us the amplitude at an instant as a number: a sample –How many samples do we need? 2 x max frequency CD quality is 44,100 samples per second

Digitizing sound in the computer Each sample is stored as a number –16 bits, 2 16 = 65,536 combinations –But we want both positive and negative values To indicate compressions and rarefactions. Each sample can be between -32,768 and 32,767

Exploring a Sound Use MediaTools to explore a Sound

Working with Sounds We’ll use getSamples to get all the sample objects out of a sound –Use getSample to get the value from a Sample obj We can also get the value at any index with getSampleValueAt Sounds also know their length (getLength) and their sampling rate (getSamplingRate) Save a sound to a file using writeSoundTo(sound,”file.wav”)

Decreasing the volume def decreaseVolume(sound): for sample in getSamples(sound): value = getSample(sample) setSample(sample,value * 0.5) This works just like decreaseRed, but decreasing the amplitude (volume) instead of the red color.

Multimedia Java CS1 We use DrJava –A free and easy development environment –With an interactions pane We have created Java classes –For the students to use and modify World Turtle Picture Sound FileChooser

Introduction to Java –Math operators, printing results, data types, casting, relational operators, Strings, variables Introduction to Programming –Creating and naming objects Using a turtle and a world –Creating new Turtle methods Draw simple shapes Using parameters Java Syllabus

Syllabus - Continued Modifying Pictures using Loops –One-dimensional arrays –Use for-each, while, and for loops to –Increase/decrease colors, fake a sunset, lighten and darken, create a negative, and grayscale

Create and Show a Picture

Java Negate Method /** * Method to negate the picture */ public void negate() { Pixel[] pixelArray = this.getPixels(); Pixel pixelObj = null; int redValue, blueValue, greenValue = 0; // loop through all the pixels for (int i = 0; i < pixelArray.length; i++)

Java Negate Method - Cont { // get the current pixel pixelObj = pixelArray[i]; // get the values redValue = pixelObj.getRed(); greenValue = pixelObj.getGreen(); blueValue = pixelObj.getBlue(); // set the pixel's color pixelObj.setColor( new Color(255 - redValue, greenValue, blueValue)); }

Testing Negate

Syllabus - Continued Modifying Pixels in a Matrix –Two-dimensional arrays –Nested loops –Copying, mirroring, blending, rotation, scaling

Syllabus - Continued Conditionally Modifying Pixels –Replacing a color, reducing red-eye, edge detection, sepia-toned, posterize, highlight extremes, blurring, background subtractions, chromakey

Syllabus - Continued Drawing on Pictures –Using existing Java classes Inheritance Interfaces –Drawing simple shapes, drawing text, general copy, general scale, shearing, gradient paint, general blending, clipping

Syllabus - Continued Modifying all Samples in a Sound –1D arrays –Loops –Conditional execution –Change volume, normalizing a sound (make it as loud as possible), force to extremes

Syllabus - Continued Modifying Samples using Ranges –Loops –Clipping, splicing, reversing, mirroring

Syllabus - Continued Combining and Creating Sounds –Class and private methods –Composing sounds, blending sounds, changing frequencies, and creating echoes –MP3 and MIDI

Syllabus - Continued Creating Classes –Defining a class –Overloading constructors –Creating and initializing an array –Creating getters and setters –Creating a main method –Javadoc comments –Using Inheritance ConfusedTurtleTurtle

Syllabus - Continued Creating and Modifying Text –Reading from and writing to files Handling Exceptions –Modifying programs –Getting text from networks –Creating random sentences –Using text to shift between media

Syllabus - Continued Making Text for the Web –Throwing exceptions, HashMap, Generics, and Iterators –Create a web page from a directory –Create a web page from other web pages –Creating a web page from a database

Syllabus - Continued Encoding, Manipulating, and Creating Movies –Frame-based animations with simple shapes and text –Special effects – fade out, fake sunset, and chromakey

Syllabus - Continued Speed –What makes programs fast? Compilers and Interpreters Searching Algorithms that can’t be written –What makes computers fast? Clock rates, Storage, Display

Syllabus - Continued Javascript –Syntax –User Interfaces –Multimedia

Multimedia CS2 in Java Driving question: “How did the wildebeests stampede in The Lion King?”

CS2 Ideas Teach data structures using multimedia –Stacks and queues for simulations –Linked lists of pictures Order in linked list is order in scene Order in linked list is the layer the picture is in Circular linked list for animation –Linked lists of sounds Weave and repeat musical phrases –Trees of pictures Scene graph –Trees of sounds Different traversal gives different result

Examples of Student Work Soup- Audio Collage Canon- LinkedList of (MIDI) Music

Results Python CS1 is used for non-majors at Georgia Tech –Business students went from 49% to 88% pass rate –Similar results at Gainesville and U.Ill-Chicago Java CS1 is used in Georgia High Schools –Both in an introductory course and in Advanced Placement courses –Also used at Dennison and Duke Java CS2 is used at Georgia Tech –Students are transferring into computer science

Why Media Computation? Works on nearly all computers –Unlike Alice which requires newer computers Low cost –Unlike LEGO Mindstorms or other robots Students can see (and hear) if their programs work Computer Science concepts are introduced to solve problems –Looping instead of changing thousands of pixels by hand Open-ended assignments allow students to be creative –Students can use their own media Students can publicly share their media artifacts with others

What’s on the CD CS1-Python materials –Chapters from the Media Computation book in Python –Course slides –Free Jython Environment for Students (JES) –Instructor Resources CS1-Java materials –Chapters from the Media Computation book in Java –Course Slides –Java Classes

What’s on the CD - Continued CS2-Java materials –Course notes –Course slides –Java Classes DrJava –Free Development environment for Java Mediasources –Royalty free pictures and sounds

Additional Resources Web Site for Media Computation – CS1 Python Book –Introduction to Computing and Programming in Python, A Multimedia Approach by Mark Guzdial, ISBN: CS1 Java Book –Introduction to Computing and Programming with Java, A Multimedia Approach by Mark Guzdial and Barbara Ericson, ISBN: CS2 Java Book –Coming in 2007

Funding Sources National Science Foundation Georgia Tech's College of Computing Georgia’s Department of Education Al West Fund Toyota Foundation