Points of View, Levels of Abstraction Jennifer Burg Department of Computer Science Wake Forest University x(t) = a x t 3 + b x t 2 + c x t + d

Digital Media mediumconcepts application programs digital imaging resolution, bit depth, color models, convolutions, etc. Photoshop, Illustrator digital audio sampling rate, bit depth, aliasing, SNR, MIDI, etc. Audition, Audacity, Sound Forge, Reason, Sonar, MAX/MSP digital video frame rate, chroma keying, compression, etc, Premiere, Final Cut, iMovie

x(t) = a x t 3 + b x t 2 + c x t + d We can tell students: sin(  ) = a/b where a is the length of the side opposite angle  and b is the length of the side adjacent to angle . and This is a sine wave: But a more dynamic presentation can make this clearer. [Excerpt from an interactive tutorial “Visualizing Waveforms in MATLAB”*]Excerpt from an interactive tutorial “Visualizing Waveforms in MATLAB” *Exercises and tutorials in this talk are from The Science of Digital Media by Jennifer Burg, to be published by Prentice-Hall in August 2008.

x(t) = a x t 3 + b x t 2 + c x t + d We can tell students: Just as a sound can be modeled as a one-dimensional wave, the pixel values in a digital image constitute a two-dimensional waveform. But allowing them to create this waveform and examine it themselves can make this clearer. [Screen capture showing how to input a digital image file and graph the pixel values in MATLAB]Screen capture showing how to input a digital image file and graph the pixel values in MATLAB

x(t) = a x t 3 + b x t 2 + c x t + d We can give students the parametric equations that define Bézier curves, and tell them that this is how the pen tool in Illustrator creates curved lines:

x(t) = a x t 3 + b x t 2 + c x t + d But allowing the students to see the equation values change as they create a curve with the pen tool makes the mathematics come alive. [Excerpt from an interactive tutorial on “The Mathematics of Curves”]Excerpt from an interactive tutorial on “The Mathematics of Curves”

x(t) = a x t 3 + b x t 2 + c x t + d We can explain the concept of indexed color to students: In indexed color mode, the number of colors in an RGB digital image is reduced from 16, 277, 216 to 256 or fewer. Methods like the median cut, uniform partitioning, and octree algorithm are used for this purpose…. But allowing them to change a digital image from RGB to indexed color and playing with the palette makes things clearer. [Screen capture showing how to change from RGB to indexed color in Photoshop]Screen capture showing how to change from RGB to indexed color in Photoshop

x(t) = a x t 3 + b x t 2 + c x t + d We can go even farther than that. We can allow them to write their own program for indexed color and dithering and thus understand how they work at a lower level of abstraction. First they have to understand the algorithm. [Interactive tutorial on the octree algorithm for indexed color]Interactive tutorial on the octree algorithm for indexed color

x(t) = a x t 3 + b x t 2 + c x t + d Then they implement the algorithm in some programming language – e.g., C++. void Image::convertToIndexed(void) { Octree tree; unsigned char rBits, gBits, bBits; OctreeNode* ptrToReduce; unsigned char color; RGB colorTable[256]; unsigned char bitmapI[300][300]; unsigned char num = 0; /*Used as an argument to putInTable; the next color index to be put in the color table*/ /* go through all pixels of the image to create the octree */ for (int i = 0; i < height; i++) { for (int j = 0; j < width; j++) { rBits = bitmap.R[i][j]; gBits = bitmap.G[i][j]; bBits = bitmap.B[i][j]; tree.root.insertColor(rBits, gBits, bBits, 0); numLeaves = 0; tree.root.countLeaves(); if(numLeaves > 256) { //traverse tree to find colors to combine as one ptrToReduce = tree.root.traverseToLeaves(0); tree.root.combineColors(ptrToReduce); }

And then the students can have the satisfaction of comparing their results from the “professional” results in Photoshop. MinePhotoshop

x(t) = a x t 3 + b x t 2 + c x t + d Similarly, we can explain the concept of dithering to students: Dithering is a technique for simulating colors that are unavailable in a palette by using available colors that are blended by the eye so that they look like the desired colors. Algorithms like noise, pattern, and error diffusion are used for this purpose…. But allowing them to change a digital image from RGB to indexed color applying dithering makes things clearer. [Screen capture showing how to applying dithering in conjunction with indexed color in Photoshop]Screen capture showing how to applying dithering in conjunction with indexed color in Photoshop

x(t) = a x t 3 + b x t 2 + c x t + d We can go even farther than that. We can allow them to write their own programs for error diffusion and pattern thus understand how they work at a lower level of abstraction. First they have to understand the algorithm. [Interactive tutorial on dithering]Interactive tutorial on dithering

Then they implement the algorithm in some programming language – e.g., C++. for (int i = 0; i <= height-3; i=i+3) { for (int j = 0; j <= width-3; j=j+3) { for (k = i, u = 0; k <= i+2; k++, u++) { for (l = j, v = 0; l <= j+2; l++, v++) { pixelNormalized = *(pixels+(w*k)+l)/28; if (pixelNormalized >= mask[u][v]) *(bitmap+(w*k)+l) = 1; else *(bitmap+(w*k)+l) = 0; }

And then the students can have the satisfaction of comparing their results from the “professional” results in Photoshop. MinePhotoshop

x(t) = a x t 3 + b x t 2 + c x t + d So what is my point? I’m just trying to reinforce, with examples, something that you already know: Not all students learn the same way. Some think in terms of mathematical abstractions, some algorithmically, some visually, some verbally. Even if they think, habitually, a certain way, it’s good for students to see the concepts presented from different points of view and at different levels of abstraction.

x(t) = a x t 3 + b x t 2 + c x t + d Computer learning environments can help us to place students at different levels of abstraction. Collaborative work in these learning environments can provide alternative perspectives from students of different disciplines.

Points of View, Levels of Abstraction sciencearts application theory

Levels of Abstraction Digital Audio and MIDI AuditionMusic Creator MATLAB MAX/MSP C Programs

x(t) = a x t 3 + b x t 2 + c x t + d Clearly, it’s helpful in science and mathematics to move back and forth among levels of abstraction. How does this work in the teaching of history? English? Languages? Art? Theater? Dance? Music?

Levels of abstraction, moving from theory to application, represent one axis. The other axis moves from science to the arts. What’s the pedagogic usefulness of moving along this other axis, between disciplines? I’ve chosen computer science and art as the two disciplines at the end of my axes, but for you it might be something else.

artists (musicians or digital sound designers) digital audio and MIDI processing Points of View computer scientists When and why compress the dynamic range? Does it sound thin? Should we add reverb? How does compression of dynamic range work? Does it affect frequencies? What filtering tools exist? Should I make my own?

My collaborator, Jason Romney, Digital Sound Designer from North Carolina School of the Arts. NSF CCLI grant: “Linking Science,Art, and Practice with Digital Sound” Jennifer Burg, PI Jason Romney, co-PI

x(t) = a x t 3 + b x t 2 + c x t + d The experiment: Three graduate students from North Carolina School of the Arts’ School of Design and Production joined seven of my Digital Media students to try some sound exercises in MATLAB and MAX/MSP.

x(t) = a x t 3 + b x t 2 + c x t + d The assignment: Try to create a single-frequency note in digital audio within MSP. See if you can generate a sine wave at the frequency you want. The sound should be in digital audio form. Try to apply a filter to the sound to determine if its frequency component or components are what you expect them to be. If you can get the frequency out of the filter as a number, see if you can then map that number to a MIDI note of the same frequency. That is, send a MIDI output to a MIDI device to see if you can play that same note through the MIDI device.

MAX/MSP patch

The experiment: Music and theatre students join computer science students in a ½ semester course entitled “Digital Sound for Music and Theatre.” How much will they be able to learn about the science and practice of digital sound production? Will they be able to produce a song or a theatre scene? How will they interact and collaborate?

The assignment: A computer science student and theatre student are paired to do the sound for the opening storm scene of Shakespeare’s The Tempest.

Storm scene from “The Tempest”

x(t) = a x t 3 + b x t 2 + c x t + d To get to the point, is there something to be gained by placing students of different disciplines in collaborative learning situations? … for example, computer science students and students of digital sound design for theatre.

It makes the work more INTERESTING, RELEVANT, and EXCITING. The computer science students learn what tools are really useful to practitioners and artists, and how these tools are used. The music and theatre students learn that they can have greater power over their tools by discovering how to do things at a lower level of abstraction.

Ideas generated: How to solve a sound design problem for the play Floyd Collins. How to solve a sound design problem for The Lion King. Things that matter to sound designers and things that don’t matter Training your ears.

This summer: An 8-week music production workshop with 3 computer science students and 3 music students. We’ll see what happens!