Celluloid An interactive media sequencing language

Today Overview of Celluloid Sample Code Syntax and Semantics Translator Architecture Framework Architecture Development Environment Testing Conclusion

Overview Interactive Media sequencing language Designed to allow users to easily construct their own media systems and allow for real time interaction with the programmer to dynamically change programs on the fly allowing programming to be elevated to a performance art Programs are translated into Java SE which can then be compiled into Java bytecode and run on any JVM

Motivation Celluloid was motivated by the growing role interactive media plays in our daily lives We’ve focused on trying to build a language that targets a wide audience and remains simple and intuitive With Celluloid, you can build the technical and artistic developments that will power the future of the media age

Important Properties Reactive programming allows modified primitives to affect later primitives defined with them Celluloid is designed to be very human readable Provides facilities for abstracting away the hardware details of underlying input and output devices to increase portability

Visualizing a Celluloid System Multiple devices are connected and abstracted through the device type. Enables software mixing that was previously not possible.

Type System Primitive Types: number, boolean, string, time. Timelines: Controls the playback of inputs in the order specified in the source program Devices: Encapsulate IO abstractions. Devices are responsible for providing the interface between external media and internal components Constraints: Enforce a set of allowed operationson a device as well as share this definition across devices. Events: Allows for simple message passing between devices and timelines.

Language Properties Compiled statically typed language. Compiled into Java code built on a custom framework. Allows for user defined functions and user defined types. Custom device, constraint, and event types are allowed.

Sample Program timeline audio input audio1 = new AudioFile(“audiofile1.mp3”) input audio2 = new AudioFile(“audiofile2.mp3”) input audio3 = new AudioFile(“audiofile3.mp3”) output main = new AudioOutput(“Audio”) in audio do play play play end in main do play end

Translator Architecture celluloid.g celluloidLexer.g celluloidParser.g celluloid.java.stg source_code.cld target_code.java org.celluloidLang.*

Framework Architecture Java Media Framework Implementation of Timelines Implementation of Devices Implementation of Constraints Announcements and Events Reactive Programming

Java Media Framework (JMF) Designed to manipulate “audio, video and other time-based media” Cross platform support Supports many popular formats ▫ Plugins available to support additional formats ▫ FOBS for JMF

Implementation of Timelines Controls the playback of inputs in the order specified in the source program Timer object keeps track of time and executes the appropriate action at the appropriate time System utilitzes 100 millisecond time-slices HashMap containing when-based events and the corresponding action to execute when the event’s condition(s) is satisfied

Implementation of Devices Two built-in devices for reading audio and video files are provided out-of-box: JMFVideo and JMFAudio Framework provides the ability to implement real- time and static media Examples of real-time media: midi devices, streaming media, instruments

Implementation of Constraints Constraint interfaces ▫ Enforce allowed operations on device classes in the framework Constraint functions ▫ Keeps track of what should be executed and when it should be executed ▫ Anonymous inner class keeps track of what is executed

Sample Constraint Translation in timelineOne do play end... would translate to... ConstraintFunction cf = new ConstraintFunction (inputOne, 0) { public void execute() { input.play(); //where (input == inputOne) } timelineOne.addConstraintFunction(cf);

Reactive Programming Reactive programming involves the propagation of change among assignment expressions. Example: ▫ A=4; ▫ B=9; ▫ C = A + B ▫ Print C //displays 13 ▫ A=10 ▫ Print C //displays 19

Announcements and Events Announcements and events allow for simple message passing between devices and timelines. Implemented as a global event broker (i.e. event listener) available to relevant types in the framework Example: when Audio5 audioGain > 7 do play Video4 end

Development Environment IDE Devolopment ▫ IntelliJ ▫ Eclipse ▫ ANTLRWorks Unit Testing JUnit Repository ▫ Git ▫ GitHub

Testing Goal: Find faults and determine whether those faults are translation faults or target framework faults. Black Box White Box Unit Tests ▫ Full unit test suite for both the parser and the framework Integration Tests ▫ Testing hand of off parser to the framework

Conclusion What we learned: ▫ Learned tools for developing language ▫ Implement language Worked well: ▫ Repository, issue tracking, Google docs What to do different: Start earlier Future development ▫ REPL- Read eval print loop

Questions?