KEYed User Interface An HCI Theoretic Design of a Novel Music Composition Interface by Farhan Mohamed supervised by Dr. Sidney Fels Human Communication Technologies Laboratory The University of British Columbia Department of Electrical and Computer Engineering Vancouver, British Columbia Canada

Motivation “ While composing with my music studio computer, I find it stressful to use my piano keyboard, computer keyboard and mouse repetitively, especially when I feel creative and work over long hours” - Composer/Producer Kevin Kowal Computer Keyboard Piano Keyboard Computer Mouse Kevin’s Studio Computer Setup

Goal Momentary Footswitch Electronic Piano Keyboard for Performance and Editing Single-Point Touchpad The KEYed User Interface project was created to bind music performance and editing tasks by using the piano keyboard as the sole input device for all tasks Kevin’s Setup using KEYed User Interface

Overview of Presentation Current music composition workstation setup Alternative approaches and their shortcomings Applying findings in HCI KEYed User Interface system Experiments and results Conclusions

Switching between multiple input devices Long device acquisition time Not comfortable to use Lots of physical space occupied by the input devices Music Composition Workstation Computer Keyboard Piano Keyboard Computer Mouse Composition Workstation Setup

Alternative Approaches Mackie’s ® Human User Interface ® Meredith Brooks’ Studio Layout Interface separates performance and editing - Mackie HUI hardware Interface integrates performance and editing - Logic® Audio Music Software

Applying HCI Findings We apply HCI studies to composition, integrating the two tasks HCI studies have been applied to performance and editing

HCI Findings Mode Errors and their Prevention Human Manual Studies - Bimanual Tacit Knowledge and Metaphors

Space - Time Multiplexing Tradeoffs in Input Devices HCI Findings

Momentary Footswitch Electronic Piano Keyboard Keyboard Single-Point Touchpad Overview ofKEYed User Interface Overview of KEYed User Interface

HCI Principles Applied toKEYed User Interface to KEYed User Interface Momentary Footswitch Electronic Piano Keyboard Keyboard Single-Point Touchpad Mode Error Prevention Bimanual Action Space Multiplexed

E2 Arrange Octave (C4-B4) Mapping Example: Transport Octave (C3-B3) Mapping Example: C3Open/Close transport window C5 Open/Close key edit window F#6 Undo TracksParts Sequence C2 B2 C3 B3 C4 B4 C5 B5 C6 B6 Touch Pad Octave (C2-B2) Mapping Example: General Edit Octave (C6-B6) Mapping Example: Key Edit Octave (C5-B5) Mapping Example: F4 Move down the track E2 Selecting volume G3 Click On/Off (B3) Note Piano Mappings

KEYed UI Prototype 1 Prototype I - Steinberg ® Cubase ® 5.0 VST 32 on Windows ® 98 KEYed Pedal for Mode Switching Octaves and the Notes are labeled

KEYedUI Prototype I Studies KEYed UI Prototype I Studies The interaction methods compared are: a)Computer Keyboard + Piano Keyboard b)KEYed User Interface Prototype I Testing for Response Time with 6 Composers Composing Tasks A&B, Repeated Once Note: Task A and Task B involve repetitive performing and editing, thereby requiring the composers to switch back and forth between modes on their own.

KEYedUI Prototype I Results KEYed UI Prototype I Results Mean Time with 95% Confidence error bars Test 1 Test 2 Learning Time: 5 minutes Results using paired t-test: Test 1 - Piano controller faster, significant (p<0.05) Test 2 - Piano controller faster, not significant (p<0.06) No mode errors detected

KEYed UI Prototype II Prototype II platform −Steinberg ® Nuendo ® 1.5 on Windows ® 2000 KEYed Pedal and Sustain Pedal A single-point touchpad for 1D and 2D tasks Single-Point Touchpad KEYed Pedal

KEYedUI Prototype II Studies KEYed UI Prototype II Studies 1.The interaction methods compared are: a) Computer Keyboard / Mouse + Piano Keyboard b) KEYed b) KEYed UI II with Audio c) KEYed c) KEYed UI II without Audio Testing for Response Time with 10 Composers Simple, Moderate and Complex editing tasks Note: All tasks involve repetitive composing and editing, thereby requiring the composers to switch back and forth between modes on their own. 2.Case study for a one week period with 1 Composer

KEYedUI Prototype II Studies KEYed UI Prototype II Studies Experiment Booth KEYed user interface Layout

Mean Completion times KEYedUI Prototype II Results KEYed UI Prototype II Results Learning Time: 16 minutes KEYed Prototype II with Audio vs. Computer Keyboard / Mouse : Response time faster with the KEYed UI with no significance KEYed Prototype II without Audio vs. Computer Keyboard / Mouse : Response time faster with the KEYed UI with no significance

KEYedUI Prototype II Results KEYed UI Prototype II Results Case study results Positive feedback Very natural to use Requested more mappings Suggested the use of musical chords Single-point touchpad too small

Piano keyboard can be used as an alternative input device in music composition workstations A momentary foot pedal like the KEYed foot pedal is easy to acquire, and is an ideal mode switch for switching between the performing and editing modes Though small, the single-point touchpad is found to be natural to use for bimanual tasks Contributions

Contributions KEYed user interface illustrates how an appropriate mapping of the layout, feedback, and context is important in the design of user interfaces. Results published at CHI’2002 and ICMC’2002.

Overlaying composition functions on other controllers Exploring musical structures and mappings, for example, scales and chords Providing user customizable mappings Future Directions

Conclusions Current approaches to music composition workstations design divide the performance and editing tasks In pursuing the KEYed project, we have found that findings in HCI can be applied to the problems in current music composition workstation design Gives rise to faster response time in devices with minimal learning Ultimately, greater comfort leads to greater creative flow and better music

Questions

KEYed UI System Design MIDI OUT MIDI IN MIDI THRU SB PCI 128 MIDI OUT MIDI IN MIDITHRU SB Live MIDI OUT MIDI IN MIDITHRU KEYed Messaging System Cubase 5.0 VST Or Nuendo 1.5 KEYed Dynamic Link Library (Window Handles) Event Windows Queue KEYed Sustain Pedal Pedal Serial Port Single-point Touchpad Figure 3.17: KEYed User Interface Software Design Windows Applications

Measured Mean Task Completion Time vs Predicted KLM Times Measured Means (Seconds) Predicted Time A (Seconds) Predicted Time B (Seconds) Task A Computer Keyboard R R1 KEYed Prototype I R1 Task B Computer Keyboard R R2 KEYed Prototype I R2 Task C Computer Keyboard (CK) / Mouse (M) 40.74CK: R3CK: 46 + R3 M: R3M: R3 KEYed Prototype II (with audio) R3 KEYed Prototype II (without audio) R3 Touchpad R R3 Task D Computer Keyboard (CK) / Mouse (M) CK: Not measurable M: R4M: R4 KEYed Prototype II (with audio) R R4 KEYed Prototype II (without audio) R R4 Touchpad R R4 Task E Computer Keyboard (CK) / Mouse (M) CK: Not measurable M: R5M: R5 KEYed Prototype II (with audio) R R5 KEYed Prototype II (without audio) R R5 Touchpad R R5

Measured Mean Task Completion Time vs Predicted KLM Times CKKP1 CK KP1 TASK A TASK B CK/M KP2A KP2NA TP CK/M KP2A KP2NA TP CK/M KP2A KP2NA TP TASK C (Simple) TASK D (Moderate) TASK E (Complex) Where, CK = Computer Keyboard KP1 = KEYed Prototype I KP2 = KEYed Prototype I M = Mouse KP2A = KEYed Prototype II with Audio KP2NA = KEYed Prototype II without Audio TP = Touchpad

Single point touchpad Ex. Drawing graphs with the touchpad