1. Musical Instrument Virtual
Iwan Sonjaya,MT

4. Technology - MIDI Musical Instrument Digital Interface
A protocol that allows electronic musical instruments to communicate with each other (or with a computer)
A MIDI instrument (or computer) usually has a MIDI-in port to receive messages and a MIDI-out port to send messages
MIDI protocol defines messages: e.g. Note-On, Note-Off

5. Technology - Creating drum sounds electronically
Can be easily done via Samplers, Drum Machines, Synthesizers, etc. (both hard- and software)
For my project: Using a free sampling software and a pre-defined drum kit (collection of drum sound samples) that can communicate with the application via MIDI

6. Technology – optical tracking
Vicon tracking / motion capture system consists of
5 infrared cameras
Dedicated computer for pattern recognition
Pattern recognition software

7. Technology – optical tracking
Objects that need to be tracked are equipped with reflective markers
Objects need distinctive marker patterns
By observing the marker pattern from the 5 different perspectives, the objects position and orientation can be calculated
Software has training mode to learn to recognize new objects

8. Tracked objects
Shutter glasses
Drum sticks
Feet

9. Technology – Display System
Barco Trace Large Screen Display
Back Projection Screen with 120 Hz refresh rate for 1280 x resolution
Active Stereo with stereo shutter glasses

10. System Design

11. Implementation Steps Create 3D Models of Drum Kit with 3DS Max
Import them scene graph
Create interface with tracking system (glasses, hand, feet)
Create MIDI connection between application and Software Sampler
Implement collision detection between virtual drum stick positions and drum skins (collision triggers Midi out note, velocity of impact determines the volume of the sound)

12. Main Program Loop Get current position of sticks
Calculate if a collision has occurred (also compare with previous position to avoid continuous sounds)
If collision has occurred
Calculate speed (again by comparing with the previous positions and their timestamps)
Send a midi message (Note On) with Note number relating to the object that was hit and volume relating to the speed
Update Scene (set new position of sticks)
Render Scene
Could be improved by seperating the collision detection/sound output and grpahics in different threads

13. Screenshot

14. Problems discovered
Missing haptic feedback seemed to be a major drawback
Some more research into haptic devices that could be used in this context, Spidar seems to be the only well suited device for drum sticks, but not available at our department

15. Change of Research Direction
Since the haptic path was a dead end due to availability of hardware …
and I was ahead of schedule with the implementation..
The implementation of two other virtual instruments was decided:
Virtual Piano – to compare both tracking performance and haptics with drums
Virtual Synth Control (Theremin like experimental instrument) – to compare replications of traditional instruments with experimental instrument

16. Virtual Piano Changes: New controller: gloves
Different MIDI sound module: free software piano
Piano model generated by the application and not by modelling package
Different collision detection algorithm (need note-on and note-off events)

17. Screenshot

18. Virtual Synth Control (VSC)
Design objectives:
Easy to use
No haptics required
Psychedelic look and sound
Input mode: drum sticks
Sound module used: Free Software synthesizer JX 220

19. Screenshot

20. Wii Piano
Basic idea: use the WiiMote’s infrared camera for very cheap finger tracking 2D
Compare with other input methods (e.g. mouse, touch screen)
Applications: Drawing, Piano

21. WiiMote
Besides the usual buttons the WiiMote has 2 novel forms of input:
Accelerometers
Infrared Camera

22. WiiMote Finger tracking
Normally the infrared camera is used to evaluate the WiiMote’s position relative to the Wii’s sensor bar
For WiiMote Finger tracking, the WiiMote’s camera is used other way round:
WiiMote doesn’t move
Infrared Light source moves

23. WiiMote Finger tracking
No active light source on finger is used but reflection (see passive markers in mocap)

24. WiiMote Finger tracking

25. User Interface: Button Activation