Musical Virtual Reality Applications

1
Musical Virtual Reality Applications

• Michael Kriegel

2
Contents

• My 2005 final MSc project at the Centre for
  Virtual Environments (University of Salford)
• A current Heriot Watt student project Wii Piano
  by Thomas Methven

3
Motivation

• Personal interest in electronic music technology
  (sampling, synthesizers, MIDI)
• Use VR as a new interface to electronic musical
  instruments
• Original idea create a Virtual Reality drum kit
  interface with optical tracking input
• Why do this?
• Might provide interesting musical performance
• good method of testing/measuring optical Vicon
  tracking system
• Virtual interface doesnt take up space (many
  people dont have space for a real drum kit)

4
Research Questions

• Can the feeling of playing the drums be
  replicated with a VR interface?
• Is optical tracking suited (fast and accurate
  enough) for this task ?

5
Literature Review 1

• First initial literature review strived to find
  similar comparable projects
• Not much on the academic sector
• University of Twente in Netherlands had a similar
  project, but used different technology also no
  papers on that project
• Axel Mulder, Phd thesis, Simon Fraser University,
  Canada data glove / hand gestures to create
  sound
• Interdisciplinary Centre for Scientific Research
  in Music (ICSRiM) at University of Leeds have a
  research framework Music Via Motion that tries
  similar things, however again with different
  technology

6
Literature Review 1

• Related commercial and art projects
• Electronic Drum Kits
• Theremin, Rolands D-Beam controller, Alesis Air
  FX
• Eye Toy
• Soundbeam
• Myron W. Kruegers installations

7
(No Transcript)
8
Technology - MIDI

• Musical Instrument Digital Interface
• A protocol that allows electronic musical
  instruments to communicate with each other (or
  with a computer)
• A MIDI instrument (or computer) usually has a
  MIDI-in port to receive messages and a MIDI-out
  port to send messages
• MIDI protocol defines messages e.g. Note-On,
  Note-Off

9
Technology - Creating drum sounds electronically

• Can be easily done via Samplers, Drum Machines,
  Synthesizers, etc. (both hard- and software)
• For my project Using a free sampling software
  and a pre-defined drum kit (collection of drum
  sound samples) that can communicate with the
  application via MIDI

10
Technology optical tracking

• Vicon tracking / motion capture system consists
  of
• 5 infrared cameras
• Dedicated computer for pattern recognition
• Pattern recognition software

11
Technology optical tracking

• Objects that need to be tracked are equipped with
  reflective markers
• Objects need distinctive marker patterns
• By observing the marker pattern from the 5
  different perspectives, the objects position and
  orientation can be calculated
• Software has training mode to learn to recognize
  new objects

12
Tracked objects

• Shutter glasses
• Drum sticks
• Feet

13
Technology Display System

• Barco Trace Large Screen Display
• Back Projection Screen with 120 Hz refresh rate
  for 1280 x 1024 resolution
• Active Stereo with stereo shutter glasses

14
System Design
15
Implementation Steps

• Create 3D Models of Drum Kit with 3DS Max
• Import them scene graph
• Create interface with tracking system (glasses,
  hand, feet)
• Create MIDI connection between application and
  Software Sampler
• Implement collision detection between virtual
  drum stick positions and drum skins (collision
  triggers Midi out note, velocity of impact
  determines the volume of the sound)

16
Main Program Loop

• Get current position of sticks
• Calculate if a collision has occurred (also
  compare with previous position to avoid
  continuous sounds)
• If collision has occurred
• Calculate speed (again by comparing with the
  previous positions and their timestamps)
• Send a midi message (Note On) with Note number
  relating to the object that was hit and volume
  relating to the speed
• Update Scene (set new position of sticks)
• Render Scene
• Could be improved by seperating the collision
  detection/sound output and grpahics in different
  threads

17
Screenshot
18
Problems discovered

• Missing haptic feedback seemed to be a major
  drawback
• Some more research into haptic devices that could
  be used in this context, Spidar seems to be the
  only well suited device for drum sticks, but not
  available at our department

19
Change of Research Direction

• Since the haptic path was a dead end due to
  availability of hardware
• and I was ahead of schedule with the
  implementation..
• The implementation of two other virtual
  instruments was decided
• Virtual Piano to compare both tracking
  performance and haptics with drums
• Virtual Synth Control (Theremin like experimental
  instrument) to compare replications of
  traditional instruments with experimental
  instrument

20
Virtual Piano

• Changes
• New controller gloves
• Different MIDI sound modulefree software piano
• Piano model generated by the application and not
  by modelling package
• Different collision detection algorithm (need
  note-on and note-off events)

21
Screenshot
22
Virtual Synth Control (VSC)

• Design objectives
• Easy to use
• No haptics required
• Psychedelic look and sound
• Input mode drum sticks
• Sound module used Free Software synthesizer JX
  220

23
Screenshot
24
Evaluation

• Main Purpose was to compare the users reception
  of the 3 instruments.
• 12 subjects
• Multiple choice questions
• Which instrument did you like most?
• Which of the instruments, if any, could be best
  used for making real music?
• Majority chose VSC for both questions (10/12 and
  11/12)
• No-one chose none so everyone could see some
  potential in the technology

25
Evaluation

• Another set of questions was aimed at evaluating
  all instruments individually on a scale of 1 to 7
• Some findings
• No depth perception problems with any of the
  instruments (question 1)
• Lack of haptics is severe, even more so for the
  piano (question 2)
• People found it hard to drum to a rhythm / easy
  to play VSC along with backing track (question 3)

26
3 ½ years later
27
Wii Piano

• Basic idea use the WiiMotes infrared camera for
  very cheap finger tracking
• 2D
• Compare with other input methods (e.g. mouse,
  touch screen)
• Applications Drawing, Piano

28
WiiMote

• Besides the usual buttons the WiiMote has 2 novel
  forms of input
• Accelerometers
• Infrared Camera

29
WiiMote Finger tracking

• Normally the infrared camera is used to evaluate
  the WiiMotes position relative to the Wiis
  sensor bar
• For WiiMote Finger tracking, the WiiMotes camera
  is used other way round
• WiiMote doesnt move
• Infrared Light source moves

30
WiiMote Finger tracking

• No active light source on finger is used but
  reflection (see passive markers in mocap)

31
WiiMote Finger tracking
32
User Interface Button Activation