Applications on TimeScale Modification of Music Synchronizing Tracks of Music the Conductor Program

1
Applications on Time-Scale Modification of Music
- Synchronizing Tracks of Music - the Conductor
Program

• Wu-Hsi Li
• Jengs Computer Music Group
• Department of EE, NTU
• Advisor Prof. Shyh-Kang Jeng

2
Outline

• Overview
• Time-scale Modification Algorithm
• Phase Vocoder
• Synchronize Tracks of Music
• Make your own concerto recording
• Music Player with Conducting Interface

3
Overview

• Two dreams
• I want to play a piano concerto with an orchestra
• I want to conduct an orchestra

4
Time-scale Modification Algorithm

• Goal
• Adjust tempo without changing other perceptual
  feature such as pitch or timbre
• Solution
• Phase Vocoder
• Flanagan and Golden in 1966
• Time Domain Harmonic Scaling (TDHS)
• Rabiner and Schafer in 1978
• Does not work well with polyphonic material
• Multiple Component Feature Extracting
• Prosoniq TimeFactory (commercial product)

5
Phase Vocoder

• STFT (short-time Fourier transform)
• Transform audio signal into time-frequency array
  through STFT
• Resample STFT
• Resample the time-frequency array according to
  the speed adjustment rate
• Fix the phase part to maintain phase-continuity
  between successive frames
• ISTFT
• Invert the time-frequency array into waveform

6
Phase Vocoder

• Phase vocoder is not a flawless technique
• smearing
• reverberation
• The artifacts are less audible while performing
  time-scaling on music than on speech signal

Music
Speech
7
From Time-scale Modification to Music Tempo
Adjustment

• Tempo adjustment on music
• Segment the music into pieces
• Each piece contain one or multiple beats
• Perform time-scale modification on each piece
• To track beat position in music
• Do it manually
• Let the user listen to the music and do the
  tapping, the program will record the position of
  each beat.

8
Make my own concerto recording

• Synchronize two tracks of music

divBi
Music B
time
Music A
divAi
Beat i1
Beat i
Speed Adjustment Rate of Beat i divBi / divAi

• Demo Chopin Concerto No.2 Mov.1 (part)

9
Music Player with Real-time Conducting Interface

• Input File
• music in wav format
• an attached file which specifies beat positions
  in the music
• Users can conduct the music player through moving
  the mouse cursor as a conductors baton

10
Music Player with Real-time Conducting Interface

• Music interacts with the user in real-time
• A mouse-cursor tracker reads the trajectory of
  cursor and interprets as tempo and volume
  information that the user conducts.
• A real-time phase vocoder modifies the music
  signal according to the tempo and volume
  information
• Demo I
• Johann Strauß II
• An der Schönen, Blauen Donau

11
Downbeat Tracking

• The tracker observes the mouse cursors
• the moving direction
• the average moving distance
• the instant speed
• In general, when the instant speed of the cursor
  increases rapidly along certain direction, the
  user is indicating a downbeat.

12
Tempo Adjustment

• As mentioned above, it is possible to perfectly
  synchronize tracks of music

Ratio divB / divA
divB
Music B
time
Music A
divA
13
Tempo Adjustment (cont.)

• Now, we try to synchronize music with a sequence
  of downbeats IN REAL TIME

The ratio cant be determined yet but the beat is
about to be played.
?
Downbeat
time
divB
Music
When the program can calculate the precise ratio,
its already too late.
divA
14
Tempo Adjustment (cont.)

• Freeze
• Freeze the music at certain point and wait for
  the next downbeat
• Jump
• Skip the music at certain beat when the next
  downbeat is detected
• Both functions make the music unnatural
• But they may be useful in some situations

15
Tempo Adjustment (cont.)

• Estimate the gap between the conductor and the
  music
• Try to eliminate the gap
• When the conductors downbeat is previous to the
  corresponding beat in the music, the music should
  speed up, and vice versa.
• Should we try to eliminate the gap ASAP?

16
Interaction between conductor and music
Look at the conductor
Listen to the orchestra
17
Interaction between conductor and music

• When the conductor speeds up suddenly
• The conducting tempo is higher than the music
  tempo
• Furthermore, it creates a time gap between the
  conductor and the music
• If the orchestra tries to eliminate the gap, the
  musics tempo must faster than the conducting
  tempo temporarily
• However, the conductor becomes aware that the
  music is faster than the tempo he/she gives
• Therefore, the timing of the next downbeat is
  affected
• The conducting tempo and the music tempo become
  unstable

18
Interaction between conductor and music

• It is more important for both the music and the
  conductor to reach a stable tempo than to
  eliminate the gap

19
Controllability of the Conductor

• When the controllability is high
• the orchestra follows the conductor tightly,
  which sometimes causes the music (and the
  conducting) tempo becomes extremely unstable
• An experienced conductor is more likely to
  control the music in high controllability mode
• When the controllability is low
• the orchestra follows the conductor loosely, the
  tempo will be stable even if the conductor
  commits mistakes (for ex., missing beats).
• And therefore, the music will be smooth for most
  of the time.

20
Other Features

• Can also use keyboard as conducting equipment
• Conducting by pressing the spacebar
• Preparatory Beat
• Determine the initial tempo of music
• Tutti Beat
• Specified in beat information file
• The music may waits until the conductor gives a
  downbeat

21
Demo

• Short Demo II
• about the use of Tutti Beat
• Demo III
• Johannes Brahms
• Hungarian Dance No.5
• Volunteer? (If there is still time)
• Johann Strauss
• Radetzky-Marsch