Digital Sound as Computer Science

1
Digital Sound as Computer Science
Jennifer Burg

• CPATH Workshop Series
• Revitalizing Computer Science Education
• Through the Science of Digital Media
• Workshop 2, July 28 and 29, 2008
• Wake Forest University

This work was funded by National Science
Foundation CPATH grant CCF 0722261, Jennifer Burg
PI, Conrad Gleber Co-PI
2
National Science Foundation CPATH Grant

• Revitalizing Computer Science Education through
  the Science of Digital Media
• Jennifer Burg, PI, Wake Forest University
• Conrad Gleber, Co-PI, La Salle University
• Three years (Aug. 2007 July 2010), seven
  workshops
• Each workshop
• A special digital media topic
• One speaker from a related academic discipline
• One speaker from a related business or industry
• What would they like to see in a computer science
  major working for or with them?

3
Workshop Series
4
Digital Sound Production Workshop

• June 2 to July 25, 2008
• Students of music and computer science working
  together
• Interdisciplinary collaborative projects
• Funded by National Science Foundation CCLI grant
  Linking Science, Art, and Practice through
  Digital Sound, Jennifer Burg, PI Jason Romney,
  Co-PI

5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
(No Transcript)
9
Goals in this workshop from the PIs perspective

• Consider in what ways digital sound is
  legitimately part of the computer science
  curriculum
• Explore concepts, assignments, experiments, and
  exercises that are interesting to students
  because they bring together science, art, and
  practice
• Figure out where these can be plugged into the
  computer science curriculum

10
Goals in this workshop from the participants
perspective

• Consider how these ideas shed light on your own
  work
• Make contacts with colleagues who share your
  interests
• Eat well

11
What does the study of digital sound entail?

• Physics
• sound waves, acoustics, resonance
• Engineering and digital signal processing (DSP)
• Sound card, microphones, speakers, hardware sound
  processors, cables
• Mathematics
• Trigonometry, logarithms, complex numbers,
  summations, integrals, transforms
• Music
• Fundamental frequencies, harmonics, octaves
• Algorithms
• Transforms, filters, compression

12
What makes digital sound a suitable topic within
computer science?

• Its based on digital encoding and manipulation
  of digital data.
• Its applied computer science, which is what
  makes it interesting.

13
Topics in Digital Sound as Computer Science

• Sound waves and acoustics
• Analog vs. digital representations, digital
  encoding, sampling and quantization
• Decibels for measuring amplitude
• Frequencies related to pitch, complex waveforms
• Implications of sampling rate the Nyquist
  theorem and aliasing
• Implications of bit depth quantization error,
  SQNR, dynamic range, dithering, noise shaping,
  dynamic compression and expansion

14
Topics in Digital Sound as Computer Science

• MIDI compared to digital audio
• MIDI message formats and protocols
• MIDI samplers vs. synthesizers
• Sound wave synthesis

15
Topics in Digital Sound as Computer Science

• Hardware for sound processing
• Sound cards ADCs and DACs connection types
  cables microphones, speakers and monitors
  frequency response of microphones, speakers, and
  monitors
• Software for sound processing
• Audition, Audacity, Sound Forge, Logic, Pro
  Tools, Reason, Cakewalk Music Creator and Sonar,
  etc.
• MATLAB
• Chuck

16
Topics in Digital Sound as Computer Science

• Fourier analysis, frequency components, the
  Fourier transform, windowing functions
• Filters (FIR and IIR), EQ, types of filters
  (shelf, low-pass, high-pass, bandpass, bandstop)
• Special effects, e.g. reverb, autotuning,
  vocoding
• Data rate, data compression, psychoacoustical
  models for compression, frequency masking

17
As is true with most topics in computer science,
you can approach digital sound at different
levels of abstraction

• Mathematical/algorithmic pencil and paper,
  chalkboard, and calculator
• Low-level programming (e.g. C under Linux)
• Chuck
• MATLAB
• Audition, Audacity, Sound Forge, Logic, Pro
  Tools, Reason, various plugins, etc.
• Sample editor vs. track editor
• Combining digital audio and MIDI

18
Frequency Components of Sound Waves

• Generate notes C4, E4, and G4. Add the waves and
  look at the result.
• In Audition
• In MATLAB
• A single-frequency wave is a single pitch.
• Voices and instruments dont produce single
  pitches. They have harmonics.
• Sound in music and nature are complex waveforms
  with frequency components.

19
Notes, Octaves, and Frequencies

• Let f1 and f2 be the frequencies of two notes
  where the second is an octave above the first.
  Then f2 2f1.
• There are 12 notes in an octave. Find x such
  that
• f2 2f1 ((((((((((((f1x)x)x)x)x)x)x)x)
  x)x)x)x)
• 2f1 f1x12
• 2 x12
• x ?? 1.0595
• Thus if fa and fb are the frequencies of two
  consecutive notes, then fb 1.0595 fa.

20
Nyquist Theorem and Aliasing

• The sampling rate must be more than twice the
  frequency of the highest frequency component of
  the sound being sampled.
• Otherwise you can have aliasing.
• A frequency component comes out lower than it
  should be.
• Demonstrated in Audition

21
How is amplitude measured?
In Audition
22
The Effect of Bit Depth in Quantization

• Rounding to discrete quantization levels causes
  error. The error is itself a wave.
• In MATLAB
• Signal to quantization noise ratio (SQNR) and
  dynamic range are also measured in decibels.

23
Audio Dithering

• Add a random amount between -1 and 1 (scaled to
  the bit depth of the audio file) to each sample
  before quantizing.
• There will be fewer consecutive samples that
  round to the same amount. Rounding to 0 is the
  worst thing, causing breaks.
• Demonstration
• In Audition
• In MATLAB

24
Noise Shaping

• Raise error wave above the Nyquist frequency.
• Do this by making the error go up if it was
  previously down and down if it was previously up.
  This raises the error waves frequency.
• The amount added to a sample depends on the error
  in previous sample.

25
Digital Filters

• Infinite impulse response (IIR) vs. finite
  impulse response (FIR) filters
• Filtering in the time domain by means of
  convolution

Click to animate
26
Digital Filters

• Filtering in the frequency domain by means of the
  Fourier transform

27
Digital Filters
28
Creating Filters
frequency response graph
29
Creating Filters
frequency response graphs
30
Creating a Low-Pass Filter
Frequency Response (frequency domain)
Impulse Response (time Domain)
31
Creating a Low-Pass Filter
32
Creating a Low-Pass Filter

• You can do it yourself in MATLAB
• Create the filter using the given function,
  sin(2?fc)/?n
• Read in an audio clip
• Since this is a filter in the time domain,
  convolve audio clip with the filter
• Listen to the result
• Graph the frequencies of filtered clip against
  the unfiltered clip. (Do this by taking the
  Fourier transform of each first.)
• See the demonstration and worksheet for details.

33
Creating FIR and IIR Filters with MATLABs
Digital Signal Processing Toolbox

• gtgt lowA wavread('440.wav')
• gtgt highA wavread('880.wav')
• gtgt highest wavread('2000.wav')
• gtgt mixed (lowAhighAhighest) / 3
• gtgt a,b butter(6,1000/4000)
• gtgt output filter(a,b,mixed)
• gtgt ideal (lowAhighA) / 2
• gtgt wavplay(ideal,8000)
• gtgt wavplay(output,8000)
• gtgt hold on
• gtgt plot(ideal)
• gtgt plot(output,'red')
• gtgt axis(1 50 -1 1)
• Demonstration

34
Filter Visualization Tool in MATLAB

• MATLAB also has a Filter Visualization Tool that
  lets you set zeros and poles for a filter and see
  the frequency, phase, and impulse responses.

Demonstration
35
C Programs for Digital Sound and MIDI
Reading Messages from dev/midi

• include ltstdio.hgt
• include ltstring.hgt
• include ltlinux/soundcard.hgt
• include ltunistd.hgt
• include ltfcntl.hgt
• /CTRL-Break out of program/
• int main()
• char device00 "/dev/midi"
• unsigned char data3
• unsigned char byte1, byte2, byte3
• int fd
• fd open(device00, O_RDONLY, 0)
• if (fd lt 0)
• printf("Error cannot open s\n", device00)
• else printf("Opened dev/midi00\n")
• byte1 byte2 byte3 -1

while (1) read(fd, data, sizeof(data))
if (!(data0 byte1 data1 byte2
data2 byte3) printf("d ",
data0) printf("d ", data1)
printf("d\n", data2) byte1
data0 byte2 data1 byte3
data2 return 0
36

• include ltunistd.hgt
• include ltfcntl.hgt
• include ltsys/types.hgt
• include ltsys/ioctl.hgt
• include ltstdlib.hgt
• include ltstdio.hgt
• include ltlinux/soundcard.hgt
• define LENGTH 3
• define RATE 8000
• define SIZE 8
• define CHANNELS 1
• unsigned char buf4LENGTHRATESIZECHANNELS/8
• int main()
• int fd, arg, status
• int i, j, k, begin, end, bufEnd
• char temp
• printf("Size of buffer is d\n", sizeof(buf))
• fd open("/dev/dsp", O_RDWR, 0)
• if (fd lt 0)

Reading from and Writing to the Sound Card
37

• arg CHANNELS
• status ioctl(fd, SOUND_PCM_WRITE_CHANNELS,
  arg)
• if (status -1)
• perror("Unable to set number of
  channels\n")
• arg RATE
• status ioctl(fd, SOUND_PCM_WRITE_RATE,
  arg)
• if (status -1)
• perror("Unable to set sampling rate\n")
• status read(fd, buf, 1)
• while (buf0 gt 125 buf0 lt 131)
• status read(fd, buf, 1)
• printf("d ",buf0)
• printf("broke the silence\n")

Reading from and Writing to the Sound Card
38

• for (i 0 i lt 3 i)
• //printf("Say something\n")
• status read(fd, buf(24000i), 24000)
• if (status ! 24000)
• perror("Read wrong number of bytes\n")
• begin 24000i
• end begin 12000
• bufEnd begin 24000 - 1
• for (j begin, k 0 j lt end j, k)
• temp (bufj)
• (bufj) (buf bufEnd - k)
• (bufj) temp
• status ioctl(fd, SOUND_PCM_SYNC, 0)
• if (status -1)
• perror("SOUND_PCM_SYNC failed\n")
• printf("You said \n")
• status write(fd, buf, sizeof(buf))

Reading from and Writing to the Sound Card
39
Creating a Vocoder in MATLAB
From http//www.paia.com/ProdArticles/vocodwrk.htm
40
Creating a Vocoder in MATLAB
function output vocoder(input1, input2, s,
window) h hanning(window)' input1
input1' input2 input2' q(s-window) output
zeros(1,s) fftdata zeros(1,s) input1fft
zeros(1,s) input2fft zeros(1,s) for
i1window/4q b iwindow-1
input1partfft fft(input1(ib).h)
input2partfft fft(input2(ib).h)
input1fft(ib) input1fft(ib)
abs(input1partfft) input2fft(ib)
input2fft(ib) abs(input2partfft) mult
input1partfft.input2partfft fftdata(ib)
fftdata(ib) abs(fft(mult)) output(ib)
output(ib)ifft(mult) end output
output/max(output)
Demonstration
41
Questions for this workshop

• How do we relate the science to art and practice?
• How much science does the artist/practitioner
  need?
• Where are the points where knowing the science
  results in better work?
• How do we change the computer science curriculum
  to retain the science but relate it more
  interestingly to art and practice?