KnoteBox

1
KnoteBox

• Joe Kramer, Leo Ovanesyan, Jimmy Thompon

2
Project Objectives
A4 B7 F5 G3 A6
KnoteBox
MIDI
3
Project Objectives

• Recognize musical notes from an analog sound
  input in real-time and output MIDI data to a
  computer
• Why use the MIDI protocol?
• It contains the information needed to display
  notes
• It is standard in the music industry
• Software already exists to manipulate MIDI data

4
Project Motivation

• Learning a new song or instrument can be
  irritating and discouraging
• Music-based video games can be used for learning
  music but have simplified controllers

• A music-based video game using real instruments
  would be enjoyable and instructional

5
Project Motivation

• There are systems which detect notes in audio
  signals
• LittleBigStar attempts to recognize chords but
  does not have much accuracy
• The KnoteBox is different and better because it
  will attempt to detect multiple notes at the same
  time, do it in real time, and with more accuracy

KnoteBox gt Current Technology
6
Applications

• Guitar Hero / Rockband clone
• Use MIDI input to create a track that must be
  played as notes scroll by
• Learning / Teaching capabilities
• Entertainment Game

7
Applications

• Sheet music generator
• Play into the system in order to get sheet music
  for the song played
• Can also generate a MIDI file

8
Applications

• Tuner
• Ability to show frequency information or MIDI
  data to tune a specific instument
• Instrument / Song learning tool
• Ability to show what note is actually being
  played

9
Overview

• Analog audio signal in
• Single instrument
• Polyphonic
• Melodic
• MIDI over USB out
• Buy MIDI to USB chip
• MIDI note on-gtnote off
• Box I/O
• Depends on algorithm

I/O
10
MIDI Protocol

• Industry standard protocol defined in 1982
• Defines interface between electronic instruments
  and file format
• MIDI events
• Note on with velocity
• Note off
• Aftertouch, pitch bend
• Pitch ranges from 0-127 (12 octaves)

MIDI Note on Pitch, velocity
MIDI Note off Pitch
11
Music Theory

• Pitch
• Fundamental frequency determines note being
  played
• Musical scale is logarithmically spaced (equal
  temperament)
• Linearly spaced overtones
• Envelope
• Time based attack, sustain, decay, release
• Timbre
• Differences in sound of voices or instruments
• Determined by differences in overtones and
  envelope

12
Frequency Analysis

• Constraints
• Delay
• Resolution
• Computation
• Time-frequency Algorithms
• Resonators
• Short form Fourier Transform
• Wavelet analysis
• Bilinear frequency analysis
• Q-constant and Tonal Centroid

13
Harmonic Peak Analysis

• Pick peaks from time-frequency image
• Thresholding, noise canceling
• Decreasing power at high notes
• Form tracks of peaks over time

• Match peaks to a combination of notes
• Trigger note on when peaks rise together
• Keep state of notes being played

14
Software
DSP / FPGA
MCU
15
Hardware
Power
DSP/ FPGA
MCU
MIDI driver chip
ADC/ filters
Mini USB out
Audio jack
JTAG
Power supply
Power
16
Hardware front end

• Audio jack
• 6.5mm connector
• Typical for electrical guitars and professional
  audio

17
Hardware pre-digital processing

• ADC chip
• Must have sampling rate of about 44.1KHz
• Standard audio CD quality
• Filter chip or RC circuit
• Anti-aliasing low-pass filter
• Amplifiers
• Op-amp BW up to 44.1KHz

18
Hardware for heavy processing

• DSP
• Hardware works, just need software
• Difficult to parallelize solution
• Limited to instruction set
• FPGA
• Solution can be parallelized in hardware
• Hardware needs to be verified

19
Hardware for heavy processing

• DSP
• TI DaVinci video processor
• DSP and MCU
• Acceleration hardware and lots of RAM
• FPGA
• Xilinx Virtex
• High performance FPGAs

20
Hardware for post-processing

• Microcontroller
• MSP430
• Cheap and popular (lots of example code)
• Low power (not an issue in the application)
• ARM
• Used in cell phones and other demanding
  applications
• 100MHz-1GHz range

21
Hardware output

• Ploytech GM5 chip
• Provides a MIDI driver through a USB interface
• Computer sees a MIDI device
• Hirose mini USB connector

22
Budget for development
Item Approximate cost
Two PCBs 130
DSP / FPGA 200 (x2?)
Microcontroller lt50 (x2?)
Amplifiers, filters, ADCs lt50
USB to MIDI chip 10
Miscellaneous 100
Total 540-790
23
Budget for production
Item Approximate cost
One PCB 50
DSP / FPGA 150
Microcontroller lt50
Amplifiers, filters, ADCs lt50
USB to MIDI chip 5
Casing 30
Total 335
24
Schedule
25
Division of Labor
Task Joe Leo Jimmy
Feature Extraction x x x
Pattern Recognition x x x
PCB Design X x
FPGA Prototyping x X
Microcontroller X x
PC Software x x x
26
Risks and Mitigations

• Inaccuracy with picking out notes and peaks
• Perform Matlab testing at the beginning to find
  and verify the best method
• Perform FPGA testing on a test board before
  building our own hardware
• Speed (Speed vs. Accuracy)
• Consider parallelizing the system
• Store information instead of having the system be
  real-time

27
Risks and Mitigations

• Noisy signal (especially at Expo)
• Have filters in place to remove the noise
• Consider having several microphones to create a
  noise array for noise cancellation
• Different instruments have different sound
  signatures
• Have different algorithms in place that can be
  used depending on the instrument
• Have a switch on the box to select what sound
  type the signal is

28
Risks and Mitigations

• Time Constraint
• Plan time accordingly
• Stick to the schedule
• Dont get funding / Need more funds
• Do the URAP application right the first time
• Find a local sponsor
• Get student discounts
• Look for other research grants

29
Risks and Mitigations

• Problems or errors with PCB design and population
  
• Fix mistakes and order new iteration
• Make room for an error in our timeline
• Do it right the first time
• Shipping delay / error
• Order parts early
• Ensure the part is available

30
Risks and Mitigations

• Leo gets hits by a bus
• Ol well, we dont need him
• Recruit from another group
• Jimmy or Joe die / have an emergency
• Triangle of stability
• Have at least two people working on all the
  different parts

31
Questions / Comments