SoundSense: Scalable Sound Sensing for People-Centric Application on Mobile Phones

1
SoundSense Scalable Sound Sensing for
People-Centric Application on Mobile Phones

• Hon Lu, Wei Pan, Nocholas D. lane, Tanzeem
  Choudhury and Andrew T. Campbell
• Department of Computer Science, Dartmouth College

2
Motivation

• Utilizing the microphone sensor to detect
  personalized sound events.
• Sound captured by mobile phones microphone is a
  rich source of information for surrounding
  environment, social environment, conversation,
  activity, location, dietary etc.

3
What is SoundSense?

• Scalable Sound Sensing Framework Capable of
  identifying any meaningful sound events of a
  users daily life.
• Implemented for resource limited devices, Apple
  iPhone.
• System solely runs in mobile phone

4
Contribution

• First general purpose sound event classification
  system designed for large number of events.
• Able to address significant sound events for
  individual users environment
• Implemented the whole system architecture and
  algorithm in Apple iPhone

5
Design Consideration

• Building a scalable sound classification system
  so that it can detect all type of sound events
  for different users.
• Privacy Issue Record and Processing audio data
  happens all in the Mobile phone.
• Light weight signal processing and classification
  of sound.

6
Design Consideration

• Phone context condition
• RMS good approximation
• of volume.
• 30 range of variation for
• different contextual position.

7
SoundSense Architecture
Remove Frames that are silent or hard to classify
8
SoundSense Architecture
1. Collect features that are insensitive to
volume. 2. Detect coarse-grain category of
sound Voice, music and ambient sound. 3.
Multilevel Classification Decision Tree and
Markov Model based classifier. 4. Two level of
classification to make the output smoothing.
9
SoundSense Architecture
1. Use previously established audio signal
processing technique 2. In this stage speech
recognition, speaker identification and music
genre classification is applied
10
SoundSense Architecture
1. Detect only ambient sound (sound other then
voice and music) 2.Unsuprvised learning
technique 3. Detect meaningful ambient sound. (
assumption sound occurrence and duration
indicates its importance) 4. Maintain a
SoundRank ranking of the meaningful sound based
on their importance 5. Prompt user, if a new
sound exceed the threshold value of minimum sound
rank.
11
Implementation

• Implemented in C,C and Objective C
• Developed for Apple iPhone
• Duty cycle 0.64 second during lack of acoustic
  event

12
Parameters Selection
Decision tree Classifier
Buffered in FIFO queue
Markov model classifier

• Increasing the buffer size (Sequence Length)
  increase the accuracy.
• However, responsiveness of the system also
  increases.
• Optimal buffer size is 5.

13
Parameters Selection
Precision is the number of frames that are
correctly classified divided by all frames.
Recall is define as the recognized occurrence of
a frame type divided by the number of overall
occurrence of that frame
MFCC frame length
This Precision and Recall plot is for ambient
sound
14
Evaluation
1. When acoustic event detected CPU usage
increase to 25. In idle situation CPU usage is
less then 5 2. Processing time of a frame (64
ms) is around 20-30ms.
15
Evaluation
Only Decision Tree Classifier
Classification accuracy improved 10 for music
and speech and 3 for ambient sound
Only Decision Tree Classifier With Markov model
16
Evaluation
No reliable sound to represent bus riding
17
Applications

• Audio Daily Diary Log everyday events for a
  users.
• To make query, how much time spend in certain
  event
• Music Detector based on Participatory Sensing
• Provides user a way to discover event that are
  associated with music being played.

18
Some music and voice samples are incorrectly
classified as ambient sound
Friday
Saturday
19
Conclusion

• General Sound Classification
• Light-weight
• Hierarchical
• Flexible and Scalable.
• All task implemented in mobile Phone.
• Able to identify new sound.
• Can be used in personalized context.

20
Thank you

• Question?