Smartphone Systems as Sensing Systems

1
Design and Implementation of Smartphone-based
Systems and Networking
Dong Xuan Department of Computer Science and
Engineering The Ohio State University, USA
2
Outline

• Smartphones Basics
• Mobile Social Networks
• E-Commerce
• E-Health
• Safety Monitoring
• Future Research Directions

2
3
Smartphone Basics

• A smartphone is a mobile phone offering advanced
  capabilities, often with PC-like functionality
• Hardware (Apple iPhone 3GS as an example)
• CPU at 600MHz, 256MB of RAM
• 16GB or 32GB of flash ROM
• Wireless 3G/2G, WiFi, Bluetooth
• Sensors camera, acceleration, proximity, light
• Functionalities
• Communication
• News Information
• Socializing
• Gaming
• Schedule Management etc.

3
4
Smartphone Popularity

• Smartphones are popular and will become more
  popular

4
5
Smartphone Accessories
5
6
Smartphone Features

• Communication/Sensing/Computation
• Inseparable from our human life

6
7
Our Smartphone Systems

• E-SmallTalker IEEE ICDCS10
• senses information published by Bluetooth to
  help potential friends find each other (written
  in Java)
• E-Shadow IEEE ICDCS11 enables rich local
  social interactions with local profiles and
  mobile phone based local social networking tools
• P3-Coupon IEEE Percom11 automatically
  distributes electronic coupons based on an
  probabilistic forwarding algorithm

7
8
Our Smartphone Systems

• Drunk Driving Detection Per-Health10 uses
  smartphone (Google G1) accelerometer and
  orientation sensor to detect
• Stealthy Video Capturer ACM WiSec09 secretly
  senses its environment and records video via
  smartphone camera and sends it to a third party
  (Windows Mobile application)

Download Run
Video sent by Email
Captured Video
8
9
Exemplary System IE-SmallTaker

• Small Talk
• A Naïve Approach
• Challenges
• System Design
• Implementation and Experiments
• Remarks

9
10
Small Talk

• People come into contact opportunistically
• Face-to-face interaction
• Crucial to people's social networking
• Immediate non-verbal communication
• Helps people get to know each other
• Provides the best opportunity to expand social
  network
• Small talk is an important social lubricant
• Difficult to identify significant topics
• Superficial

10
11
A Naive Approach of Smartphone-based Small Talk

• Store all users information, including each
  users full contact list
• User report either his own geo-location or a
  collection of phone IDs in his physical proximity
  to the server using internet connection or SMS
• Server performs profile matching, finds out small
  talk topics (mutual contact, common interests,
  etc.)
• Results are pushed to or retrieved by users

11
12
However

• Require costly data services (phones internet
  connection, SMS)
• Require report and store sensitive personal
  information in 3rd party
• Trusted server may not exist
• Server is a bottleneck, single point of failure,
  target of attack

12
13
E-SmallTalker A Fully Distributed Approach

• No Internet connection required
• No trusted 3rd party
• No centralized server
• Information stored locally on mobile phones
• Original personal data never leaves a users
  phone
• Communication only happens in physical proximity

13
14
Two Challenges

• How to exchange information without establishing
  a Bluetooth connection
• Available data communication channels on mobile
  phones
• Cellular network (internet, SMS, MMS), Bluetooth,
  WiFi, IrDA
• Bluetooth is a natural choice
• Bluetooth connection needs users interaction due
  to security reasons
• How to find out common topics while preserving
  users privacy
• No pre-shared secret for strangers
• Bluetooth Service Discovery Protocol can only
  transfer limited service information

14
15
System Architecture

• Context exchange
• Context encoding and matching
• Context data store
• User Interface

15
16
Context Exchange

• Exploit Bluetooth service discovery protocol
• No Bluetooth connection needed
• Publish encoded contact data (non-service
  related) as (virtual) service attributes
• Limited size and number( e.g. 128 bytes max each
  attribute)

16
17
Context Encoding

• Example of Alices Bloom filter
• Alice has multiple contacts, such as Bob, Tom,
  etc.
• Encode contact strings, Firstname.lastname_at_phone_n
  umber, such as Bob.Johnson_at_5555555555 and
  Tom.Mattix_at_6141234567

17
18
Implementation

• J2ME
• about 40 java classes, 127Kb jar file
• On real phones
• Sony Ericsson (W810i), Nokia (5610xm, 6650, N70,
  N75, N82)

18
19
Experiments

• Settings
• 6 phones, n150, k7, m1024 bits, default
  distance4m, average of 10 runs
• Performance Metrics
• Discovery time the period from the time of
  starting a search to the time of finding someone
  with common interest, if there is any
• Discovery rate percentage of successful
  discoveries among all attempts
• Power consumption
• Factors
• Bluetooth search interval
• Number of users
• Distance

19
20
Experiment Results

• Minimum, average and maximum discovery time are
  13.39, 20.04 and 59.11 seconds respectively
• Always success if repeat searching, 90 overall
  if only search once
• Nokia N82 last 29 hours when discovery interval
  is 60 seconds

20
21
Related Work

• Social network applications on mobile phones
• Social Serendipity
• Centralized, Bluetooth MAC and profile matching,
  SMS, strangers
• PeopleTones, Hummingbird, Just-for-Us, MobiLuck,
  P3 Systems, Micro-Blog, and Loopt
• Centralized, GPS location matching, Internet,
  existing friends
• Nokia Sensor and PeopleNet
• Distributed, profile, Bluetooth / Wifi
  connection, existing friends
• Private matching and set intersection protocols
• Homomorphic encryption based
• Too much computation and message overhead for
  mobile phone
• Limitations
• Require costly data services (phones internet
  connection, SMS)
• Require report and store sensitive personal
  information
• Bottleneck, single point of failure, target of
  attack

21
22
Remarks

• Propose, design, implement and evaluate the
  E-SmallTalker system which helps strangers
  initialize a conversation
• Leveraged Bluetooth SDP to exchange these topics
  without establishing a connection
• Customized service attributes to publish
  non-service related information.
• Proposed a new iterative commonality discovery
  protocol based on Bloom filters that encodes
  topics to fit in SDP attributes to achieve a low
  false positive rate

22
23
Exemplary System IIE-Shadow

• Concept
• Application Scenario
• Goals and Challenges
• System Design
• Implementation and Experiments
• Remarks

23
24
Concept

• Motivation
• Importance of Face-to-Face Interaction
• Prevalence of mobile phones
• Distributed mobile phone-based local social
  networking system
• Local profiles
• Mobile phone based local social interaction tools

25
Application Scenario Conference
26
Goals and Challenges

• Design Goals
• Far-reaching and Unobtrusive
• Privacy and Security
• Auxiliary Support for Further Interactions
• Broad Adoption
• Challenges
• Lack of Communication Support
• Power and Computation Limitation
• Non-pervasive Localization Service

27
Layered Publishing

• Spatial Layering
• WiFi SSID
• at least 40-50 meters, 32 Bytes
• Bluetooth Device (BTD) Name
• 20 meters, 2k Bytes
• Bluetooth Service (BTS) Name
• 10 meters, 1k Bytes
• Temporal Layering
• For people being together long or repeatedly
• Erasure Code

28
E-Shadow Publishing Procedure
29
Matching E-Shadow with its Owner

• Intuitive Approach Localization
• However, imprecision beyond 20-25 meters

30
Human Direction-driven Localization

• Direction more important than distance
• Human observation
• A new range-free localization technique
• RSSI comparison Less prone to errors
• Space partitioning Tailored for direction
  decision

31
Walking Route and Localization

• We allow users to walk a distance
• Triangular route A-gtB-gtC in (a), for
  illustration purposes
• Semi-octogonal route A-gtB-gtC-gtD-gtE in (c), more
  natural
• Take measurements on turning points
• Calculate the direction through RSSI comparison
  and space partitioning

32
Implementation

• Information Publishing Module
• Database
• Generator
• Buffers
• Control Valve
• Broadcasting Interfaces
• Retrieval Matching Module
• Receivers
• Localization
• Decoding Storage
• Sensing Module
• User Interface

33
Evaluations (1)-Time Energy

• E-Shadow Collection Time
• WiFi SSID 2 seconds
• BTD 12-18 seconds
• BTS 25-35 seconds
• E-Shadow Power Consumption
• 3 hours in full performance operation
• gt12 hours in typical situation

34
Evaluations (2)-Localization
3 Outdoor Experiments Open field campus
2 Indoor Experiments Large classroom
35
Evaluation (3)-Simulations
Large-Scale Simulations Angle deviation CDFs
12 times of exemplary direction decisions
36
Related Work

• Centralized mobile phones applications
• Social Serendipity
• Centralized, Bluetooth MAC and profile matching,
  SMS, strangers
• Decentralized mobile phone applications
• Nokia Sensor
• Distributed, profile, Bluetooth / Wifi
  connection, existing friends
• E-Smalltalker
• Distributed, no Bluetooth / Wifi connection,
  strangers
• Localization techniques for mobile phones
  applications
• GPS
• Virtual Compass
• peer-based relative positioning system using
  Wi-Fi and Bluetooth radios
• Limitations
• Privacy compromise
• Unable to capture the dynamics of surroundings
• No mapping between electronic ID and human face
• Localization techniques either not pervasive or
  not accurate for long range

36
37
Remarks

• Propose, design, implement and evaluate the
  E-Shadow system which lubricates local social
  interactions
• E-Shadow concept
• Layered publishing to capture the dynamics of
  surroundings
• Human-assisted matching that works for mapping
  E-Shadow with its owner in a fairly large
  distance
• Implementing and evaluating E-Shadow on real
  world mobile phones

37
38
Exemplary System IIIP3-Coupon

• Coupon Distribution
• A Naïve Approach
• Challenges
• System Design
• Implementation and Experiments
• Remarks

38
39
Electronic Coupon Distribution

• Electronic coupons
• Similar to paper coupons
• Can be stored on mobile phones
• Two distribution methods
• Downloading from Internet websites
• Need to define target group
• Limited coverage
• Hard to maintain dynamic preferences lists on
  central databases
• Peer to Peer Distribution
• No special destination/target group
• More coverage
• More flexible user-maintained preferences list

39
40
A Naive Approach of Peer-to-Peer Coupon
Distribution

• A store periodically broadcast the coupon
• Users within broadcast range receive the coupon
• User can decide whether to use, forward or
  discard the coupon
• Users forward the coupon to others in physical
  proximity
• Forwarders IDs are recorded in a dynamically
  expanding list
• The coupon is used by some user
• The store reward all users who have forwarded the
  coupon

40
41
However

• Require manually establishing wireless
  connections
• Cumbersome
• Not prompt
• Not possible for coupon forwarding among
  strangers
• Require recording the entire forwarding path
• Potential privacy leakage
• Discourage users forwarding incentives

41
42
Challenge

• How to design a prompt coupon distribution
  mechanism that
• Incentivize coupon forwarder appropriately for
  keeping the coupons circulating
• Preserve the privacy of coupon forwarders

42
43
P3-Coupon A Probabilistic Coupon Forwarding
Approach

• Probabilistic sampling on forwarding path
• Keep only one forwarder for each coupon NO
  privacy leakage
• Probabilistically flip ownership at each hop
• Accurate approximation of coupon rewards
• plenty of chances of interpersonal encounters
• Accurate bonus distribution with 50 coupons and
  5000 people
• Adaptive to different promotion strategies
• Flip-once model
• Always-flip model
• No manual connection establishment
• Connectionless information exchange via Bluetooth
  SDP

43
44
System Architecture

• Store Side
• A central server for broadcasting and redeeming
  coupons
• Client side
• Coupon forwarding manager, coupon exchange,
  coupon data store, user interface

44
45
Probabilistic Forwarding Algorithm

• Always-Flip Model
• The coupon ownership keeps flipping with certain
  probability at each hop.
• Good at assigning relative bonuses affected by
  the whole path lengths
• E.g. the parent forwarder receives k times the
  bonus given to children forwarders
• The flip probability can be calculated in advance
  by the store, once k is fixed, using the
  following formula

45
46
Probabilistic Forwarding Algorithm

• Extension Flip-Once Model
• Once flipped, a coupons ownership remain the
  same in a forwarding path.
• Good at assigning absolute bonuses irrelevant of
  the number of following forwarders
• E.g. hop 1 user gets 10, hop 2 user gets 5,
  etc.
• The flip probability can be calculated in advance
  by the store using the following formula

46
47
Coupon Format

• Coupon description
• Product description
• Discounts
• Coupon issuer
• Coupon code
• Start/end date
• Coupon forwarder information
• The current owner
• Digital signature
• Prevent forging fraud coupons

47
48
Implementation

• J2ME
• about 17 java classes, 1390Kb jar file
• On real phones
• Samsung (SGH-i550), Nokia (N82, 6650, N71x)

48
49
Experiments

• Experimental evaluations
• Coupon forwarding time
• Power consumption
• Simulation evaluation
• Number of Coupon holders vs. Time
• Distribution saturation time vs. Number of Seeds
• Coupon ownership distribution for probabilistic
  sampling
• Deviation between theoretical and actual bonus
  (Always-Flip, Flip-Once)
• Factors
• Number of coupons
• Number of users
• Number of initial coupon holders

49
50
Experiment Results

• Average coupon forwarding time is 33.52 seconds
• Nokia N82 last 25 hours with P3-Coupon running in
  background
• One coupon could be delivered to 5000 people
  within 32 hours
• Very small deviation between theoretical and
  actual bonus distribution with 50 coupons
  circulating among 5000 people

50
51
Remarks

• Propose, design, implement and evaluate the
  P3-Coupon system which helps prompt and privacy
  preserving coupon distribution
• Probabilistic one-ownership coupon forwarding
  algorithm
• Implement the system on various types of mobile
  phones
• Extensive experiments and evaluations show that
  our approach accurately approximate the
  theoretical coupon distribution in which the
  whole forwarding path needs to be recorded
• Practical for real-world deployment

51
52
52
Exemplary System IV Drunk Driving Detection

• Motivation
• Our Contributions
• Detection Criteria
• Our System
• Related Work
• Implementation and Evaluation
• Remarks

52
53
53
Motivation

• Crashes caused by alcohol-impaired driving pose a
  serious danger to the general public safety and
  health
• 13,041 and 11,773 driving fatalities happened in
  2007 and 2008
• 32 of the total fatalities in these two years
• Drunk driving also imposes a heavy financial
  burden on the whole society
• Annual cost of alcohol-related crashes totals
  more than 51 billion
• Data from U.S. NHTSA (National Highway Traffic
  Safety Administration)
• Data from U.S. CDC (Central of Disease Control)

53
54
54
Motivation

• Detection of drunk driving so far still relies on
  visual observation by patrol officers
• Drunk drivers usually make certain types of
  dangerous maneuvers
• NHTSA researchers identify cues of typical drunk
  driving behavior
• Visual observation is insufficient to prevent
  drunk driving
• The number of patrol officers is far from enough
• The guidelines are only descriptive and
  qualitative
• Usually, it is too late when drunk drivers are
  stopped by officers
• It is essential to develop systems actively
  monitoring drunk driving and to prevent accidents

54
55
55
Our Contributions

• Propose utilizing mobile phones as a platform for
  active drunk driving detection system
• Design a real-time algorithm for drunk driving
  detection system using mobile phones
• Simple sensors required only
• i.e., accelerometers and orientation sensors
• Design and implement a mobile phone-based active
  drunk driving detection system
• Reliable, Non-intrusive, Lightweight and power
  efficient, and No extra hardware and service cost

55
56
56
Cues for Drunk Driving Detection

• Cues related to lane position maintenance
  problems
• E.g., weaving, drifting, swerving and turning
  with a wide radius
• Cues related to speed control problems
• E.g., accelerating or decelerating suddenly, and
  braking erratically
• Cues related to judgment and vigilance problems
• E.g., driving with tires on lane marker, slow
  response to traffic signals

56
57
57
Drunk Driving Detection Criteria

• Extract fundamental detection criteria from these
  cues
• Capture the acceleration features
• E.g., for the lane position maintenance problems

57
58
58
Drunk Driving Detection Criteria

• Focus on the first two categories of cues
• They correspond to higher probabilities of drunk
  driving
• Map them into patterns of acceleration
• Probability of drunk driving detection goes
  higher while the number of observed cues increases

58
59
59
Our System
59
60
60
Implementation

• Develop the prototype system on Android G1 phone
  with accelerometer and orientation sensor
• Implement the prototype in Java, with Eclipse and
  Android 1.6 SDK
• The whole prototype system can be divided into
  five major components
• ? User interface ? System configuration ?
  Monitoring daemon
• ? Data processing ? Alert notification

60
61
61
Evaluation - Testing Data Collection

• Test data
• 72 sets of data with simulated drunk driving
  related behaviors
• Weaving, swerving, turning with a wide radius
• Changing speed erratically (accelerating or
  decelerating)
• 22 sets of data for regular driving
• Each one for 5 to 10 minutes
• Mobile phone positions in the vehicle

61
62
62
Evaluation - Detection Performance

• Study the accuracy of detecting drunk driving
  related behaviors
• In terms of false negative and false positive
• Study performance in the special case, such as
  the phone slides in the vehicle during driving
• Slides has obvious impacts on detection accuracy
• May add additional calibration procedure to solve
  it (future work)

62
63
63
Evaluation Energy Efficiency

• Curves of battery level states during mobile
  phone running
• Phone runs without drunk driving detection system
• Monitoring daemon of system keeps running,
  sensing and doing the pattern matching on the
  monitoring results

63
64
64
Related Work

• Driver vigilance monitoring and driver fatigue
  prevention
• Monitoring the visual cues of drivers to detect
  fatigue in driving
• Installed cameras just in front of drivers are
  potential safety hazard
• Monitoring through vehicle-human interface
• Capture fatigued or drunk driving through
  monitoring interactions
• Low compatibility, vehicles need to couple with
  auxiliary add-ons
• Detect abnormal driving through GPS and
  acceleration data
• Pattern matching with GPS and acceleration data
• However, GPS data are not always available

64
65
65
Remarks

• First to propose utilizing mobile phones as a
  platform for developing active drunk driving
  detection system
• Design and implement an efficient detection
  system based on mobile phone platforms
• Experimental results show our system achieves
  good detection performance and power efficiency
• In the future work, to improve the system with
  additional calibration procedure and by
  integrating all available sensing data on a
  mobile phone such as camera image

65
66
Exemplary System V Stealthy Video Capturer

• Background
• SVC Overview
• Challenges
• Our Approaches
• Experimental Evaluations
• Remarks

66
67
Background

• More and more private information is entrusted to
  our friend, the 3G Smartphone, which is getting
  more and more powerful in performance and
  diversified in functionality.

67
68
SVC Overview

• Almost every 3G Smartphone is equipped with a
  camera and the wireless options, such as 3G
  networks, BlueTooth, WiFi or IrDA.
• These wireless connections are good enough to
  handle certain types of video transmission.
• We turn 3G Smartphones into an online stealthy
  video-recorder.

68
69
System Architecture
69
70
Challenges

• Stealthily install SVC into 3G Smartphones
• Windows Hiding
• Infection Method
• Collect the video information from 3G Smartphones
• DirectShow Controls
• Data Compressing
• Send the video file to the SVC intender
• File Sending

70
71
Infection Method

• To embed SVC in a 3G Smartphone is called a
  infection process.
• We employ Trojan horse for downloads as the
  infection approach.
• Our experimental SVC is hidden in the game of
  tic-tac-toe that we develop in Windows Mobile
  environment.

71
72
The Scenario of Tic-Tac-Toe
72
73
Triggering Schemes

• Triggering Algorithm is designed to determine
  when to turn on the video capture process and
  send the captured video to make SVC stealthier
  and get more useful information.
• Three scenarios are under consideration.
• The first scenario is tracking.
• The second scenario is related with political or
  business espionage.
• The third scenario is a hybrid one, where SVC is
  used for much diversified everyday purposes.

73
74
Applications

• Suspects tracking
• Kids care

74
75
Kids tracking
75
76
Implementation
76
77
Experimental EvaluationsPower Consumption

• Power curve

77
78
Experimental EvaluationsCPU and Memory Usage

• CPU and Memory

78
79
Remarks

• The initial study exploited from SVC will draw
  wide attentions on 3G Smartphones privacy
  protection and open a new horizon on 3G
  Smartphones security research and applications.
• We are currently investigating the modeling of
  smart spyware from the study of spear and
  shield.

79
80
A Summary
80
81
Future Research Directions

• Smartphone-based Systems and Networking
• Mobile social networking, e-commerce, e-health,
  safety monitoring etc.
• Easy to start and exciting but too many
  competitors, lack of scientific depth
• Smartphone Core Improvement
• Multitasking, power management, efficient local
  communication protocol, accurate localization,
  security/privacy protection
• Deep but hard to start

81
82
Final Remarks

• Smartphones have brought significant impacts to
  our daily life.
• We present five exemplary systems on mobile
  social networking, e-commerce, e-health and
  safety.
• Research and development on smartphones will be
  hot.

82