From TimeSync to EmStar: A Vertical Slice of Life in Wireless Sensor Networks

1
From TimeSync to EmStarA Vertical Slice of
Lifein Wireless Sensor Networks

• Jeremy Elson
• Department of Computer Science
• jelson_at_cs.ucla.edu
• http//google.com/search?qjeremyelson

March 2004
2
Outline

• Time Synchronization
• Why did we need something new?
• What are some new techniques we used?
• Some first implementations
• 3D, ad-hoc localization systems for DARPA
• Over several generations, what did we learn?
• EmStar Software for sensor networks
• Sub-problems like time sync are 5 of the work
• Put it all together, reactive to dynamics - HARD!

3
Does timesync matter?

• Internet Time Synchronization
• Critical in some contexts (e.g. crypto,
  distributed packet traces)
• A convenience in many other contexts
• Sensor Network Synchronization
• Fundamental to its purpose data fusion
• Physical time needed to relate events in the
  physical world

4
Time Sync can be the Limiting Factor
Localization Experiment
Courtesy of H. Wang, Prof K. Yao, et. al.
5
Isnt this solved?

• NTP (Network Time Protocol)
• Ubiquitous in the Internet
• Variants appearing in sensor networks
• 802.11 synchronization
• Precise clock agreement within a cluster
• GPS, WWVB, other radio time services
• High-stability oscillators (Rubidium, Cesium...)

6
So whats wrong?

• Existing work is a critical building block

BUT...

• This isnt the Internet
• Important assumptions no longer hold
• (fewer resources available for synchronization)
• Sensor apps have stronger requirements
• (but we have to do better than the Internet
  anyway)
• Energy, energy energy
• Listening to the network is no longer free even
  occasional CPU use can have a major impact

7
Infrastructure vs. Ad-Hoc

• NTP provides UTC to the entire Internet
• But wait, youre cheating! UTC is distributed to
  many places in the network out of band via
  various radio services (WWV, GPS, )
• Path from clients to a canonical clock is short
• Infrastructure isnt ubiquitous in sensor nets
• GPS doesnt work indoors, in the forest,
  underwater, on Mars
• What happens without infrastructure?

8
Poor Topologies
Master
Stratum 2
Stratum 3
A
C
?????
B
Should A or C serve as Bs master? Either
decision leads to poor sync with the other.
9
Mundane Reasons

• Cost
• We cant put a 500 Rubidium oscillator or a 50
  GPS receiver on a 5 sensor node
• Form factor
• Nodes are small, extra components are large
• Not actually a mundane limitation if it changes
  the economics of the sensor net

10
Traditional sync
Problem Many sources of unknown,
nondeterministic latency between timestamp and
its reception
Sender
Receiver
Send time
Receive Time
At the tone t1
NIC
NIC
Access Time
Propagation Time
Physical Media
11
Reference Broadcasts
Sync 2 receivers with each other, NOT sender with
receiver
Sender
Receiver
Receiver
Receive Time
NIC
NIC
NIC
I saw it at t4
I saw it at t5
Propagation Time
Physical Media
12
RBS changes the critical path
NIC
Sender

Receiver 1

Receiver 2
Critical Path
Traditional critical path From the time the
sender reads its clock, to when the receiver
reads its clock
RBS Only sensitive to the differences in receive
time and propagation delay
13
Receiver Determinism
1st testbed Berkeley motes with narrowband
(19.2K) radios
14
Time
15
Clock Resolution
6us avg
51us avg
1.5us avg
NTP also got much worse under heavy load RBS
didnt
16
Multi-Hop RBS

• Some nodes broadcast RF synchronization pulses
• Receivers in a neighborhood are synced by using
  the pulse as a time reference. (The pulse
  senders are not synced.)
• Nodes that hear both can relate the time bases to
  each other

Red pulse 2 secafter blue pulse!
Here 3 sec after red pulse!
Here 1 sec after blue pulse!
Here 1 sec afterred pulse!
Here 0 sec after blue pulse!
17
Time Routing
The physical topology can be easily converted to
a logical topology links represent possible
clock conversions
1
2
5
A
B
6
3
4
7
C
8
9
D
10
11
Use shortest path search to find a time
route Edges can be weighted by error estimates
18
Applications

• A few brief anecdotes

19
Acoustic Localization
20
Correlation

• Green is reference, Red is measured
• Signals aligned to offset yielding max correlation

Amplitude
Time in Samples (20 uS)
21
Commercial Version

• Sensoria Corp under contract from DARPA
• Goal Nodes localize themselves within 1m, MOVE
  to fill in gaps
• Network completely self-organizing, autonomous
• Uses RBS timesync for acoustic ranging

22
Commercial Version
10 MOBILE nodes, 1 is deactivated, and then...
23
It might seem simple...

• Just use timesync for acoustic ranging, build a
  map, react to it
• Would you believe the system had about 20
  application-specific daemons?

• Time synchronization
• Cluster multilateration
• Cluster map transmission/federation
• Healing with holdoff
• Rocket thruster device driver
• Network arm/disarm/safety
• Visualization proxy
• Command proxy (debugging)

• Process management
• Radio device drivers (x2)
• Fragmentation and reassembly (x2)
• Neighbor discovery (x2)
• Cluster formation/leader election
• Flood routing
• Acoustic ranging
• Acoustic range scheduling

Now add dynamics into the mix! The Science was
maybe 10 of the work
24
The EmStar Approach

• Allow software to be finely decomposed,
  modularized
• Toolchest of mechanism, not policy modules
• Provide rich forms of inter-module communication
• Isolate as much domain knowledge as possible
• Run-time environments for deep debugging
• Debug in a transparent context before the
  necessarily opaque deployment
• Same code runs in simulation, reality, and
  hybrids
• High visibility into the system is key -- status
  is exposed in both human- and machine-readable
  form

25
FUSD IPC

• Inter-module IPC FUSD
• Creates device file interfaces
• Semantics of system calls easily customized
• Text/Binary on same file
• Packet boundaries respected
• Standard interface
• Language independent
• No client library required

User
Kernel
26
Interacting With EmStar

• Text/Binary on same device file
• Text mode enables interaction from shell and
  scripts
• Binary mode enables easy programmatic access to
  data as C structures, etc.
• EmStar device patterns support multiple
  concurrent clients
• IPC channels used internally can be viewed
  concurrently for debugging
• Live state can be viewed in the shell (echocat
  w) or using emview
• Result designers can see the state of all IPC
  paths in a live system

27
A Slow Descent into Reality

• EmStar allows the same Linux code to be used
• In a pure (low-fidelity) simulation
• Mostly simulated, but using a real wireless
  channel
• In a real testbed, small-scale but
  high-visibility
• Deployed, in-situ, at scale -- but low
  visibility
• Advantage over traditional simulators the
  debugged code itself, not just the high-level
  concepts, flow from simulation into the real
  world
• To maintain high visibility, we trade scale for
  reality

28
EmStar Run-Time Environments
The spectrum allows high-visibility
debugging before jumping into low-visibility
deployment
29
The Ceiling Array A Real Wireless Channel
Motes used to transmit and receive packets -- A
real-world augmentation to a virtual simulation
30
Portable Array In-Situ, so Smaller Scale
Cables (green, invisible) attach to in-situ motes
31
Future (current) work

• A Wireless Seismic Testbed
• Seismology is a data-driven science, but data
  collection is often biased by GPS availability
• Real-time data only near cities (infrastructure)
• The Center for Embedded Networked Sensing is a
  collaboration of science and systems research
• Use EmStar and RBS to distribute time, collect
  data
• Result (ideally) A system that both pushes the
  boundaries of distributed systems and provides
  real data

32
Thank you!