EmStar: a Software Environment for Developing and Deploying Wireless Sensor Networks

1
EmStar a Software Environment for Developingand
Deploying Wireless Sensor Networks

• Lewis Girod
• CENS Systems Lab
• Jeremy Elson, Alberto Cerpa, Thanos Stathopoulos,
  Nithya Ramanathan, Deborah Estrin

2
Moores Law and Microfabrication
Small, cheap, plentiful computing resources
SPEC (J. Hill) 4MHz/8bit, 3K/0K
Mica2Dot (Berkeley/Xbow) 8MHz/8bit, 4K/128K
Stargate (Intel/Xbow) 400Mhz/32bit, 64M/32M
Liquid Chromatograph (YC Tai)
iMEMS Accelerometer (Analog Devices)
Marine Algae Detector (C Zhao)
Small, cheap, plentiful sensing technologies
3
Embedded Networked Sensing (ENS)
NIMS
Seismic
The Self-Healing Minefield. Detects failures of
nodes that cause a breach in the field.
Reconfigures itself to maintain coverage of the
field.
Habitat investigation, using NIMS (Networked
Info-Mechanical Systems). NIMS is a robotic
system that rides on a cable above a forest
floor. It can visit sites in the forest and
interact with static nodes
Seismic detection, analysis arrays, in the CENS
Seismic Array, a proposed array of 50 nodes that
seeks to provide real-time recovery of seismic
data from fielded nodes
4
EmStar is

• A software system designed to support ENS apps
• Includes a collection of useful Tools and
  Services
• Runs on Linux, on PDA-class devices
• Integrates with applications on smaller embedded
  sensors
• Key Features Robustness and System Transparency

EmStar S/W
Animal Call Localizer
collab_detect
gradients
data
MicroDiffusion
sensor/frog
detect
link/ls0/neighbors
neighbors
sensor/audio/fft
Intel/Xbow Stargate 400MHz Xscale 64M RAM/32M
Flash Runs Linux
sync/hist
FFT
timehist
clients
emproxy
link/ls0
linkstats
sync/params
sensor/audio/0
audiod
syncd
status
link/udp0
emlog/
udpd
emrun
802.11
ADC
802.11 NIC
5
What makes ENS hard?

• Unexpected failures and new bugs in the field
• Fielded systems tend to degrade more quickly than
  in the lab
• Environmental conditions weather, animals, RF
  and sensor channel
• They also encounter problems that dont occur in
  the lab
• e.g. acoustic ranging system encountered new
  kinds of noise, leading to new kinds of
  inconsistencies in geometry, crashing NLLS alg
• EmStar is designed to make it easy to
• Deploy systems Leverage existing tools,
  services, and structure
• Debug running systems Transparency
• Keep running despite unexpected failures and
  bugs Robustness

6
Components of the EmStar System

• EmStar provides tools and services for building
  ENS applications
• Libraries and IPC Support
• FUSD IPC via device file interfaces
• Device Patterns Libraries that provide standard
  kinds of devices.
• Status Device
• Packet Device
• Services
• Link/Neighborhood estimation
• Time Synchronization
• Routing Flooding, Sink Tree, Diffusion
• Tools
• EmRun Manage running EmStar processes and
  collect logs
• EmSim/EmCee A real-code simulator that can
  support real radios
• EmView A visualization tool

7
FUSD

• Framework for User Space Devices (FUSD)
• Linux kernel module
• Proxies system calls on a chrdev to a user program

Device Registration
FUSD
Client
Server
fd open(/dev/fusd, O_RDWR) Write
registration message for /dev/servicename
Create chrdev /dev/servicename
Wait for messages on fd
8
FUSD Client Open
Client Open
FUSD
Client
Server
fd open(/dev/servicename, O_RDWR)
Lookup server for device Marshal open()
call Queue for server
fd is readable Read message and dispatch to
open handler. If open is accepted, create
client state. Marshal return message with return
value or error code
Wake client process, return from system call
System call returns new fd or error
9
FUSD Client Calls
Client System Call
FUSD
Client
Server
status write(fd, buf, size)
Marshal write() call Queue for server
fd is readable read message and dispatch to
write handler. Process the write. When the
write is completed, marshal a response message
containing the return value and error code and
write it back to FUSD.
Wake client process, return from syscall
System call returns

• System calls may be blocked by the Server, by
  delaying before sending a message back. System
  calls from other clients must be processed in the
  interim.
• The message from FUSD includes metadata about the
  calling process and file
• If the client dies, FUSD will issue close() calls
  for all of its open files. The Server must clean
  up any blocked system calls and other state.

10
FUSD Poll
Client Poll
FUSD
Client
Server
status poll(fds, 1, 0)
Check cached poll state. If no poll_diff pending,
issue poll_diff
Clean up any pending poll_diff message. Check
internal poll state vs. new poll_diff msg. If
different, respond with internal state, else
respond later when internal state changes
System call returns
Update cached state Issue a new poll_diff

• Poll must determine an answer immediately. To
  address this, FUSD caches the last known poll
  state for each open file, and maintains a
  poll_diff message, that the server can use to
  change that cached state when its internal state
  changes.
• In addition, whenever a read() or write()
  completes, FUSD assumes that the file is no
  longer readable or writable, and issues a new
  poll_diff to verify that.

11
Performance Considerations

• FUSD introduces significant system call latency

12
Cost is Negligible for Low-Rate Communication

• ENS systems use low-rate wireless interfaces
• Mote Radios (20 Kbit/sec, 60ms typ. packet
  latency)
• 802.11 (11 Mbit/sec)

802.11 (11Mbit/sec)
13
Components of the EmStar System

• EmStar provides tools and services for building
  ENS applications
• Libraries and IPC Support
• FUSD IPC via device file interfaces
• Device Patterns Libraries that provide standard
  kinds of devices.
• Status Device
• Packet Device
• Services
• Link/Neighborhood estimation
• Time Synchronization
• Routing Flooding, Sink Tree, Diffusion
• Tools
• EmRun Manage running EmStar processes and
  collect logs
• EmSim/EmCee A real-code simulator that can
  support real radios
• EmView A visualization tool

14
EmStar IPC and FUSD

• All IPC in EmStar uses FUSD devices
• Most modules are both clients and servers
• Provide one or more devices that other modules
  can use
• Use devices provided by underlying modules
• Ex. Neighbor Discovery
• Provides a status device (/dev/link/mote0/neighb
  ors) to provide a list of active neighbors to
  other modules
• Uses a link device (/dev/link/mote0/data) to
  send and receive ping packets via the Mote
  radio to probe for neighbors
• Device patterns define semantics of device files
• e.g. status device, packet device

15
Neighbord Provides a Status Device

• A Status Device is a way a server can expose some
  internal state to clients, and notify them when
  the state changes
• The neighbord module exposes the current state of
  its neighbors via the /dev/link/mote0/neighbors
  device
• Access from the shell is useful for debugging
• The programmatic usage is to open the device and
  use poll() to find out when the state changes

cat /dev/link/mote0/neighbors Node ID
Interface State Location Conn IN
Conn OUT Last HB Last Ack -------
---------------- ---------- --------------
------- -------- ------- -------- 1
0.0.60.1 self (43.84, 91.07) --
-- -- -- 2
0.0.20.2 asymmetric (41.36, 8.95) 100.00
-- 3.27 -- 5
0.0.108.5 asymmetric (16.27, 71.43) 100.00
-- 3.07 -- 4
0.0.124.4 active (13.69, 2.77) 100.00
-- 2.18 2.68 3
0.0.28.3 active (44.22, 75.93) 100.00
100.00 1.47 2.68
16
A Client Uses Neighbords Status Device
EmStar includes a client library for reading from
status devices, as well as a library for reading
from the neighbors service itself. To illustrate
more of the mechanism, we show how it is done
without using the library
/ open and set binary mode / int fd
open(/dev/link/mote0/neighbors, O_RDONLY) int
mode STATUS_MODE_BINARY ioctl(fd,
_IOW(STATUS_IOCTL_TYPE, STATUS_SETMODE, int),
mode) / ..Whenever fd is readable.. / if
(readable(fd)) buf_t buf buf_new() char
rd_buf4096 int status / Read until EOF
/ while ((status read(fd, rd_buf,
sizeof(rd_buf)) gt 0) buf_cpy(buf, rd_buf,
status) / check for read error / if
(status lt 0) elog(LOG_WARNING, Read error
from neighbors service m) / otherwise
process new neighbor list / else
neighbor_t n buf-gtbuf int n_count
buf-gtlen / sizeof(neighbor_t) / ..process
it!.. / buf_free(buf)
Open and set binary mode Whenever
readable Read until EOF (buf_t is an
expanding buffer frequently used in
EmStar) Check for errors Interpret the buffer
as a vector of neighbor_t structs Wait until
readable again
17
Filtering and Robustness Neighbord Status Service

• Soft state Neighbord service always reports full
  neighbor list
• Immune to loss of sync or dropped updates
• All clients see the same data you can use cat
  to see what your program sees
• Client library encapsulates reconnect logic
• Typically used by clients of neighbor devices
• Automatically reconnects if the device disappears
  (e.g. if neighbors service restarts)
• Filtering
• Rate limits updates to maximum of 1 per second
• Periodic refresh every 10 seconds
• Neighbor state hysteresis drop neighbor on 3
  dropped pings
• Filtering reduces cost of soft state approach

18
Components of the EmStar System

• EmStar provides tools and services for building
  ENS applications
• Libraries and IPC Support
• FUSD IPC via device file interfaces
• Device Patterns Libraries that provide standard
  kinds of devices.
• Status Device
• Packet Device
• Services
• Link/Neighborhood estimation
• Time Synchronization
• Routing Flooding, Sink Tree, Diffusion
• Tools
• EmRun Manage running EmStar processes and
  collect logs
• EmSim/EmCee A real-code simulator that can
  support real radios
• EmView A visualization tool

19
EmSim/EmCee

• EmSim is a real code simulation environment for
  EmStar
• EmSim runs N copies of an EmStar system on a
  single machine
• Since all IPC is through device files,
• Each node gets its own device space through name
  mangling, e.g. /dev/X ? /dev/sim/group5/node001/X
• Name mangling is done using a macro sim_path(),
  used by clients and servers whenever a device
  name is used.
• Simulation components provide interface to a
  simulated world
• sim_radio models an RF channel and MAC layer,
  provides a link device interface for each
  simulated node
• sim_sensor models or replays sensor data,
  providing a sensor device interface for each
  simulated node.
• EmCee uses real radios for communication
• Runs N copies of an EmStar system, connects each
  nodes link device to a real Mote radio connected
  by a serial multiplexor

20
EmStar Goals Revisited

• EmStar is designed to make it easy to
• Deploy systems Leverage existing tools,
  services, and structure
• Debug running systems Transparency
• Keep running despite unexpected failures and
  bugs Robustness

21
Transparency and Debugging

• Why is Transparency important?
• Reveals internal state of modules
• Reveals traffic between modules, e.g.
• Observe when each neighbor update is issued
• Observe data traffic through network stack
• How Browsable Device File Hierarchy
• Similar to /proc, modules report their internal
  state
• Human readable and binary versions
• Binary channel used for IPC
• Same info visible interactively from the shell
• Enables Debugging
• Locate faults by verifying modules input and
  output
• Visualize distributed system including dynamics
• In simulation
• In real life with debugging backchannel

cat node001/link/mote0/status Root Device
Simulated BMAC Mote Stack sim,mote0 Interfac
e Addr 0.0.0.1 MTU 200 Stats packets_rx 1
packets_tx 1 bytes_rx 164 bytes_tx 164
errors_tx 0 errors_rx 0 Active 1 Promisc
0 POT 6
22
Robustness and Fault Tolerance

• Why is robustness so important?
• Degradation in presence of permanent HW, SW
  faults
• Recovery from transient faults, limiting
  cascading failures
• e.g. unanticipated sensor data
• Unusual cases that yield inconsistent or
  confusing data
• Microkernel Implementation
• FUSD (Framework for User Space Devices)
• Fault isolation between client, server, and
  kernel
• Servers robust to faulty clients
• Modules communicate through POSIX Device API
• Inter-module Fault Tolerance
• Similar approach as in distributed systems
• Survive a range of errors and module failures
• Soft-state protocols between modules
• Rate limiting, filtering, refresh at module
  interfaces

23
Future Work

• EmTOS
• A compatibility layer that enables code written
  against the TinyOS API to be run as an EmStar
  module
• Enables simulation of whole systems composed of
  Motes and Microservers (to appear, SenSys 2004)
• Sensor Device
• Standard device interface for exposing sensor
  interfaces
• Initial version currently in use, future versions
  will improve performance for high-rate sensors
• Improved Simulation Capabilities
• World-modeling capabilities to support node
  mobility (for NIMS)
• Sensor data replay and modeling
• New, improved channel models
• For more information
• http//cvs.cens.ucla.edu/emstar

24
Thank You

• More information
• http//cvs.cens.ucla.edu/emstar

25
Situated Systems

• Situatedness Existing in, and having one's
  behavior strongly affected by a complex
  environment.1

• Goal Achieve a high-level task, given
  challenging inputs
• Unreliable
• packet loss, varying link quality
• node failure
• Inconsistent
• Noise that can be modeled
• Persistent errors reflecting environ
• Complex
• Environment is complex and dynamic cant
  generally be known

1Maja J Mataric, in the Encyclopedia of Cognitive
Science, Macmillan 2002