Distributed Computation in the Physical World

1
Distributed Computation in the Physical World

• David Culler
• UCB
• ICDCS 2005

2
Enabling Technology for Science
the complex

• macroscope
• P. Anthony 1969
• J. de Rosnay, 1979

Perceive
the imperceptible
the atomic
the small
the far
3
The (A) Promise of Sensor Networks

• Dense monitoring and analysis of complex
  phenomena over large regions of space for long
  periods
• Many, small, inexpensive sensing devices
• Frequent sampling over long durations
• Non-perturbing
• Close to the physical phenomena of interest
• Compute, communicate, and coordinate
• Many sensory modes and vantage points
• Observe complex interactions

4
Diverse Applications

• Monitoring Spaces
• Env. Monitoring, Conservation biology, ...
• Precision agriculture,
• built environment comfort efficiency ...
• alarms, security, surveillance, treaty
  verification ...
• Monitoring Things
• condition-based maintenance
• disaster management
• Civil infrastructure
• Interactions of Space and Things

5
Case Study Redwood Ecophysiology

• 70 of H2O cycle is through trees, not ground
• Complex interactions of tree growth and
  environment
• Effected by and effect the microclimate
• Need to understand dynamic processes within the
  trees

6
State of the Art

• Solid understanding of leaf physiology
• Good models, good empirical data, good fit
• Extension to the entire tree canopy is open
  problem
• Various models focused on particular aspects
• Nutrient transport, transpiration,
• Extremely limited empirical basis
• Data Dirth
• Satellite observations wide coverage, low
  resolution, canopy surface
• Spot weather stations single point in space
• Instrument elevator haul data logger along
  vertical transect
• Wide range of sensors climate, sap-flow, dew,
• Goal dense monitoring throughout canopy of
  sampling of trees throughout forest

7
The alternative
8
What was Todd looking for?
10m
20m
34m
30m
36m
2003, unpublished
9
The Systems Challenge

• Monitoring Managing Spaces and Things

applications
Store
Comm.
uRobots actuate
MEMS sensing
Proc
Power
technology
Miniature, low-power connections to the physical
world
10
Mote Platform Evolution
11
Wireless Micro-weather Mote

• Incident Light Sensors
• TAOS total solar
• Hamamatsu PAR
• Mica2 dot mote
• Power board
• Power supply
• SAFT LiS02 battery, 1 Ah _at_ 2.8V
• Weatherproof Packaging
• HDPE tube with coated sensor boards on both ends
  of the tube
• O-ring seal for two water flows
• Additional PVC skirt to provide extra shade and
  protection against the rain
• Radiant Light Sensors
• PAR and Total Solar
• Environmental Sensors
• Sensirion humidity temp
• Intersema Pressure temp

mote
Deep Collaboration with Intel
12
Canonical Architecture
13
Embedded Network System

• TinyOS 1.1.x
• Rich, stable event-driven OS for embedded
  networking
• Component-based design methodology for robustness
• Wireless stack, uart, sensor stack, storage,
  timer
• BMAC expressive, power-aware sub-MAC with
  low-power listening
• MINT route adaptive N-1 multihop min-cost
  routing with link-by-link retransmission
• TinyDB
• Stream SQL-like in-network query processing
• Periodic sampling with local filters stream data
  to root
• Epoch-based power mamangement
• Tiny Application Sensor Kit
• Stream to archival table, metadata table, query
  GUI
• Postgress on stargate linux gateway
• Push the technology to meet the science vision
• Backfill in case reality falls short (logged all
  the physical samples)

14
Study Format

• Site Sonoma Coast Grove of the big Trees
• Near Occidental, CA
• Adjacent to winery
• Interior and Exterior Trees
• 40-50 nodes per tree
• Range of elevations
• Multiple nodes per level, center and periphery
• 25 day duration
• 5 min sample interval
• - all sensors, battery, parent
• Ground level weather station, fog sensors
• Sap Flow sensors
• Internal indicator of level of photosynthetic
  activity

15
Instrument Positions
16
May 15
17
Groundbreaking Science
Unpublished in preparation
18
Just the beginning

• Real time and post hoc correlations of dense in
  situ instrumentation with satellite data (modis)
• Sensor networks as a part of the larger
  information enterprise
• Weather, remote sensing,
• Correlation with internal physiology
• Sap flow as estimate of photosynthetic activity
• Model-based analysis
• Feed empirical data into models to estimate
  transpiration
• Compare with assumed microclimate
• Sampling across forest regions
• (10 year) Understanding role of trees in
  (coastal) hydrological cycle
• (25 year) Understanding the elements of the
  carbon fixation cycle
• and in-network data analysis

19
Message 1

• Wireless sensor network technology is getting
  mature enough to deliver on the promise.
• Sensor / Mechanical / Power are custom
• Let the expect decide
• Systems Networking are common
• Starts with pure data collection
• Grows into in-network data analysis

20
How does a bunch of wireless devices become a
(programmable) network?

• Localized algorithms Distributed computation
  where each node performs local operations and
  communicates within some neighborhood to
  accomplish a desired global behavior
• D. Estrin, 21st Century Challenges

21
Vast Networks of Tiny Devices

• Past 25 years of internet technology built up
  around powerful dedicated devices that are
  carefully configured and very stable
• local high-power wireless subnets at the edges
• 1-1 communication between named computers
• Here, ...
• every little node is potentially a router
• work together in application specific ways
• collections of data defined by attributes
• connectivity is highly variable
• must self-organize to manage topology, routing,
  etc
• and for power savings, radios may be off 99 of
  the time

22
Common Communication Patterns

• Internet
• Many independent pt-pt stream
• Parallel Computing
• Shared objects
• Message patterns (any, grid, n-cube, tree)
• Collective communications
• Broadcast, Grid, Permute, Reduces
• Sensor Networks
• Dissemination (broadcast epidemic)
• Collection
• Aggregation
• Tree-routing
• Neighborhood
• Point-point

Disseminate the Query - eventual
consistency Collect (aggregate) results
The Emergence of Networking Abstractions and
Techniques in TinyOSPhilip Levis, Sam Madden,
David Gay,  Joseph Polastre, Robert Szewczyk,
Alec Woo, Eric Brewer, and David Culler, NSDI'04
23
The Basic Primitive

• Transmit a packet
• Received by a set of nodes
• Dynamically determined
• Depends on physical environment at the time
• What other communication is on-going
• Each selects whether to retransmit
• Potentially after modification
• And if so, when

24
Routing Mechanism?

• Upon each transmission, one of the recipients
  retransmit
• determined by source, by receiver, by
• on the edge of the cell

25
How Should We Think About Routing?

• Classical View
• Discover the connectivity graph
• Determine the routing subgraph
• relative to traffic pattern
• Compute a path and Route data hop-by-hop
• Destination selection
• Queuing, multiplexing, scheduling,
  retransmission, coding,
• Here?
• What does it mean to be connected?
• What does it mean to route?
• Do you need the connectivity graph?

26
The Most Basic Neighborhood

• Direct Reception
• Non-isotropic
• Large variation in affinity
• Asymmetric links
• Long, stable high quality links
• Short bad ones
• Varies with traffic load
• Collisions
• Distant nodes raise noise floor
• Reduce SNR for nearer ones
• Many poor neighbors
• Good ones mostly near, some far

27
BCAST Fundamental building block

• Commands
• Wake-up
• Form routing tree
• Discover route
• Source-destination discovery (DSR, AODV)
• Exploration in directed diffusion
• Time-synchronization
• Constructed from underlying local broadcast

28
Flooding

• Simple Address-Free Algorithm Schema
• if (new bcast msg) then
• take local action
• retransmit modified request
• Naturally adapts to available connectivity
• Minimal state and protocol overhead
• gt surprising complexity in this simple mechanism

29
Radio Cells
30
Flood Spanning Tree
0
31
Empirical Dynamics of Flood

• Experimental Setup
• 13x13 grid of nodes
• separation 2ft
• flat open surface
• Identical length antennas, pointing vertically
  upwards.
• Fresh batteries on all nodes
• Identical orientation of all nodes
• The region was clean of external noise sources.
• Range of signal strength settings
• Log many runs

Ganesan, Krishnamachari, Woo, Culler, Estrin and
Wicker, Complex Behavior at Scale An
Experimental Study of Low-Power Wireless Sensor
Networks , UCLA Computer Science Technical Report
UCLA/CSD-TR 02-0013
32
Final Tree
33
Factors

• Long asymmetric links are common
• Many children
• Nodes out of range may have overlapping cells
• hidden terminal effect
• Collisions gt these nodes hear neither parent
• become stragglers
• As the tree propagates
• folds back on itself
• rebounds from the edge
• picking up these stragglers.
• Mathematically complex because behavior is not
  independent beyond singe cell
• Redundancy
• Geometric overlap gt lt41 additional area

Ni, S.Y., Tseng, Y.C., Chen, Y.S., Sheu, J.P.
The broadcast storm problem in a mobile ad hoc
network. MobiCom'99
34
Selective Retransmission Schemes

• Probabilistic Retransmission
• Fixed prob.
• What would be the right choice?
• Counter
• When hear msg, start random delay
• If hear C msgs during wait, dont retransmit
• Distance
• If nearest node from which msg is heard is less
  than some threshold, dont retransmit
• Location
• If portion of cell not covered by transmitting
  neighbors is less than some threshold, dont
  retransmit
• Cluster-based
• Partition graph into cluster heads, gateways, and
  members
• Members dont transmit

35
Adaptive BCAST rate

• Upon first msg
• Start random delay
• If new msg arrives during delay
• Filter message (eg., discard if signal strength
  below threshold)
• If passes filter, Utilize message
• Start new delay
• Upon expiration of delay
• Complete local processing
• E.g., pick lowest depth node with strongest
  signal as parent
• Retransmit
• Delay is proportion to cell density
• Wait till ngbrs go quiet before transmit
• gt Approx uniform transmissions per unit area,
  regardless of node density
• Exploit long links when appropriate

36
Example Tree
37
Flooding vs Gossip

• In gossip protocols, at each step pick a random
  neighbor
• Assumes an underlying connectivity graph
• Typically used when graph is full connected
• E.g., ip
• Much slower propagation

38
Reliable, Epidemic Dissemination

• TinyDB query dissemination
• Periodically transmit current query
• If hear new query, accept it and start
  retransmitting it
• Does not scale well over density
• Is not responsive at low rate of management
• gt Trickle (Levis, NSDI 04)

39
Epidemic Dissemination - Trickle

• Key Idea maintain constant flux of communication
  per unit area, regardless of node density
• More nbrs gt listen more, talk less
• Announcement rate a 1/cell_density
• Trickle Algorithm
• Listen for random interval
• If too few announcements, announce current
  version
• Low maintenance costs
• Interval increases in absence of new
  announcements
• Quick response
• Contract window upon new announcement
• Low contention
• If hear old version announcement, open
  suppression wider

Trickle A Self-Regulating Algorithm for Code
Propagation and Maintenance in Wireless Sensor
Networks, Levis et al, NSDI'04
40
Deluge Network Programming

• Every byte must (eventually) be correctly
  received by all nodes!
• Reliable Pipelined Epidemic Distribution of
  series of pages
• Constrained storage hierarchy
• Packet (32 bytes) ltlt RAM (4K) ltlt program (128K) lt
  external flash (512K)
• Lossy links, Critical Contention
• Density-aware
• Robust to asymmetric links
• Dynamic adjustment of advertisements
• Minimize set of concurrent data broadcasts
• Spatial multiplexing
• Page Advertise, Request/Fix, Xfer
• Density-aware suppression and snoop on each
• Packet CRC Page CRC
• 159 Byte memory footprint

flash

41
Deluge Basic Protocol (1)

• Advertise Version Periodically (trickle)
• Dynamic adjustment of period
• Duplicate suppression

1
1
1
1
1
1
1
42
Deluge Basic Protocol (1)

• Advertise Version Periodically (trickle)
• Dynamic adjustment of period
• Duplicate suppression

Version 2 Pages (0,1,2)
1
2
1
1
1
1
1
1
43
Deluge Basic Protocol (2)

• Request Advertised Page
• Lowest numbered page needed
• Bit-vector for specific packets (initially all)
• Suppress requests upon hearing others

Page 0
1
2
Page 0
1
Page 0
1
1
1
1
1
44
Deluge Basic Protocol (3)

• Broadcast page contents
• Series of packets
• Targeted receiver, but others snoop

Page 0, Packet 15
1
2
Version 2
1
1
1
1
1
1
45
Deluge Basic Protocol (4)

• Selective Negative Ack (SNACK)
• Notify sender of all missing packets
• Gives up after a few tries in case of asymmetric
  links
• Snoopers can present additional snacks

Page 0, Packet 24
1
2
1
Page 0, Packet 15
Packet 2
1
1
1
1
1
46
Deluge Basic Protocol (5)

• Advertise New Pages
• Basic After receiving entire image
• Pipelined After receiving each page (with delay)

1
2
1
Page 0
2
1
2
1
1
47
Deluge Simulation (10x10)
48
Deluge Simulation (16x16)
49
Deluge _at_ EXSCAL under TOSSIM
9 m pitch


Stargate source

• 2 of 30 sections, 721 nodes, 21 Stargate
• 5 pages (512 B) color coded
• empirical 4 minutes for 33k to 40 nodes 1 hop

50
Message 2

• Many important routing protocols utilize whatever
  connectivity occurs
• Never build the graph
• Extremely robust
• boil down to whether to retransmit
• and when
• Routine infrastructure can present surprisingly
  interesting problems

51
Collection

• Common use monitoring
• Collection of nodes take periodic samples
• Stream data towards a root node
• Root announces interest
• depth 0
• Nodes listen
• When hear neighbor with smaller depth
• start transmitting data to best lower neighbor
• set own depth to one greater (and include with
  data)
• Data transmission continuously reinforces
  adjusts routes
• Aggregation within nodes or within the tree

52
Collectively Build/Maintain Spanning Tree
0
53
Building Neighborhoods Routes

• Node transmits to some unknown set
• Candidate nbrs are sources of incoming packets
• Estimate of inbound link reliability
• Occasionally announce inbound link states
• Provides reverse link estimate to outbound
  neighbors
• Basis for cost-based routing
• Cost-based Parent Selection
• depth(me) MIN nbr(me) depth(i)
• loss(me) MIN nbr(me) loss(i)est(me,i)
• trans(me) MIN nbr(me) trans(i)etrans(me,i)

Taming the Challenges of Reliable Multihop
Routing in Sensor Networks, Alec Woo and David
Culler,  SenSys. 2003.
54
Behavior over Time
Est. Link Quality
Tree Depth
1
2
3
55
Impact of Long Links
56
Power-aware Routing

• Cost-based Routing
• Minimize number of hops
• Minimize loss rate along the path
• Perform local retransmissions, minimize number
  along path
• Energy balance
• Utilize nodes with larger energy resources
• Utilize redundancy
• Nodes near the sink route more traffic, hence use
  more energy
• Give them bigger batteries or provide more of
  them and spread the load
• Randomize routes
• Utilize heterogeneity
• Route through nodes with abundant power sources
• All are variants on cost metric and parent set

57
Whats the neighborhood?

• 3 regions
• transitional region is large
• Typical Deployment Strategy
• Nearest node distance effective range

Transitional Region
Effective Region
many
Clear Region
58
Neighbors in Transitional Region

• Hear many potential neighbors
• Small fraction of them are good neighbors
• Potential nbrs gtgt Nbr Table
• Hard to differentiate without history
• Neighbor table Mgmt?
• FIFO?
• Cache?
• Frequency?

?
Neighbor Table
59
Neighbor Management

• Fixed-size table (40) as cell density increases

GN lt PN lt NT
NT lt GN lt PN
GN lt NT lt PN
Number of Potential Neighbors
60
Neighbor Management

• 50 good neighbor entries

GN lt PN lt NT
NT lt GN lt PN
GN lt NT lt PN
Number of Potential Neighbors
61
The Amoeboed cell
Distance
62
Which node do you route through?
63
Message 3

• Always routing through nodes at the hairy edge
• Wherever you set the threshold, the most useful
  node will be close to it
• The underlying connectivity graph changes when
  you use it
• More connectivity when less communication
• Discovery must be performed under load

64
Towards a sensor net architecture?

• Wide range of long-lived applications
• Diverse, constrained, evolving resources
• Low duty cycle
• Small tables
• Loss, noise change
• Embedded in adapting to phy. env.
• In-network processing, not E2E
• Highly application specific
• Few applications over many nodes
• WSN needs a narrow waist too!

65
Sensor Network Protocols Today
Appln
EnviroTrack
Hood
TinyDB
Regions
FTSP
Diffusion
SPIN
Transport
TTDD
Trickle
Deluge
Drip
MMRP
Arrive
Routing
TORA
Ascent
MintRoute
CGSR
AODV
GPSR
ARA
DSR
GSR
GRAD
DBF
DSDV
TBRPF
Scheduling
Resynch
SPAN
FPS
GAF
ReORg
Topology
PC
Yao
SMAC
WooMac
PAMAS
BMAC
TMAC
WiseMAC
Link
Pico
802.15.4
Bluetooth
Phy
eyes
RadioMetrix
CC1000
nordic
RFM
66
Communication and Power
listen
off
off
RX
TX

• Energy Cost RX TX Listen
• Costs power whenever radio is on
• Transmitting, receiving, or just listening
• Transmit is easy, Rcv is whats tricky
• Want to turn it on just when there is something
  to hear
• Two approaches
• Schedule transmission intervals
• Statically, dynamically, globally, locally
• Make listening cheap

67
Power States at Node Level
Active
Active
Telos Enabling Ultra-Low Power Wireless
Research, Polastre, Szewczyk, Culler, IPSN/SPOTS
2005
68
Straw view of SNA stack
Applications Compose what they need
Tracking Application
Sensing Application
Multiple Network Layer Protocols
69
Implications of lowering the waist

• Abstract over and provide uniform interface to a
  range of current and future link layers
• CC1000, IEEE 802.15.4, infineon, nodic, 15.4.b,
• Utilize underlying power management mechanisms
• Utilize capabilities, while working within
  constraints
• Optimize network protocol in terms of link
  abstraction (SP), not particular underlying link
• More expressive interface
• Feedback from lower layers
• Multiple network protocols coexist
• How do they cooperate?
• Permit (encourage) innovation in network
  protocols, unconstrained to particular link de
  jeur
• Separation of concerns

70
Properties of SP

• SP provides mechanisms for network protocols to
  operation
• Network protocols may introduce policy
• Three key elements of SP
• Data Reception
• Data Transmission
• Neighbor Management

71
Message Reception
Receive
SP

• Message arrives from link
• SP dispatches
• Network protocols establish
• naming/addressing
• filtering

72
Message Transmission
Send
Receive
SP

• Messages placed in shared message pool
• All entries are a promise to send a packet in
  the future
• Messages include
• Pointer to first packet and of packets
• Message Futures Similar to Lazy Task Creation
• Control information reliability and urgency
• Feedback information congestion and phase

73
Neighbor Management
Neighbors
Send
Receive
SP

• SP provides a shared neighbor table
• Cooperatively managed
• SP mediates interaction using table
• No policy on admission/eviction by SP
• Scheduling information

74
SP Architecture
75
Initial Experiment

• Four representative network protocols
• Single Hop Neighbors
• Dissemination Trickle
• Collection MintRoute
• Aggregation Synopsis Diffusion
• Two representative link layers
• Sampled BMAC CSMA with LPL
• Communication is unsynchronized
• Aloha with Preamble Sampling, Mica1 LPL, CC2500,
  Reactive Radio, PAMAS
• Slotted 802.15.4
• S-MAC, T-MAC, TRAMA

76
Slotted Protocols 15.4 Beacons

• 15.4 Protocol
• Each node beacons on its own schedule
• Other nodes scan for 15.4 beacons, synchronize
• SP does
• Neighbor information inserted by 15.4
• Instructs 15.4 to wake during other beacon
  periods

CSMA Contention Period
Beacon
Beacon
Data
Data
Ack
sleep
Superframe Duration
Beacon Frame Duration
77
Sampling Protocols BMAC LPL

• Periodically samples the channel for activity
• Messages are sent at wakeup time
• Receivers can synchronize to senders
• Subsequent messages piggybacked on long messages

wakeup
TX preamble
sleep
sleep
sleep
packet
Node 1
time
RX
sleep
sleep
sleep
Node 2
packet
time
78
Sampling Protocols BMAC LPL

• What does SP do?
• Controls the length of the preamble
• Controls backoffs based on message type
• Counts backoffs for congestion feedback
• Controls clear channel assessment
• B-MAC
• Returns schedule information about wakeups
• Provides phase feedback hints

79
Single Hop mica2 Throughput
80
Single Hop mica2 Throughput
81
Single Hop mica2 Throughput
82
Single Hop mica2 Throughput
83
Single Hop mica2 Throughput
84
Single Hop 802.15.4
85
MintRoute
Send
Receive
Intercept
forwarding queue
SP Message
SendRoute Beacons
UpdateNeighbor ETX
ChooseParent
MintRoute
1st packet
MultiHop Engine
MultiHop Neighbors
Next PacketHandler
Send
Receive
Send
Receive
Neighbors
SP
Message Dispatch
Neighbor Functions
Link Estimator
Link Protocol
86
Collection MintRoute
Telos Results Min Med Avg Max
Duty Cycle () 3.1 4.5 4.4 4.9
Delivery 94.1 96.6 97.4 100
Retx/pkt 0 .057 .059 .095
Parent Changes 0 1 1.58 5
Parent Evictions 0 0 0 0

• Code Size
• mica2 28 smaller
• Telos 23 smaller
• Comparable if SP code size included

87
Dissemination Trickle
88
Synopsis Diffusion

• Sends synopses towards a collection point
• Needs a gradient to know which way to aggregate

Synopsis Diffusion
Simple Implementation
Gradient Manager
Node Address
Send
Receive
Neighbors
SP
Message Dispatch
Neighbor Functions
Link Estimator
Link Protocol
89
Synopsis Diffusion

• Requires a gradient to the collection point

Collaborative Implementation
Synopsis Diffusion
MintRoute Maintaining Hop Count
Gradient Manager
Send
Receive
Neighbors
SP
Message Dispatch
Neighbor Functions
Link Estimator
Link Protocol
90
Combining Network Protocols (mica2)

• Neither MR nor SD know about each other
• SPs message pool allows batching and power
  savings
• Overall power savings of 35extends node
  lifetime by over 54

91
Sensor Net Application Taxonomy
Power Availability
scarce

• Also
• Scale
• Processing
• Time correlation

trickle
plentiful
burst
benign
Sampling
harsh
Environment
92
Bells Law new class per decade
log (people per computer)
streaming information to/from physical world

• Enabled by technological opportunities
• Smaller, more numerous and more intimately
  connected
• Brings in a new kind of application
• Used in many ways not previously imagined

year
93
Small Technology, Broad Agenda

• Social factors
• security, privacy, information sharing
• Applications
• long lived, self-maintaining, dense
  instrumentation of previously unobservable
  phenomena
• interacting with a computational environment
• Programming the Ensemble
• describe global behavior, synthesis local rules
  that have correct, predictable global behavior
• Distributed services
• localization, time synchronization, resilient
  aggregation
• Networking
• self-organizing multihop, resilient, energy
  efficient routing
• despite limited storage and tremendous noise
• Operating system
• extensive resource-constrained concurrency,
  modularity
• framework for defining boundaries
• Architecture
• rich interfaces and simple primitives allowing
  cross-layer optimization
• Components
• low-power processor, ADC, radio, communication,
  encryption, sensors, batteries

94
The Time is Right

• Dont be afraid to go out and tackle REAL
  problems.
• They often reveal interesting challenges.
• The technology is (just barely) ready for it.
• There is much innovation ahead.

95
Acknowledgments

• Supported by
• DARPA Network Embedded Systems Technology (NEST)
  program,
• National Science Foundation,
• Intel, Sun, CrossBow, Microsoft, HP, NTT DoCoMo,
  Samsung
• CITRIS and California MICRO.
• Where to go for more
• www.tinyos.net
• webs.cs.berkeley.edu

Thanks