Fjording the Stream: An Architecture for Queries over Streaming Sensor Data

1
Fjording the Stream An Architecture for Queries
over Streaming Sensor Data

• Samuel Madden, Michael Franklin

2
Pathway

• General Sensor Environment
• Sensor Query Processing
• Fjords
• Traffic Implementation and Results
• Optimization Issues

3
Sensor Environment
Sensor Proxy
Results
Sensors
Queries
User Terminal
Query Processor
Sensor Proxy
4
Current Trend
3
3
10cm
1mm
Mote
Sensor
5
Sensor Query Processing

• Limitations of Sensors
• Streaming Data
• Processing Concurrent Queries

6
Fjords
Remote Source
WAN
Remote Operator
Controller
M
Thread1
Instantiate Fjord
Disk
Thread2
A
Scheduler
Sensor Source
B
Wireless Net
Remote tuples pulled
Sensor
Data Flow
Join Remote,Sensor
Sensor Tuples Pushed
Local Node
7
Operators and Queues
O
Q
Q
i
o
Filter
Operator
Output Queues
Input Queues
8
State Based Execution
Sj,o
Si,i
Transition
State Transition of an Operator
9
Selection Operator Code

• public Tuple transition(State state)
• mesg inputQueue.get()
• if( mesg ! null )
• if( mesg instanceof TupleMsg
• filter.apply((TupleMsg)mesg).getTuple(
  )) )
• tuple ((TupleMsg)mesg).getTuple(
  )
• else // handle other messages
• return tuple

10
Pull Queue Code

• Public Msg get()
• Tuple tup null
• while( tup null )
• tup below.transition(bSt)
• return new TupleMsg(tup)

11
Sensor Sensitive Operators

• Selections and Projections?
• Average, Sort, Count?
• Join?

12
Sensor Proxy
Sensor Proxy and Query Processor
User
Sensors
13
Building a Fjord

• Query Language
• Restrictions Joins of streams, sorting a stream
• How to build a fjord?
• Selections and Projections
• Joins
• Sorts and Aggregates

14
Multiple Queries in a Fjord
user 1
Query Processor
Sensor Proxy
user 2
Common Tuple
user k
15
Traffic Implementation and Results

• Query 1
• SELECT AVG(s.speed, w )
• FROM sensorReadings AS s
• WHERE s.segment IN knownSegments

16
Traffic Implementation and Results

• Query 2
• SELECT AVG(s.speed, w ), i.description
• FROM incidents AS i, sensorReadings as s
• WHERE i.time gt now timeWindow
• GROUP BY i.description
• HAVING speedThreshold gt
• (SELECT AVG( s.speed, w )
• FROM sensorReadings as s
• WHERE i.segment s.segment
• AND s.segment IN knownSegments)

17
Observations Single-Fjord is better. Spikes
at Q2. The slope decreases.