CBWare - Distributed Middleware for Autonomous Ground Vehicles

1
CBWare - Distributed Middleware for Autonomous
Ground Vehicles

• Masters Thesis Defense
• By
• Vidhyalakshmi Venkitakrishnan
• Advisor Dr. Arun Lakhotia
• Center for Advanced Computer Studies
• University of Louisiana at Lafayette
• October 18, 2006

2
Presentation Outline

• Part I - Motivation and Research Contributions
• Part II - Background and Related Work
• Part III - CBWare and Evaluation Results
• Part IV - Conclusions and Future Work

3
Middleware in AGVs - Motivation
4
I. Real-time Information Exchange

• Processes are computationally intensive and
  complex
• Need for distributing components on multiple
  machines to achieve fairness and efficiency
• Middleware Key to information exchange among
  distributed components

5
CajunBot
6
II. Sensor Data Fusion

• LIDAR

(Range, Theta)
INS
(x, y, z)
Global (x, y, z)
7
Sensor Data Fusion contd.

• LIDAR Queue
• INS Queue
• Fusing most recent reading from queues not
  consistent
• Both sources generate data at different
  frequencies

8
CajunBot on rough terrain DARPA GC 2005
9
Sensor Data in Rough Terrain
t1
10
III. Log Server

• Need data from processes for post-analysis
• Collect data to tune parameters
• CajunBot NQE 2005
• Z spike - wall of obstacles detected from log data

11
IV. Remote Real-time Monitoring
12
Why Real-time monitoring?

• Visual/graphical views of the world seen by
  system
• Internal states of Obstacle Detection
• Visualize live data during field testing
• Debug problems with processes in real-time

13
Research Contributions

• Real-time Information Exchange in distributed
  system
• Support for fusion of consistent data
• Log server
• Real-time monitoring capability

14
Middleware Background

• What is Middleware?

• Layer of software that connects different
  application components.
• Supports complex, distributed applications
• Hides heterogeneity of underlying platforms

15
Types of Middleware

• Publish/Subscribe
• Client/Server
• Remote Procedure Calls
• Other IPC Mechanisms Shared Memory, Message
  Queues, UNIX IPC

16
Publish/Subscribe Model
Subscriber subscribes to data
Publish/Subscribe Middleware
Publisher
Subscriber
Publisher publishes data
Middleware delivers data

• Powerful abstraction for distributed systems
• Message-based anonymous communication
• Publishers and Subscribers are decoupled
• Good solution for scalability
• Examples CMU-IPC, NDDS, IBM-Gryphon, Siena

17
Client/Server Model
Client 1
Request
Client 3
Server
Response
Client 2

• Clients send request to the server
• Server processes requests and responds back
• Clients blocks until response from server

18
Remote Procedure Calls

• Protocol to extend Local Procedure
• Involves two independent processes, which may
  reside on different machines
• Caller (Process A on Host A) issues procedure
  call to Callee (Process B on Host B) with list of
  argument values
• Caller is suspended
• Callee executes procedure, returns values to
  Caller
• Caller resumes execution
• Examples Java RMI, CORBA, Microsoft DCOM

19
Robotic Middlewares

• IPT (1996)
• Object-oriented, message passing toolkit
• Unmanned Ground Vehicle
• Client/Server Model
• MIRO (2002)
• Middleware for Mobile Robots
• Based on CORBA
• Broker (2005)
• IPC Toolkit for Multi-Robot Systems
• Works on Publish/Subscribe Model

20
CBWare Architecture

• Works on Publish/Subscribe Model
• Two types of Interfaces
• CBQueues
• CBPackets
• Log server
• Remote monitoring

21
Dedicated Logging Machine

• Separate Machine Why?
• Hard disk Most sensitive part of a machine
• Bumps Common reason for hard disk crash/failure
• Disk crash Affects autonomous operations
• Provision to log only sample of data to disk

22
Sampling data for remote monitoring

• Why sample data sent over wireless network?
• Wireless network cannot handle all the data
  produced
• Onboard network 1 GB LAN Ethernet switch
• Wireless network 802.11g Wireless
• Bandwidth
• Ethernet LAN 1000 Mbps
• 802.11g wireless 54 Mbps

23
CBQueues

• Interprocess communication on a single machine
• Interface to read/write messages
• Built using POSIX Shared memory

24
Shared Memory Model

• Area of memory shared by multiple processes.
• Shared Memory area -indistinguishable to a
  process from unshared memory

Code
Code
Private data
Private data
Shared data
Shared data
25
Why Shared Memory Model?

• Fastest form of IPC available
• Negligible communication overhead
• Need to deliver high bandwidth sensor data in a
  timely manner

26
CajunBots Shared Memory Model

• Queues for every message type in shared memory
  area
• Messages in every queue temporally ordered
• Crucial for Interpolation support

27
Single Writer Restriction

• Only one writer for each message queue, no limit
  on the number of readers
• Enables temporal ordering of data in distributed
  queues
• Multiple producers for same message type
• Separate queue maintained for each producer

28
Data Fusion and Interpolation

• INS
• generates data at 100 Hz
• Produces data at 10 ms interval
• LIDAR
• LIDAR generates data at 75 Hz
• Scans at 13 ms interval
• Most recent INS data may be up to 10 ms old when
  LIDAR scan is read

29
Stabilization of sensors

• In CajunBot, no stabilization of sensors
• Fusing most recent data can give erroneous
  results
• 0.5 degree inaccuracy of angular difference -gt 2
  feet displacement of global point from correct
  location
• Leverage rough terrain to increase visibility
• In vehicles with sensor stabilization, can fuse
  most recent data
• Stabilization dampens rotating movements
• Sensors wont experience significantly different
  orientations in the 10 ms period

30
CBPackets

• Distributed interprocess communication across
  machines
• UDP Broadcast
• Support for multiple readers and writers

31
Why UDP Protocol?

• Real-time applications
• Deliver messages in time
• Small Packet Header overhead
• TCP Header 20 bytes, UDP Header 8 bytes
• For a 20 byte message, compare 40 bytes Vs 28
  bytes
• Example
• Case 1 Message Size 12 bytes
• Efficiency using TCP (12 / 1220) 0.37 37
• Efficiency using UDP (12/ 128) 0.6 60
• Case 2 Message Size 1000 bytes
• Efficiency using TCP (1000 / 100020) 0.98
  98
• Efficiency using UDP (1000 / 10008) 0.992
  99

32
Why UDP Protocol? contd.

• Broadcast/Multicast capabilities
• Send messages to all processes at once
• Connectionless nature
• Useful for remote monitoring
• No problems if wireless network fails

33
Data Marshaling

• Why data marshaling?
• Distributed System Machines with heterogeneous
  architectures/OS follow different byte-ordering,
  alignment strategies
• Data converted to neutral format for transmission
  over the network
• Examples of Data Encoding standards XDR, NDR,
  CDR
• XDR used by CBWare

34
XDR Translation
Decode data
Encode data
Receiver
Sender
XDR Format

• Encode Convert data from native format to XDR
  format
• Decode Convert data from XDR format to native
  format
• Every message marshaled by CBWare before sending
  over network

35
Monitoring Process Status

• Processes on multiple machines generate
  status/warning/error messages
• Monitoring messages on each machine
• Tedious task when system scales to higher level
• Cbmesg Interface for logging and real-time
  monitoring of process status information

36
Replication of Shared Memory

• Combination of CBQueues and CBPackets
• Subscriber replicates local shared memory
• Distribute queues to multiple machines
• Easy transfer of programs
• Scalability of computational power

37
CBWare Network Evaluation

• CBWare evaluation metrics
• End-to-End Latency (msec)
• Packet Rate
• Bandwidth (bytes/second)
• Packet Order
• Experimental setup
• Dell Poweredge servers
• 1 GB Ethernet LAN switch

38
CBWare Network Evaluation Results

• End-to-End Latency
• Delays ranging from 0.4 msec to 6 msec depending
  on packet size
• Transmission delay Packet reception time
  Packet sent time

39
CBWare Network Evaluation Results contd.

• Bandwidth and Packet Rate
• Packet Rate (Bandwidth/Message Size)
• Maximum rate ranging from 1200 packets/second to
  90 packets/second
• Packet order
• Measurement based on timestamp
• No out of order packets on CBWare network
• No network congestion
• Network Monitoring Tools used Ethereal, tcpdump

40
Conclusion

• CBWare Middleware for Autonomous Ground Vehicle
• Publish/Subscribe Model
• Distributed Information exchange - Shared Memory
  and UDP Broadcast
• Support for fusion of sensor data
• Log server
• Sampling for remote real-time monitoring

41
CBWares comparison with other Middlewares
CBWare CMU-IPC NML NDDS Broker IPT
Data interpolation Fusion Yes No No No No No
Sampling data for Remote Monitoring Yes No No No No No
Protocol Shared Memory UDP TCP UDP Shared Memory UDP UDP UDP TCP Sockets
Languages C C Java LISP C Java C Java C Java Python Perl C
Pub/Sub Models Yes Yes Yes Yes Yes No
42
Future Work

• Fault tolerance
• Compression techniques for larger messages
• Port CBWare to other Operating Systems

43
Questions
44
Extra slides
45
Distributed Computing System of CajunBot

• Onboard Computing system of CajunBot
• Four machines
• NTP
• Separate networks

46
Why XDR?

• XDR format of data is same on all machines
• Any machine can decode data encoded by any
  machine
• Easily add machines with different architectures

47
Log Control

• Remote control of central log server (cb_logd)
• TCP Connection
• Communication using predefined ASCII Protocol
• Three types of control signals
• Enable logging data
• Disable logging data
• Change log directory on log server

48
CBQueues Utilities

• Remove corrupted shared memory cb_qclean
• Clean shared memory area
• Read/Write to shared memory cb_qtest
• Testing and debugging problems with shared memory

49
CBWares Publish/Subscribe Components

• Publisher cb_publisher
• Handles socket operations
• Constructs CBPackets
• Marshal data
• Broadcast on network
• Central Log Server cb_logd
• Logs data received from broadcast network to disk
• Runs on separate machine to prevent disk failure
  affecting autonomous operations
• Provision to log only a sample of data to disk
• Samples data at prescribed interval and
  broadcasts data on wireless network for remote
  monitoring

50
CBWares Publish/Subscribe Components contd.

• Subscriber cb_subscriber
• Runs in two modes
• Mode 1 Replicate shared memory queues across
  machines
• Mode 2 Real-time monitoring on remote laptop
• Easy transfer of programs to multiple machines
• Easy scalability of computational power

51
CBQueues Interface

• Read interfaces
• Most recent message
• Next message
• Check for new message
• Write interface

52
CBPacket format

• CBPacket CBHeader Data
• CBHeader ltChannel, Timestamp, Size, Checksum,
  Encoding Formatgt
• Maximum transmission size upto 65507 bytes

53
CajunBot on flat terrain DARPA GC 2005
54
Sensor data in Flat Terrain

LIDAR Data
10
INS Data
Pitch of the sensor (degrees)
t1
0
10
Pitch of the vehicle (degrees)
0
1
2
3
4
5
6
7
8
9
10
11
Time Instance (t)