The DTN Reference Implementation

1
The DTN Reference Implementation

• Michael Demmer
• IETF DTNRG Meeting - March 9, 2005

2
Implementation Overview

• Written primarily in C
• 22,000 non-comment lines of C (5,000 C)
• 10K in generic system support classes (oasys)
• 189 individual classes
• Multithreaded (pthreads), mutex, spin lock
• STL for data structures (string, list, hashtable,
  )
• Emphasize clarity, cleanliness, flexibility
• Ported to Linux, Solaris, Win32 (Cygwin), Linux
  on PDA (ARM), FreeBSD, Mac OSX

(Source line statistics generated using David A.
Wheeler's 'SLOCCount)
3
Bundle Daemon (dtnd) Components
API Server
Event Queue
Bundle Actions
BundleDaemon
BundleRouter
ContactManager
UDP
TCP
FragmentationManager
Sneakernet
Convergence Layers
Tcl Console /Configuration
Thread
dtnd
4
Bundle Daemon Components

• Bundle Daemon core event queue and dispatcher
• Bundle Router all smarts in the system,
  effects policies via a the Bundle Actions
  interface
• API Server handler for application IPC interface
• Convergence Layers transport-specific mechanism
  for transmitting bundles to a next-hop
• Contact Manager service for detecting connection
  opportunities with neighbors
• Fragmentation Manager proactive and reactive
  fragmentation, maintains reassembly buffers
• Persistent Store object database abstraction for
  durable storage, uses Berkeley DB or SQL backends

5
Bundle Daemon Events
API Server
Event Queue
Bundle Actions
BundleDaemon
BundleRouter
ContactManager
UDP
TCP
FragmentationManager
Sneakernet
All events (e.g. bundle arrival, contact
disruption, reassembly completed, etc.) posted
to the daemon event queue.
Convergence Layers
Tcl Console /Configuration
Thread
dtnd
6
Bundle Router Actions
API Server
Event Queue
Bundle Actions
BundleDaemon
BundleRouter
ContactManager
UDP
TCP
FragmentationManager
Sneakernet
Events are then forwarded to the router, which
responds with a set of actions that
are dispatched to components.
Convergence Layers
Tcl Console /Configuration
Thread
dtnd
7
Bundle Router

• Not just traditional routing all policy
  decisions in the daemon
• e.g where / when to send bundles, whether to
  accept custody, apply fragmentation, store
  bundles, etc
• Insulated by event/action interface
• Multiple implementations, chosen at config
• Why this design?
• Believe routing is the hardest problem, may need
  all events
• Easier to change policies if consolidated in one
  place

8
Evaluation
HOP
E2E
user kernel
user kernel
user kernel
user kernel

• Approaches compared (both e2e and hop)
• SFTP (simple ftp-like TCP app) kernel IP or
  plug proxy
• MAIL (Sendmail, no ckpoint/restart) both end
  to end and
• DTN (store-and-forward with restart) hop
  configurations
• Compare robustness to interruption / link errors
• Link up/down patterns aligned, shifted,
  sequential, random

9
DTN Overhead No Disruptions
DTN does best of any for small msgs, modest
overhead overall
10
Interruption Tolerance
Zero throughput for e2e
Note zero throughput for all E2E cases
Up/down 1m/3min 40KB messages shift 10s
11
Simulator Framework

• Implementation also includes basics of a discrete
  event simulator for prototyping and testing
• TCL based simulation script to generate contact
  schedules, test traffic patterns, link bandwidth
  / delay, etc.1
• Uses the same events and abstractions
• Implemented via a SimConvergenceLayer, Bundle
  Payload option for no data, modified event queue,
  and SimBundleActions
• Other core classes and architecture are
  unmodified
• Allows testing of the whole code base except
  convergence layers
• Key characteristic is that to the router, there
  is no distinction between simulation or deployment

1 not all planned simulation features are yet
implemented
12
Current Status

• Basic system operational and (fairly) stable
• Primitives for in-memory and on-disk Bundle
  management
• Extensible naming and addressing framework
• Next-hop taxonomy of peers, links, contacts
• TCP and UDP convergence layers (with pending ID
  specs)
• Implemented (mostly) Bundling protocol v3
• Berkeley db and sql storage implementations
• Sockets-like application interface
• Sample applications dtnping, dtnsend, dtnrecv,
  dtncp
• Proactive and reactive fragmentation and
  reassembly

13
Major ToDo Items
Bundle Router Split into Bundle Engine / Bundle Router Implement a non-static routing protocol
Bundle Management Implement Bundle expirations Custody transfer, delivery status notification Class of Service based forwarding decisions
API Server Improve error reporting and robustness Change from loopback UDP to TCP Handle application disconnection
Convergence Layers Some more ad-hoc protocol (e.g. raw 802.11) More non-traditional networking (e.g. USB key)
Contact Manager Multicast or broadcast based peer discovery Scheduled links and contacts
Simulator Complete implementation
14
Implementation Details

• ISO Stack View
• Example Event / Action Flow
• Naming / Address Families
• Bundle Class Representation
• Bundle Consumers (next hop)
• Bundle Lists
• Convergence Layer API
• Persistent Storage Interface

15
ISO Stack View
DTN Application
DTN2 socket-like API
Application
XDR
IPC
(plan to change to TCP)
UDP
Embedded Application
Bundle Daemon
Application can also run the daemon code as a
thread.
Bundle
Embedded Application
Bundle Protocol
Presentation
TCP
UDP
Sneakernet
Transport
(in progress)
to/from next hop
16
Example Event / Action Flow
User configures a new ONDEMAND TCP link and a
route for the link.
Router checks bundle mappings and finds no other
destinations
Database stores the bundle.
event LINK_CREATED
TCPCL resolves the destination admin string and
establishes a connection to the next hop.
action STORE_DEL
event LINK_AVAILABLE
Database removes the bundle.
Router adds link to route table.
event CONTACT_UP
Router checks the link, finds a pending bundle,
moves the bundle to the contact.
TCPCL interface accept()s a new connection and
constructs a new bundle from the wire.
Disruption occurs on the connection, TCPCL
times out on a keepalive message
action ENQUEUE_BUNDLE
event BUNDLE_ARRIVED
event CONTACT_DOWN
event LINK_UNAVAILABLE
TCP Connection thread picks bundle off contact
queue, sends it to the next hop.
Router matches the link in the route table as the
only next hop for the bundle.
Router checks queues for any bundles that need to
be re-routed to other links.
17
Bundle Router Events

• BUNDLE_RECEIVED // New bundle arrival
• BUNDLE_TRANSMITTED // Bundle or fragment
  successfully sent
• BUNDLE_EXPIRED // Bundle expired
• BUNDLE_FORWARD_TIMEOUT // A Mapping timed
  out
• CONTACT_UP // Contact is up
• CONTACT_DOWN // Contact abnormally
  terminated
• LINK_CREATED // Link is created into the
  system
• LINK_DELETED // Link is deleted from the
  system
• LINK_AVAILABLE // Link is available
• LINK_UNAVAILABLE // Link is unavailable
• REGISTRATION_ADDED // New registration
  arrived
• REGISTRATION_REMOVED // Registration removed
• REGISTRATION_EXPIRED // Registration expired
• REASSEMBLY_COMPLETED // Reassembly complete
• ROUTE_ADD // Add a new entry to the route
  table
• ROUTE_DEL // Remove an entry from the route
  table

18
Bundle Action Interface

• enqueue_bundle(Bundle bundle, BundleConsumer
  nexthop, bundle_fwd_action_t fwdaction,
  int mapping_grp, u_int32_t
  expiration, RouterInfo router_info)
• dequeue_bundle(Bundle bundle, BundleConsumer
  nexthop)
• move_contents(BundleConsumer source,
  BundleConsumer dest)
• store_add(Bundle bundle)
• store_del(Bundle bundle)
• open_link(Link link)
• close_link(Link link)

19
Naming / Address Families
URI (schemescheme-specific-part)
AF Admin String Format
internet host//lthostgtltport?gt/demux
fixed 0x0102
smtp1 smtp//ltusergt_at_lthostgt/demux
wildcard or / or
bundles//ltregiongt/ltadmingt
addressfamily (AF)
dns-style string
Examples Examples
Bundle Destination bundles//internet.dtn/host//test.dtnrg.org/myapp1
Local Interface (admin only) host//0.0.0.05000
RouteTable (destination pattern) bundles//dtnrg.org/smtp//_at_dtnrg.org/
RouteTable (next hop) bundles//some_region/host//xyz.dtnrg.org5000
RouteTable (default pattern) bundles///
1 smtp addressing has not actually been
implemented (yet)
20
Naming / Address Families (2)
AddressFamily
- schema string
validate(admin_str) resolve(admin_str) match(admin_str, pattern)
BundleTuple
- tuple string - region string - admin string proto string family AddressFamily
parse(string) valid() compare(tuple)
InternetAddressFamily
FixedAddressFamily
WildcardAddressFamily
BundleTuplePattern

match(tuple)

• AF abstraction allows essentially any format for
  admin strings
• BundlePattern class implements AF-specific
  wildcard matching

21
Bundle Class Representation
SerializableObject

• Bundle Metadata always in memory
• Reference-counted C object for safety
• Various fields parsed from the wire format into
  member variables
• BundleProtocol class to convert to and from
• Object serialization layer for synchronizing with
  the database
• BundlePayload abstraction
• Manages on-disk representation of payload
• Methods for read_data(), write_data()
• For small bundles, keeps in-memory copy

Bundle
source BundleTuple- destination BundleTuple- custodian BundleTuple- reply-to BundleTuple- payload BundlePayload- mappings BundleMappings- priority int- custody_req boolean- return_rcpt boolean- creation_ts timestamp_t- refcount int
serialize() add_ref(), del_ref() add_mapping(), del_mapping()
22
Bundle Consumers
BundleConsumer
next_hop string bundle_list BundleList
enqueue_bundle(Bundle) dequeue_bundle(Bundle)
link0 Scheduled, UDP
link1 Opportunistic, SneakerNet
peer0
link2 OnDemand, TCP
peer1
Contact
Registration
Peer
Link
peer a next-hop dtn daemon
link means to communicate with a peer
contact created when link is open
registration locally attached application
OndemandLink
ScheduledLink
OpportunisticLink

• Routing can return any consumer as a valid
  next-hop
• virtual override of enqueue_bundle to implement
  different abstractions
• Allows router to use most-natural expression of
  an algorithm
• enqueue bundle on peer implies send on next
  contact on any link to the peer

23
Bundle Lists / Bundle Mappings
Pending Bundles
Contact 1
Contact 2
Bundle Lists
bundle_list Pending Bundles
list_position 1op
NONE bundle_list Contact 1 list_position
2op FORWARD_ONCEgroup
id 0x10timeout
1097563442.125245 bundle_list Contact 2
list_position 1op
FORWARD_ONCEgroup id 0x10timeout
1097553431.324565
Bundle 1
Bundle 2
Bundle 3
BundleMappings
24
Convergence Layer API
Convergence Layer API Convergence Layer API
add_interface(Interface) create a new ingress point for bundle arrivals
add_link(Link, argc, argv) prepare the given link for transmission
open_contact(Contact) open a connection for the given contact
send_bundles(Contact) try to send bundles queued on the given contact
del_interface(), del_link(), close_contact() remove the given interface/link/contact

• Though the abstraction is intended to be general,
  to date only two functional implementations (TCP
  and UDP)
• API may (will likely) need to be extended to
  account for other transports
• Other things of note
• May add an MTU (e.g. UDP limited to 16K max
  bundle)
• Likely need abstractions for neighbor discovery,
  peer identification, etc.

25
Persistent Storage Interface
PersistentStore

load() add(int, SerializableObject) del(int) get(int) update(int, SerializableObject)
BundleStore
store PersistentStore
load() add(Bundle) del(int) get(int) update(Bundle)
RegistrationStore
store PersistentStore
load() add(Registration) del(int) get(int) update(Registration)
GlobalStore
store PersistentStore
load() next_bundleid() next_regid() update()
SQLStore
impl SQLImplementation
BerkeleyDBStore
BerkeleyDBBundleStore
SQLBundleStore
SQLImplementation

• Several layers of abstraction for implementation
  transparency
• Easy to extend with new storage types (e.g.
  routing state) or storage implementations

MYSQLImpl
PostgresSQLImpl
26
Persistent Storage Interface (2)

• SerializableObject interface allows a single
  function to define marshal / unmarshal for all DB
  types

int BundleSerialize(SerializeAction a)
a-gtprocess(bundleid, bundleid_)
a-gtprocess(source, source_)
a-gtprocess(dest, dest_)
a-gtprocess(priority, priority_)
a-gtprocess(custreq, cust_req_)
a-gtprocess(return_rcpt, return_rcpt_) //
...

• Adding a new DB implementation requires
  implementing a handful of functions for different
  basic types (int, string, etc)
• Berkeley DB implementation packs fields into a
  byte array
• SQL implementations generate an update command
  e.g. update bundles set custreq true where
  bundleid 10