Modeling and Design of WDM LANs for Modern Avionics Platforms

1
Modeling and Design of WDM LANs for Modern
Avionics Platforms

• Casey Reardon
• HCS Research Laboratory
• University of Florida

2
Outline

• Introduction
• Benefits and challenges of WDM networking
• OSI Model and WDM LANs
• Physical layer
• Available components, topologies, defining device
  requirements
• Data Link layer
• Flow control, error control, multiplexing
• Network layer
• Transport layer and above
• Key issues for a WDM avionics LAN
• Packet-switched vs. Circuit-switched networks
• Achieving deterministic performance
• Fault-tolerance, reliability, and recovery
• Definition and calculation of important metrics
• Realizing protocol transparency
• Brief intro of select legacy architectures
• MIL-STD-1553, ARINC 429, AFDX, Fibre Channel
• Summary
• Questions

3
Personal Background and STTR

• Completion of Phase-I of Navy STTR for design and
  virtual prototyping of WDM network architectures
• Related presentations at previous two AVFOP
  conferences
• LION Library for Integrated Optical Networking
  (UF)
• Bridge gap between optic-centric and
  network-centric modeling and simulation analysis
  tools
• Design and evaluation of preliminary set of WDM
  architecture designs
• Results summarized in recent conference paper
  presented at the 2006 MILCOM conference,
  Comparative Simulative Analysis of WDM LANs for
  Avionics Platforms.
• Our work and involvement in this area has been
  partially sponsored by the Naval Air Systems
  Command (NAVAIR)

4
Introduction to Architecture Design

• Design of an entire network architecture is a
  daunting task
• Requires a wide range of expertise to make
  intelligent decisions at every stage of
  development
• Key steps must be taken before designing the
  final product
• Before defining an architecture, define the
  requirements
• Impossible to design an architecture to best meet
  a users needs without knowing what their needs
  are
• Mission of the WDM LAN requirements task subgroup
• Learn from those who came before us
• Layered approach to architecture design
• Divide and conquer using the OSI protocol stack
  model
• List key design criteria at each layer of
  architecture development, and the solutions a WDM
  network can offer

• Illustrate networking principles applied in
  legacy networks
• How popular avionics networks meet their design
  requirements
• Which techniques used by previous networks are
  worth reusing today, in avionics and beyond

5
Introduction to WDM Networks

• WDM Wavelength Division Multiplexing
• Similar to FDM, but with light!
• Each wavelength is an independent channel
• Typical implementations can utilize up to 40
  wavelengths in one fiber, each 1-10 Gbps
• Most commonly used in telecom and long-haul
  networks
• WDM links ideal for high bandwidth trunks (SONET)
• Previously not considered for deployment in LAN
  settings
• Bandwidth overkill
• Higher costs associated with optical components
• Component capabilities rapidly maturing and
  decreasing in cost
• Modeling/Simulation work underway to design WDM
  LAN as future standard in avionics

Fig. 1 Lasers
6
Benefits of WDM

• Bandwidth scalable adding extra channels, more
  bandwidth/channel, does not require upgraded
  cable plant (potentially Terabits/s aggregate
  bandwidth)
• Protocol transparency different wavelengths
  can use different proto-cols without interfering
• Resistance to electromagnetic interference -
  unlike standard electrical networks optical
  networks are impervious to EMI
• Weight lighter cabling plant, since there is
  no need for electrical shielding
• Electronic Speed- all WDM devices only have to
  operate at electronic speed
• Security- Much more difficult to tap into an
  optical fiber undetected (more relevant in
  free-space optics)

Figure 2 EMP
7
Challenges of WDM

• Avionic temperature range
• Devices must operate from -55 C to 155 C. This is
  a difficult constraint for lasers since the
  output frequency tends to shift with temperature
• Lasers are directional
• Light will only propagate in one direction. This
  makes broadcasting more difficult than it is in
  electrical networks
• Packet Routing incurs an OEO delay
• Routing based on headers in the packet currently
  has to be done by performing and optical to
  electrical conversion and then converting back to
  optical. This conversion takes around a
  microsecond at each hop
• Wavelength routing is fixed
• Using current technologies wavelength routing
  follows a static routing table
• Collision detection
• Typically used in electrical networks, but
  expensive to do with optical components
• Large-scale all optical LANs are still under
  development

Fig. 5 Wavelength Router
8
WDM Architecture Design with the OSI Model
9
OSI Protocol Stack

• Open Systems Interconnection
• Developed by International Organization for
  Standardization (ISO)
• OSI Model contains seven layers
• 7-layer model illustrated on next slide
• Rare that a network architecture fully implements
  7 layers
• A theoretical system delivered too late!
• But useful for classifying communications
  functions
• Good starting point for architecture design
• Layers can be reorganized and pared down as
  needed during design process to fit our specific
  needs
• TCP/IP is de facto standard of prominence
• Is TCP/IP a protocol that future avionics
  networks should be targeted for and/or designed
  around?
• Pros Most widely accepted and popular networking
  protocol currently used, facilitate
  interoperability in future systems
• Cons Few legacy avionics applications designed
  for IP environment

10
OSI Model Illustration
11
Simplified Protocol Stack

• For embedded systems, a protocol implementing the
  full OSI stack may be unnecessary
• The requirements and complexity of an avionics
  network should be far less than large-scale data
  networks
• A simpler protocol architecture may be preferable
• How early in the standardization process should
  the layers of the WDM LAN be concretely defined?

Simplified 3-Layer Network Architecture
12
Physical (PHY) Layer

• Defines the physical interface between data
  transmission device (e.g. computer) and
  transmission medium or network
• The PHY layer is responsible for the transfer of
  bits between compliant nodes over the networks
  communication channel
• Some key characteristics defined at the WDM PHY
  Layer
• Characteristics of transmission medium, i.e.
  optical fiber
• Max length of fiber runs
• Fiber loss characteristics
• Optical transmitter power levels, receiver
  thresholds
• Connector and interface specifications
• Amplifier operation ranges
• Methods for establishing and releasing physical
  connections
• E.g. link training sequences
• Data rate(s)
• Wavelength ranges and spacing
• Synchronization methods
• Network topology
• The choice of physical components and topology
  will define abilities/limitations for
  communication system and upper layers
• CWDM vs. DWDM?

13
Typical WDM Network Components

• Laser Transmitter
• Injects optical signal into attached waveguide
• Optical Detector
• Converts optical streams back to electrical
  signals by counting photons absorbed
• WDM Mux/Demux
• Spatially combine/split wavelengths to or from a
  single channel
• Network Interfaces
• OTM Optical Terminal Multiplexers
• Connect clients to network through a single
  channel interface, terminate incoming optical
  signals
• OADM Optical Add/Drop Multiplexer
• Network access element that may add or remove
  signals from attached channel
• ROADM ReconfigurableOADM

A WDM Demux feeding an array of detectors
OADM illustration
14
Typical WDM Network Components

• Optical Couplers/Splitters
• Splits and/or combines a set of optical signals
• Often implemented by fusing fiber cores in a
  central region, optical power is then
  distributed to outputs
• OLA Optical Line Amplifiers
• Amplifies power of optical signal (and noise) on
  line to compensate for signal attenuation
• Essential for long-haul communication
• OXC Optical Cross-connect
• Distribute optical signals arriving on one of
  many inputs to one or more desired outputs
• Many different approachesto OXC design
• MEMS, SOA, OEO, etc.
• OBS Optical burst switch
• Optical device that performs high-speed
  sub-wavelength switching

1x8 splitter
Example of a 2x2 MEMS-based OXC
15
Calculation of Bit-Error Rate
Laser Calculations
Signal and noise power used as OL_PhyStats
parameters
Parameters of Laser transmitter
Detector Calculations
Intermediate components (fibers, amplifiers) will
alter the signal and noise of each message
Certain events will guarantee errors (BER 1)
e.g. offset wavelength, collisions, Psig lt
Pth OL_Packet data and the BER will be passed to
upper-layer network modules for error calcs.
16
Data Link Layer

• Provides the reliable exchange of blocks of data
  across a link that directly connects two nodes
• Sends frames with the necessary synchronization,
  error control, and flow control
• In the case where multiple transmitters share a
  broadcast medium, a medium-access control (MAC)
  policy is needed
• Some common multiple access (MA) protocols
• CSMA/CD (Ethernet)
• CSMA/CA (802.11, wireless LANs)
• Synchronous TDMA, w/ polling (Mil-1553)
• Statistical TDMA (POTS)
• Token Passing (Token Ring, Token Bus)
• The physical media will dictate feasible
  multiple access protocols
• Network requirements will define which
  strategies are preferred
• E.g. determinism more difficult is collision
  detection is implemented
• Multiple access policies typically not required
  in switched networks
• Can (or should) a WDM architecture be defined
  which doesnt require sub-wavelength multiple
  access policies?
• Simplicity at the cost of scalability and
  flexibility
• Will be highly dependant on wavelength
  assignment, protocol mapping

17
Data Link Layer

• Wavelength assignment how are wavelengths
  assigned/used in the WDM?
• For any approach, a mechanism is needed to ensure
  both transmitter and receiver are tuned to
  correct wavelength before communication
• Wavelength addressing assign wavelengths to
  destinations
• Can simplify design, as transmitters know desired
  wavelength a priori
• When does the size of the network outgrow this
  approach?
• Does the topology support reuse of local
  wavelengths?
• Dynamic wavelength allocation
• Increased overhead to continuously allocate
  wavelengths
• Offers a higher level of flexibility
• What control mechanisms exist to reassign
  wavelengths
• What wavelengths are reserved for fault
  conditions, control channels
• Flow control
• Used by receiving entities for limiting the rate
  of received traffic
• If traffic flow is deterministic, flow control
  need not be defined
• Error control
• Defines what happens when a frame transmission is
  unsuccessful
• Error detection bits attached to each frame to
  detect corrupted bits
• For military applications, is forward-error
  correction desirable?
• ARQ mechanism used to retransmit lost or damaged
  frames

18
Network Layer

• Used to transport packets across a communication
  network
• This layer governs the transfer of packets
  between any two nodes on the network, even those
  separated my multiple hops
• Routing protocols typically become a major
  network layer component
• Network layer functions are not needed on direct
  links
• Thus architectures (such as a single WDM ring)
  that do not rely on intermediate network
  components wont require network layer functions
• The network layer will be key if the WDM backbone
  is used to interconnect separate local networks
• E.g. connecting a group of MIL-1553 networks with
  the WDM LAN
• Internetworking protocols, if needed, will be
  directly linked to the architectures protocol
  transparency scheme
• Would require the definition of bridges or
  gateways between legacy networks and WDM LAN
• Additional functionality (may appear in other
  layers)
• Priorities and Quality of Service (QoS)
• Provide and ensure timely delivery of
  high-priority traffic
• Discussed further under connection-oriented
  services
• Security
• Separate sensitive and non-sensitive data,
  physically and/or logically

19
Transport Layer

• Transport layer controls the end-to-end transfer
  of messages between remote processes
• Uses services of lower layers to provide
  higher-level message transfer
• May also perform segmentation and reassembly of
  long messages
• Allows more efficient error control, more
  equitable access to network facilities, shorter
  delays, and smaller buffers needed
• Also leads to increased overheads, interrupts at
  receiver and processing time
• Transport protocols can vary from
  connection-based (e.g. TCP) to datagram-oriented
  (e.g. UDP)
• Connection-based transport services can include
  connection-setup, sequencing, flow control, error
  control, QOS
• Higher overhead, but offers reliable service to
  upper layers
• Datagram transports send data without connection
  establishment or acknowledgements, upper layers
  are responsible for dealing with errors
• Less overhead, less deterministic, additional
  functionality required above
• Transport-layer addressing needed to address data
  to peer process

20
Upper Layers

• Application layer provides services required by
  applications that involve communications
• Defines interface between the programmer and the
  network
• A set of function calls needs to be defined and
  provided to the application that utilizes the
  communication network
• Popular examples include HTTP, Telnet, SMTP, etc.
• Specialized embedded applications may not need a
  generic application interface, such as HTTP
• These applications likely interface with the
  architecture at the transport layer
• Legacy applications could maintain current forms
  if an interface is provided from their upper
  layers to the WDM LAN lower layers
• Discussed further in protocol transparency
  section
• Presentation layer keeps applications independent
  of differences in data representation
• Big vs. little endian
• Session and presentation layer often
  encapsulated in other layers

21
Selected Key Issues For an Avionics WDM LAN
22
Addressing

• What levels of addressing are required in our
  architecture?
• Is a network or node address enough? Probably
  not.

Example of addressing between applications over
TCP/IP
23
Packet-switched vs. Circuit-switched

• Characteristics of circuit-switched networks
• Dedicated communication path provided between two
  stations
• Effective for fixed-rate continuous data, such as
  voice streams
• Not ideal for data transmissions, which are often
  bursty and random in nature
• Three phases
• Circuit establishment, data transfer, circuit
  disconnect
• Characteristics of packet-switched networks
• Data transmitted in small packets
• e.g. packet length of 1000 octets
• Longer messages split into series of packets
• Each packet contains a portion of user data from
  message, plus some control info
• At each switching node
• Each packet received, stored briefly (buffered),
  and passed on to next node (store and forward)
• Offers more flexible performance and behavior
  than circuit switching
• Packets queued and transmitted as soon as possible

24
Packet-switched vs. Circuit-switched

• WDM technology is inherently circuit-switched
• Without reliable optical processing and optical
  buffering, in-transit processing of optical data
  is not feasible without O-E-O conversions
• The WDM LAN should provide the performance and
  flexibility of packet-switched architectures,
  while using technology and offering services that
  are circuit-switched
• Utilize immense bandwidth of WDM communications
  to realize this goal
• Connection-oriented services
• Packet-switched networks can offer
  connection-oriented services by reserving
  bandwidth and switching resources for virtual
  connections
• Hybrid protocols attempt to satisfy both periodic
  and sporadic data transmissions (e.g. ARINC 629)
• Use time cycles, where one portion is reserved
  for transferring periodic traffic, while the
  remainder of the cycle is used for sporadic data
  transmissions
• Nodes must contend for media during sporadic
  transmissions
• Priority schemes can also be used to achieve
  connection-oriented behavior
• When priority levels alone are used for realizing
  latency-bound transmissions, low priority traffic
  can be unfairly neglected

25
Achieving Deterministic Performance

• It is important that timely deliver of critical
  data can be guaranteed
• If the latency of real-time and mission-critical
  data is unbounded, the success of the mission is
  in jeopardy
• QoS mechanisms need to be implemented to provide
  such guarantees
• Centralized vs. distributed QoS
• Centralized QoS rely on network managers to
  administer and uphold QoS policies
• Intermediary connection elements are often in the
  best position to manage and monitor traffic flow
  for the network
• Simplifies node design, who only respond to flow
  control commands
• Distributed QoS policies are put in place at
  each node that ensure QoS requirements can be
  maintained
• Relieves the need for central QoS managers, ideal
  for direct-link networks
• Increased intelligence and planning required in
  nodes
• Several multiple access strategies can provide
  deterministic behavior
• Synchronous TDMA, statistical TDMA, token
  passing, etc.
• Collision avoidance is almost mandatory for
  deterministic networks

26
Fault-tolerance, Reliability, and Recovery

• System needs ability to detect and overcome
  faults in the network
• Different than error control, which deals with
  errors to individual bits (SEUs) or frames
• The requirements of the architecture will define
  the levels of redundancy and fault-tolerance
  provided
• What metric do we use to define the
  fault-tolerance of the network?
• The number of faults supported is not the only
  way to define fault-tolerance
• See metrics on next slide
• Identify and avoid single point of failures
• Add redundancy to single points of failures to
  increase fault-tolerance, or
• Increase reliability of such points to satisfy
  system reliability requirements
• Principle of Amdahls law
• Define a fault model
• How do we classify the faults we expect in our
  system?
• Link faults, node faults, lightpath faults, etc.
• Process for handling a fault
• How are faults detected?
• Who is notified of the fault, and how?
• What actions are taken to respond to detected
  fault?
• Analyzing the cost of fault-tolerance
• Switch count, port count or link count, power,
  etc.

27
Fault-tolerance, Reliability, and Recovery

• Key metrics in fault-tolerance
• Average number of faults till failure
• MTTF (Mean time to failure)
• MTTR (Mean time to recovery)
• Fault isolation (how many other components are
  affected by each fault)
• Calculating MTTF
• A component can be characterized by either its
  hazard function Z(t), or its reliability function
  R(t)
• For elements in series, the reliability functions
  are combined to determine system reliability
• For elements in parallel, the union of the
  reliability functions is the system reliability
• The final term prevents the instance of both
  elements working from being accounted for twice
  in the overall reliability
• Calculating MTTR will depend on recovery
  procedures

28
Realizing Protocol Transparency

• Desire for compatibility with legacy devices
• Allow legacy devices to communicate over single
  unified network
• WDM network offers potential for protocol
  transparency
• Is it as simple as mapping each protocol to a
  wavelength?
• Where do legacy protocols interface to the WDM
  LAN architecture?
• For pure transparency, the lower the better
• If protocols are simply mapped onto separate
  wavelengths, then the interface can be at the
  physical layer, requiring minimal modifications
  to existing devices
• Define aggregators to multiplex a number of
  legacy buses onto a single physical channel for
  better utilization
• On the other hand, legacy devices can not utilize
  services offered by the WDM LAN on layers above
  the insertion point
• Devices entering at the physical layer will rely
  on their own control schemes and services
• Compatibility of legacy protocols will heavily
  depend on the WDM LAN data link and network
  layers
• In cases where compatibility is not easily
  realized, legacy device payloads can be
  encapsulated into WDM LAN frames
• In this case, the flow and error control of the
  WDM LAN can be utilized, but adds extra an
  layer(s) and complexity to protocol stack
• Common legacy protocols in avionics systems
• MIL-STD-1553B (SAE AS-15531 commercial
  equivalent), Fibre Channel, AFDX (full-duplex
  Ethernet for avionics), ARINC 429, HSDB

29
Brief Introductions to Legacy Avionics
NetworksIncluding MIL-1553, ARINC 429, AFDX,
and Fibre Channel
30
MIL-STD-1553

• MIL-STD-1553 written in 1973 defines a TDM
  Command/Response Data Bus
• Widely popular in avionics for being simple and
  highly resilient
• Three network elements described by standard
• Each 1553 Bus can contain 31 remote terminals
  (RTs)
• Each 1553 Bus requires gt1 Bus Controller (BC)
• Controls pattern of communication
• Optional Bus Monitors allowed (BMs)
• Three word-types defined in 1553
• 20-bit word sizes used exclusively (16 bits of
  data)
• Command words specify functions to perform
• Data words used to transfer data between RTs
• Status words conveys the state of the RT after
  transmission
• 1553 uses a Command/Response protocol for
  communication
• All transfers initiated and controlled by the BC
• BC will poll each RT, status word reply informs
  BC of intent to transfer
• Most transfers go through BC, RT-RT messaging is
  rare
• Highly inefficient, as most data must be
  transmitted twice over the bus
• RTs must be prepared to respond to Command words
  at all times (defined response time)
• RTs transfer only in response to BC commands

31
ARINC 429

• A unidirectional bus, allowing only a single
  transmitter on each bus
• If bidirectional communication is needed,
  multiple buses are required
• Used almost exclusively on commercial platforms
  such as the Boeing B-767, Airbus A-310, etc.
• Such platforms can contain up to or more than 150
  ARINC 429 buses
• Two data rate ranges
• Low speed (12.0-14.5 kbps), and high speed (100
  kbps, rare)
• All data is transmitted is 32-bit words
• Each word includes an 8-bit label, and single
  parity bit at the end
• Words are expected to be sent periodically, even
  when no new information is available to transmit
• Often, the same data word will be transmitted
  repeatedly before updated information is
  available for transmission
• Two primary forms of data transfer in ARINC 429
• Standard broadcast protocol, all nodes may accept
  the transmitted data
• Each word represents the current status of the
  transmitting device
• Bit-oriented, or Williamsburg, protocol used for
  data file transfer over a 429 bus
• Handshaking used to setup and tear down
  bit-oriented transfer
• Data files segmented across a number of 429 data
  words
• System address label (SAL) is used to signify
  destination system
• Easy to aggregate multiple 429 networks on a
  single channel in WDM LAN

32
Avionics Full Duplex Ethernet (AFDX)

• Goal to provide a deterministic network utilizing
  a mature and proven networking technology,
  Ethernet
• AFDX requires a switched network supporting
  full-duplex links
• Eliminates possibility of collisions, which can
  cause non-deterministic behavior
• Deterministic performance fully realized by
  regulating outgoing traffic at each node,
  preventing network congestion
• The use of virtual links (VLs) differentiates
  AFDX from traditional Ethernet
• Frames on a VL can only originate from a single
  node
• Each VL is analogous to a ARINC 429 bus
• Two parameters with each VL limit outgoing
  traffic
• Bandwidth-allocation gap (BAG) - lower limit on
  delay between sending consecutive frame
• This limits variable and unpredictable queueing
  delays in switches
• Lowest defined BAG parameter is 1 ms, which may
  handicap future low-latency applications
• Maximum frame size
• Two parameters will define an upper limit on
  bandwidth generated by each VL
• Two independent networks are required, for
  increased reliability
• Each frame is sent over both networks
• Redundancy managers in nodes verify and handle
  redundant frames

33
Fibre Channel Avionics Environment

• Fibre Channel (FC) defines a generic transport
  for Upper Level Protocols (ULPs)
• Two primary forms of FC defined
• Switched Fabric (FC-SW)
• Arbitrated Loop (FC-AL)
• FC protocol independent of physical media
• The Fibre can be coax, optical fiber, etc.
• FC supports a Fabric Hierarchy to allow
  interoperability between FC implementations
• A Fabric defines the transport medium connecting
  end nodes in a FC network
• Regions define sections of a fabric that have
  compatible service parameters
• Regions and sub-Fabrics may be divided into
  Zones, independent partitions divided
  administratively
• 5 classes of service defined in FC
• Classes of service range from dedicated
  connections (class 1), to fractional
  connection-oriented (class 4), to unreliable
  datagram (class 3)
• FC supports strictly point-to-point communication
• Ports configured to communicate (only) with the
  addressed port attached to fibre
• Optical switches are not supported by FC
• Destination port is not constant for many optical
  switches

34
Summary

• Designers will rely on the users requirements
  for making crucial design decisions throughout
• Thus the high importance of the SAE requirements
  task group!
• The design and definition of a network
  architecture is too much for one person
• Architecture definition can be simplified by
  breaking it down into layers
• Define the layers of our network, the
  requirements of each layer, and the interfaces
  between them
• Accumulate experts for each level of the
  architecture
• Many issues are unique and critical for an
  avionics WDM LAN
• Topics such as wavelength assignment, protocol
  transparency, etc.
• Not all concepts can be neatly mapped into the
  layers of the stack
• Analyze and understand previous avionics networks
• Utilize the strengths that made them popular
• Figure out the ideal approach for incorporating
  these legacy protocols onto the WDM LAN
• Some additional figures and information are
  available in the appendix
• Thank you!