C Network Programming Mastering Complexity with ACE

1
C Network ProgrammingMastering Complexity with
ACE Patterns
Dr. Douglas C. Schmidt d.schmidt_at_vanderbilt.edu ww
w.cs.wustl.edu/schmidt/tutorials-ace.html
Professor of EECS Vanderbilt University
Nashville, Tennessee
2
Motivation Challenges of Networked Applications

• Observation
• Building robust, efficient, extensible
  concurrent networked applications is hard
• e.g., we must address many complex topics that
  are less problematic for non-concurrent,
  stand-alone applications

3
Presentation Outline
Cover OO techniques language features that
enhance software quality

• Patterns, which embody reusable software
  architectures designs
• ACE wrapper facades, which encapsulate OS
  concurrency network programming APIs

• OO language features, e.g., classes, dynamic
  binding inheritance, parameterized types

4
The Evolution of Information Technologies

• Extrapolating this trend to 2010 yields
• 100 Gigahertz desktops
• 100 Gigabits/sec LANs
• 100 Megabits/sec wireless
• 10 Terabits/sec Internet backbone

In general, software has not improved as rapidly
or as effectively as hardware
5
Component Middleware Layers

• Tedious, error-prone, costly over lifecycles

There are layers of middleware, just like there
are layers of networking protocols

• Standards-based COTS middleware helps
• Control end-to-end resources QoS
• Leverage hardware software technology advances
• Evolve to new environments requirements
• Provide a wide array of reuseable, off-the-shelf
  developer-oriented services

There are multiple COTS layers research/
business opportunities
6
Operating System Protocols

• Operating systems protocols provide mechanisms
  to manage endsystem resources, e.g.,
• CPU scheduling dispatching
• Virtual memory management
• Secondary storage, persistence, file systems
• Local remove interprocess communication (IPC)
• OS examples
• UNIX/Linux, Windows, VxWorks, QNX, etc.
• Protocol examples
• TCP, UDP, IP, SCTP, RTP, etc.

7
Host Infrastructure Middleware

• Host infrastructure middleware encapsulates
  enhances native OS mechanisms to create reusable
  network programming components
• These components abstract away many tedious
  error-prone aspects of low-level OS APIs

Domain-Specific Services
Common Middleware Services
Distribution Middleware
Host Infrastructure Middleware
8
Distribution Middleware

• Distribution middleware defines higher-level
  distributed programming models whose reusable
  APIs components automate extend native OS
  capabilities

Domain-Specific Services
Common Middleware Services
Distribution Middleware
Host Infrastructure Middleware
Distribution middleware avoids hard-coding client
server application dependencies on object
location, language, OS, protocols, hardware
9
Common Middleware Services

• Common middleware services augment distribution
  middleware by defining higher-level
  domain-independent services that focus on
  programming business logic

Domain-Specific Services
Common Middleware Services
Distribution Middleware
Host Infrastructure Middleware
10
Domain-Specific Middleware

• Domain-specific middleware services are tailored
  to the requirements of particular domains, such
  as telecom, e-commerce, health care, process
  automation, or aerospace

Domain-Specific Services
Common Middleware Services
Distribution Middleware
Host Infrastructure Middleware
11
Overview of Patterns
12
Overview of Pattern Languages

• Motivation
• Individual patterns pattern catalogs are
  insufficient
• Software modeling methods tools that just
  illustrate how, not why, systems are designed

• Benefits of Pattern Languages
• Define a vocabulary for talking about software
  development problems
• Provide a process for the orderly resolution of
  these problems, e.g.
• What are key problems to be resolved in what
  order
• What alternatives exist for resolving a given
  problem
• How should mutual dependencies between the
  problems be handled
• How to resolve each individual problem most
  effectively in its context
• Help to generate reuse software architectures

13
Taxonomy of Patterns Idioms
Type Description Examples
Idioms Restricted to a particular language, system, or tool Scoped locking
Design patterns Capture the static dynamic roles relationships in solutions that occur repeatedly Active Object, Bridge, Proxy, Wrapper Façade, Visitor
Architectural patterns Express a fundamental structural organization for software systems that provide a set of predefined subsystems, specify their relationships, include the rules guidelines for organizing the relationships between them Half-Sync/Half-Async, Layers, Proactor, Publisher-Subscriber, Reactor
Optimization principle patterns Document rules for avoiding common design implementation mistakes that degrade performance Optimize for common case, pass information between layers
14
The Layered Architecture of ACE
www.cs.wustl.edu/schmidt/ACE.html

• Features
• Open-source
• 200,000 lines of C
• 40 person-years of effort
• Ported to many OS platforms

• Large open-source user community
• www.cs.wustl.edu/schmidt/ACE-users.html

• Commercial support by Riverace
• www.riverace.com/

15
Sidebar Platforms Supported by ACE

• ACE runs on a wide range of operating systems,
  including
• PCs, e.g., Windows (all 32/64-bit versions),
  WinCE Redhat, Debian, SuSE Linux Macintosh
  OS X
• Most versions of UNIX, e.g., Solaris, SGI IRIX,
  HP-UX, Digital UNIX (Compaq Tru64), AIX, DG/UX,
  SCO OpenServer, UnixWare, NetBSD, FreeBSD
• Real-time operating systems, e.g., VxWorks, OS/9,
  LynxOS, Pharlap TNT, QNX Neutrino RTP, RTEMS
• Large enterprise systems, e.g., OpenVMS, MVS
  OpenEdition, Tandem NonStop-UX, Cray UNICOS
• ACE can be used with all of the major C
  compilers on these platforms
• The ACE Web site at http//www.cs.wustl.edu/schmi
  dt/ACE.html contains a complete, up-to-date list
  of platforms, along with instructions for
  downloading building ACE

16
Key Capabilities Provided by ACE
17
The Pattern Language for ACE

• Pattern Benefits
• Preserve crucial design information used by
  applications middleware frameworks components
• Facilitate reuse of proven software designs
  architectures
• Guide design choices for application developers

18
POSA2 Pattern Abstracts
Service Access Configuration Patterns The
Wrapper Facade design pattern encapsulates the
functions data provided by existing
non-object-oriented APIs within more concise,
robust, portable, maintainable, cohesive
object-oriented class interfaces. The Component
Configurator design pattern allows an application
to link unlink its component implementations at
run-time without having to modify, recompile, or
statically relink the application. Component
Configurator further supports the reconfiguration
of components into different application
processes without having to shut down re-start
running processes. The Interceptor architectural
pattern allows services to be added transparently
to a framework triggered automatically when
certain events occur. The Extension Interface
design pattern allows multiple interfaces to be
exported by a component, to prevent bloating of
interfaces breaking of client code when
developers extend or modify the functionality of
the component.
Event Handling Patterns The Reactor architectural
pattern allows event-driven applications to
demultiplex dispatch service requests that are
delivered to an application from one or more
clients. The Proactor architectural pattern
allows event-driven applications to efficiently
demultiplex dispatch service requests triggered
by the completion of asynchronous operations, to
achieve the performance benefits of concurrency
without incurring certain of its liabilities. The
Asynchronous Completion Token design pattern
allows an application to demultiplex process
efficiently the responses of asynchronous
operations it invokes on services. The
Acceptor-Connector design pattern decouples the
connection initialization of cooperating peer
services in a networked system from the
processing performed by the peer services after
they are connected initialized.
19
POSA2 Pattern Abstracts (contd)
Synchronization Patterns The Scoped Locking C
idiom ensures that a lock is acquired when
control enters a scope released automatically
when control leaves the scope, regardless of the
return path from the scope. The Strategized
Locking design pattern parameterizes
synchronization mechanisms that protect a
components critical sections from concurrent
access. The Thread-Safe Interface design pattern
minimizes locking overhead ensures that
intra-component method calls do not incur
self-deadlock by trying to reacquire a lock
that is held by the component already. The
Double-Checked Locking Optimization design
pattern reduces contention synchronization
overhead whenever critical sections of code must
acquire locks in a thread-safe manner just once
during program execution.
Concurrency Patterns The Active Object design
pattern decouples method execution from method
invocation to enhance concurrency simplify
synchronized access to objects that reside in
their own threads of control. The Monitor Object
design pattern synchronizes concurrent method
execution to ensure that only one method at a
time runs within an object. It also allows an
objects methods to cooperatively schedule their
execution sequences. The Half-Sync/Half-Async
architectural pattern decouples asynchronous
synchronous service processing in concurrent
systems, to simplify programming without unduly
reducing performance. The pattern introduces two
intercommunicating layers, one for asynchronous
one for synchronous service processing. The
Leader/Followers architectural pattern provides
an efficient concurrency model where multiple
threads take turns sharing a set of event sources
in order to detect, demultiplex, dispatch,
process service requests that occur on the event
sources. The Thread-Specific Storage design
pattern allows multiple threads to use one
logically global access point to retrieve an
object that is local to a thread, without
incurring locking overhead on each object access.
20
The Frameworks in ACE
ACE Framework Inversion of Control
Reactor Proactor Calls back to application-supplied event handlers to perform processing when events occur synchronously asynchronously
Service Configurator Calls back to application-supplied service objects to initialize, suspend, resume, finalize them
Task Calls back to an application-supplied hook method to perform processing in one or more threads of control
Acceptor-Connector Calls back to service handlers to initialize them after they are connected
Streams Calls back to initialize finalize tasks when they are pushed popped from a stream
21
Example Applying ACE in Real-time Avionics

• Goals
• Apply COTS open systems to mission-critical
  real-time avionics

• Key System Characteristics
• Deterministic statistical deadlines
• 20 Hz
• Low latency jitter
• 250 usecs
• Periodic aperiodic processing
• Complex dependencies
• Continuous platform upgrades

22
Example Applying ACE to Time-Critical Targets
23
Example Applying ACE to Large-scale Routers

• Goal
• Switch ATM cells IP packets at terabit rates

• Key System Characteristics
• Very high-speed WDM links
• 102/103 line cards
• Stringent requirements for availability
• Multi-layer load balancing, e.g.
• Layer 34
• Layer 5

www.arl.wustl.edu
24
Example Applying ACE to Hot Rolling Mills

• Goals
• Control the processing of molten steel moving
  through a hot rolling mill in real-time

• System Characteristics
• Hard real-time process automation requirements
• i.e., 250 ms real-time cycles
• System acquires values representing plants
  current state, tracks material flow, calculates
  new settings for the rolls devices, submits
  new settings back to plant

25
Example Applying ACE to Real-time Image
Processing

• Goals
• Examine glass bottles for defects in real-time

www.krones.com

• System Characteristics
• Process 20 bottles per sec
• i.e., 50 msec per bottle
• Networked configuration
• 10 cameras

26
Networked Logging Service Example

• Key Participants
• Client application processes
• Generate log records
• Server logging daemon
• Receive, process, store log records
• The logging server example in CNPv2 is more
  sophisticated than the one in CNPv1

• C code for all logging service examples are in
• ACE_ROOT/examples/ CNPv1/
• ACE_ROOT/examples/ CNPv2/

• Theres an extra daemon involved

27
Patterns in the Networked Logging Service
Leader/ Followers
Monitor Object
Active Object
Half-Sync/ Half-Async
Reactor
Pipes Filters
Acceptor- Connector
Component Configurator
Proactor
Wrapper Facade
Thread-safe Interface
Strategized Locking
Scoped Locking
28
ACE Basics Logging

• ACEs logging facility usually best for
  diagnostics
• Can customize logging sinks
• Filterable logging severities
• Portable printf()-like format directives
  (thread/process ID, date/time, types)
• Serializes output across multiple threads
• Propagates settings to threads created via ACE
• Can log across a network
• ACE_Log_Msg class use thread-specific singleton
  most of the time, via ACE_LOG_MSG macro
• Macros encapsulate most usage. Most common
• ACE_DEBUG ((severity, format , args))
• ACE_ERROR_RETURN ((severity, format
  ,args), return-value)
• See ACE Programmers Guide (APG) tables 3.1
  (severities), 3.2 (directives), 3.3 (macros)

29
ACE Logging Usage

• The ACE logging API is similar to printf(), e.g.
• ACE_ERROR ((LM_ERROR, "(t) fork failed"))
• generates
• Oct 31 145013 1992_at_ics.uci.edu_at_2766_at_LM_ERROR_at_cl
  ient(4) fork failed
• and
• ACE_DEBUG ((LM_DEBUG, "(t) sending to server
  s", host))
• generates
• Oct 31 145028 1992_at_ics.uci.edu_at_1832_at_LM_DEBUG_at_d
  rwho(6) sending to server tango

Format Action
l Displays the line number where the error occurred
N Displays the file name where the error occurred
n Displays the name of the program
P Displays the current process ID
p Takes a const char argument displays it the error string corresponding to errno (similar to perror())
T Displays the current time
t Displays the calling threads ID
30
Logging Severities

• You can control which severities are seen at run
  time using two masks
• Process-wide mask (defaults to all severities
  enabled)
• Per-thread mask (defaults to all severities
  disabled)
• Message is displayed is logged severity is
  enabled in either mask
• Set process/instance mask with
• ACE_Log_Msgpriority_mask (u_long mask,
  MASK_TYPE which)
• MASK_TYPE is ACE_Log_MsgPROCESS or
  ACE_Log_MsgTHREAD

• Per-thread mask initializer can be adjusted
  (default is all severities disabled)
• ACE_Log_Msgdisable_ debug_messages()
• ACE_Log_Msgenable_ debug_messages()
• These static methods set clear a (set of) bits
  instead of replacing the mask, as priority_mask()
  does
• Any set of severities can be specified (ORd
  together)

• Since default is to enable all severities
  process-wide, all severities are logged in all
  threads unless you change it
• See ACE_Logging_Strategy for interesting ways of
  changing it

31
Logging Severities Example

• To allow threads to decide their own logging, the
  desired severities must be
• Disabled at process level enabled in the
  thread(s) to display them.
• e.g.,
• ACE_LOG_MSG-gtpriority_mask (0, ACE_Log_MsgPROCES
  S)
• ACE_Log_Msgenable_debug_messages ()
• ACE_Thread_Managerinstance ()-gtspawn (service)
• ACE_Log_Msgdisable_debug_messages ()
• ACE_Thread_Managerinstance ()-gtspawn_n (3,
  worker)
• LM_DEBUG severity (only) logged in service thread
• LM_DEBUG severity (and all others) not logged in
  worker threads
• Note how severities are inherited when threads
  spawned

32
Redirect Logging to a File

• Default logging sink is stderr
• Redirect to a file by setting the OSTREAM flag
  assigning a stream
• Flag can be set in two ways
• ACE_Log_Msgopen (const ACE_TCHAR prog_name,
  u_long options_flags ACE_Log_MsgSTDERR,
  const ACE_TCHAR logger_key 0)
• ACE_Log_Msgset_flags (u_long flags)
• Assign a stream
• ACE_Log_Msgmsg_ostream (ACE_OSTREAM_TYPE
  )(Optional 2nd arg to tell ACE_Log_Msg to
  delete the ostream)
• ACE_OSTREAM_TYPE is ofstream where supported,
  else FILE
• To also stop output to stderr, use open() without
  STDERR flag, or ACE_Log_Msgclr_flags (STDERR)

33
Redirect Logging to Syslog

• Redirected log output to ACE_Log_MsgSYSLOG goes
  to
• Windows NT4 up systems Event Log
• UNIX/Linux syslog facility (uses LOG_USER syslog
  facility)
• Cant set this with set_flags/clr_flags must
  open. For example
• ACE_LOG_MSG-gtopen(argv0, ACE_Log_MsgSYSLOG,
  ACE_TEXT (syslogTest))
• Windows 3rd arg, if supplied, replaces 1st as
  program name in event log
• To turn it off, call open() again with different
  flag(s)
• This seems odd, but youre effectively resetting
  the logging think of it as reopen().

34
Logging Callbacks

• Logging callbacks are useful for adding special
  processing or filtering to log output
• Derive a class from ACE_Log_Msg_Callback
  reimplement
• virtual void log (ACE_Log_Record log_record)
• Use ACE_Log_Msgmsg_callback() to register
  callback
• Also call ACE_Log_Msgset_flags() to add
  ACE_Log_MsgMSG_CALLBACK flag
• Beware
• Callback registration is specific to each
  ACE_Log_Msg instance
• Callbacks are not inherited when new threads are
  created
• See ACE_ROOT/examples/Log_Msg for an example

35
Useful Logging Flags

• There are some other ACE_Log_Msg flags that add
  useful functionality to ACEs logging
• VERBOSE Prepends program name, timestamp, host
  name, process ID, message priority to each
  message
• VERBOSE_LITE Prepends timestamp message
  priority to each message (this is what ACE test
  suite uses)
• SILENT Dont display any messages of any
  severity
• LOGGER Write messages to the local client logger
  daemon
• Example of using VERBOSE_LITE
• ACE_DEBUG ((LM_DEBUG, ACE_TEXT ("This is ACE
  Version u.u.u\n\n"),
• ACE_MAJOR_VERSION, ACE_MINOR_VERSION,
  ACE_BETA_VERSION))
• Outputs
• Mar 29 095034.891 2005_at_LM_DEBUG_at_This is ACE
  Version 5.4.4

36
Tracing Logging

• ACEs tracing facility logs function/method entry
  exit
• Uses logging with severity LM_TRACE, so output
  can be selectively disabled
• Just put ACE_TRACE macro in the method
• include ace/Log_Msg.h
• void foo (void)
• ACE_TRACE (foo)
• // do stuff
• Says
• (1024) Calling foo in file test.cpp on line 3
• (1024) Leaving foo
• Clever indenting by call depth makes output
  easier to read
• Huge amount of output, so tracing no-opd out by
  default
• Enable by rebuilding with config.h having
  define ACE_NTRACE 0
• See ACE_ROOT/examples/Misc for an example

37
Networked Logging Service Example

• Key Participants
• Client application processes
• Generate log records
• Server logging daemon
• Receive, process, store log records
• Well develop architecture similar to ACEs, but
  not same implementation.

• C code for all logging service examples are in
• ACE_ROOT/examples/ CNPv1/
• ACE_ROOT/examples/ CNPv2/

38
Network Daemon Design Dimensions

• Communication dimensions address the rules, form,
  level of abstraction that networked
  applications use to interact
• Concurrency dimensions address the policies
  mechanisms governing the proper use of processes
  threads to represent multiple service
  instances, as well as how each service instance
  may use multiple threads internally

• Service dimensions address key properties of a
  networked application service, such as the
  duration structure of each service instance
• Configuration dimensions address how networked
  services are identified the time at which they
  are bound together to form complete applications

39
Communication Design Dimensions

• Communication is fundamental to networked
  application design
• The next three slides present a domain analysis
  of communication design dimensions, which address
  the rules, form, levels of abstraction that
  networked applications use to interact with each
  other
• We cover the following communication design
  dimensions
• Connectionless versus connection-oriented
  protocols
• Synchronous versus asynchronous message exchange
• Message-passing versus shared memory

40
Connectionless vs. Connection-oriented Protocols

• A protocol is a set of rules that specify how
  control data information is exchanged between
  communicating entities

SYN
SYN/ACK
ACK
Acceptor
Connector
3-way handshake in TCP/IP
Connection-oriented protocols provide a reliable, sequences, non-duplicated delivery service, which is useful for applications that cant tolerate data loss Examples include TCP ATM Connectionless protocols provide a message-oriented service in which each message can be routed delivered independently Examples include UDP IP

• Connection-oriented applications must address two
  additional design issues
• Data framing strategies, e.g., bytestream vs.
  message-oriented
• Connection multiplexing (muxing) strategies,
  e.g., multiplexed vs. nonmultiplexed

41
Alternative Connection Muxing Strategies

• In multiplexed connections all client requests
  emanating from threads in a single process pass
  through one TCP connection to a server process
• Pros Conserves OS communication resources, such
  as socket handles connection control blocks
• Cons harder to program, less efficient, less
  deterministic

• In nonmultiplexed connections each client uses a
  different connection to communicate with a peer
  service
• Pros Finer control of communication priorities
  low synchronization overhead since additional
  locks aren't needed
• Cons use more OS resources, therefore may not
  scale well in certain environments

42
Sync vs. Async Message Exchange

• Asynchronous request/response protocols stream
  requests from client to server without waiting
  for responses synchronously
• Multiple client requests can be transmitted
  before any responses arrive from a server
• These protocols therefore often require a
  strategy for detecting lost or failed requests
  resending them later

• Synchronous request/response protocols are the
  simplest form to implement
• Requests responses are exchanged in a lock-step
  sequence.
• Each request must receive a response
  synchronously before the next is sent

43
Message Passing vs. Shared Memory

• Message passing exchanges data explicitly via the
  IPC mechanisms
• Application developers generally define the
  protocol for exchanging the data, e.g.
• Format content of the data
• Number of possible participants in each exchange
  (e.g., point-to-point unicast), multicast, or
  broadcast)
• How participants begin, conduct, end a
  message-passing session

• Shared memory allows multiple processes on the
  same or different hosts to access exchange data
  as though it were local to the address space of
  each process
• Applications using native OS shared memory
  mechanisms must define how to locate map the
  shared memory region(s) the data structures
  that are placed in shared memory

44
Sidebar C Objects Shared Memory
Allocating a C Object in shared Memory void
obj_buf // Get a pointer to location in
shared memory ABC abc new (obj_buf) ABC //
Use C placement new operator

• General responsibilities using placement new
  operator
• Pointer passed to placement new operator must
  point to a memory region that is big enough is
  aligned properly for the object type being
  created
• The placed object must be destroyed by explicitly
  calling the destructor
• Pitfalls initializing C objects with virtual
  functions in shared memory
• The shared memory region may reside at a
  different virtual memory location in each process
  that maps the shared memory
• The C compiler/linker need not locate the
  vtable at the same address in different processes
  that use the shared memory
• ACE wrapper façade classes that can be
  initialized in shared memory must therefore be
  concrete data types
• i.e., classes with only non-virtual methods

45
Overview of the Socket API (1/2)
Sockets are the most common network programming
API available on operating system platforms

• Originally developed in BSD Unix as a C language
  API to TCP/IP protocol suite
• The Socket API has approximately two dozen
  functions classified in five categories
• Socket is a handle created by the OS that
  associates it with an end point of a
  communication channel

• A socket can be bound to a local or remote
  address
• In Unix, socket handles I/O handles can be used
  interchangeably in most cases, but this is not
  the case for Windows

46
Overview of the Socket API (2/2)
47
Taxonomy of Socket Dimensions
The Socket API can be decomposed into the
following dimensions

• Type of communication service
• e.g., streams versus datagrams versus connected
  datagrams
• Communication connection role
• e.g., clients often initiate connections
  actively, whereas servers often accept them
  passively
• Communication domain
• e.g., local host only versus local or remote host

48
Limitations with the Socket APIs (1/2)

• Poorly structured, non-uniform, non-portable
• API is linear rather than hierarchical
• i.e., the API is not structured according to the
  different phases of connection lifecycle
  management the roles played by the participants
• No consistency among the names
• Non-portable error-prone
• Function names read() write() used for any I/O
  handle on Unix but Windows needs ReadFile()
  WriteFile()
• Function semantics different behavior of same
  function on different OS e.g., accept () can take
  NULL client address parameter on Unix/Windows,
  but will crash on some operating systems, such as
  VxWorks
• Socket handle representations different
  platforms represent sockets differently e.g.,
  Unix uses unsigned integers whereas Windows uses
  pointers
• Header files Different platforms use different
  names for header files for the socket API

49
Limitations with the Socket APIs (2/2)

• Lack of type safety
• I/O handles are not amenable to strong type
  checking at compile time
• e.g., no type distinction between a socket used
  for passive listening a socket used for data
  transfer

• Steep learning curve due to complex semantics
• Multiple protocol families address families
• Options for infrequently used features such as
  broadcasting, async I/O, non blocking I/O, urgent
  data delivery
• Communication optimizations such as scatter-read
  gather-write
• Different communication connection roles, such
  as active passive connection establishment,
  data transfer

• Too many low-level details
• Forgetting to use the network byte order before
  data transfer
• Possibility of missing a function, such as
  listen()
• Possibility of mismatch between protocol
  address families
• Forgetting to initialize underlying C structures
  e.g., sockaddr
• Using a wrong socket for a given role

50
Example of Socket API Limitations (1/3)
Possible differences in header file names

• 1 include ltsys/types.hgt
• 2 include ltsys/socket.hgt
• 3
• 4 const int PORT_NUM 10000
• 5
• 6 int echo_server ()
• 7
• 8 struct sockaddr_in addr
• 9 int addr_len
• 10 char bufBUFSIZ
• 11 int n_handle
• 12 // Create the local endpoint.

Forgot to initialize to sizeof (sockaddr_in)
Use of non-portable handle type
51
Example of Socket API Limitations (2/3)

• 13 int s_handle socket (PF_UNIX, SOCK_DGRAM,
  0)
• 14 if (s_handle -1) return -1
• 15
• 16 // Set up address information where server
  listens.
• 17 addr.sin_family AF_INET
• 18 addr.sin_port PORT_NUM
• 19 addr.sin_addr.addr INADDR_ANY
• 20
• 21 if (bind (s_handle, (struct sockaddr )
  addr,
• 22 sizeof addr) -1)
• 23 return -1
• 24

Use of non-portable return value
Protocol address family mismatch
Wrong byte order
Unused structure members not zeroed out
Missed call to listen()
52
Example of Socket API Limitations (3/3)

• 25 // Create a new communication endpoint.
• 26 if (n_handle accept (s_handle, (struct
  sockaddr ) addr,
• 27 addr_len) ! -1)
• 28 int n
• 29 while ((n read (s_handle, buf, sizeof
  buf)) gt 0)
• 30 write (n_handle, buf, n)
• 31
• 32 close (n_handle)
• 33
• 34 return 0
• 35

SOCK_DGRAM handle illegal here
Reading from wrong handle
No guarantee that n bytes will be written
53
ACE Socket Wrapper Façade Classes

• ACE defines a set of C classes that address the
  limitations with the Socket API
• Enhance type-safety
• Ensure portability
• Simplify common use cases
• Building blocks for higher-level abstractions

These classes are designed in accordance with the
Wrapper Facade design pattern
54
The Wrapper Façade Pattern (1/2)

• Context
• Networked applications must manage a variety of
  OS services, including processes, threads, socket
  connections, virtual memory, files
• OS platforms provide low-level APIs written in C
  to access these services

• Problem
• The diversity of hardware operating systems
  makes it hard to build portable robust
  networked application software
• Programming directly to low-level OS APIs is
  tedious, error-prone, non-portable

55
The Wrapper Façade Pattern (2/2)

• Solution
• Apply the Wrapper Facade design pattern (P2) to
  avoid accessing low-level operating system APIs
  directly

This pattern encapsulates data functions
provided by existing non-OO APIs within more
concise, robust, portable, maintainable,
cohesive OO class interfaces
56
ACE Socket Wrapper Façades Taxonomy

• The structure of the ACE Socket wrapper facades
  reflects the domain of networked IPC properties
• The ACE Socket wrapper façade classes provide the
  following capabilities
• ACE_SOCK_ classes encapsulate Internet-domain
  Socket API functionality
• ACE_LSOCK_ classes encapsulate UNIX-domain
  Socket API functionality

• ACE also has wrapper facades for datagrams
• e.g., unicast, multicast, broadcast

57
Roles in the ACE Socket Wrapper Facade

• The active connection role (ACE_SOCK_Connector)
  is played by a peer application that initiates a
  connection to a remote peer
• The passive connection role (ACE_SOCK_Acceptor)
  is played by a peer application that accepts a
  connection from a remote peer
• The communication role (ACE_SOCK_Stream) is
  played by both peer applications to exchange data
  after they are connected

58
ACE Socket Addressing Classes (1/2)

• Motivation
• Network addressing is a trouble spot in the
  Socket API
• To minimize the complexity of these low-level
  details, ACE defines a hierarchy of classes that
  provide a uniform interface for all ACE network
  addressing objects

59
ACE Socket Addressing Classes (2/2)

• Class Capabilities
• The ACE_Addr class is the root of the ACE network
  addressing hierarchy
• The ACE_INET_Addr class represents TCP/IP
  UDP/IP addressing information
• This class eliminates many subtle sources of
  accidental complexity

60
ACE I/O Handle Classes (1/2)

• Motivation
• Low-level C I/O handle types are tedious,
  error-prone, non-portable
• Even the ACE_HANDLE typedef is still not
  sufficiently object-oriented typesafe

int buggy_echo_server (u_short port_num)
sockaddr_in s_addr int acceptor socket
(PF_UNIX, SOCK_DGRAM, 0) s_addr.sin_family
AF_INET s_addr.sin_port port_num
s_addr.sin_addr.s_addr INADDR_ANY bind
(acceptor, (sockaddr ) s_addr, sizeof s_addr)
int handle accept (acceptor, 0, 0) for ()
char bufBUFSIZ ssize_t n read
(acceptor, buf, sizeof buf) if (n lt 0)
break write (handle, buf, n)
int is not portable to Windows
Reading from wrong handle
61
ACE I/O Handle Classes (2/2)

• Class Capabilities
• ACE_IPC_SAP is the root of the ACE hierarchy of
  IPC wrapper facades
• It provides basic I/O handle manipulation
  capabilities to other ACE IPC wrapper facades
• ACE_SOCK is the root of the ACE Socket wrapper
  facades it provides methods to
• Create destroy socket handles

• Obtain the network addresses of local remote
  peers
• Set/get socket options, such as socket queue
  sizes,
• Enable broadcast/multicast communication
• Disable Nagles algorithm

62
The ACE_SOCK_Connector Class

• Motivation
• There is a confusing asymmetry in the Socket API
  between (1) connection roles (2) socket modes
• e.g., an application may accidentally call recv()
  or send() on a data-mode socket handle before
  it's connected
• This problem can't be detected until run time
  since C socket handles are weakly-typed

int buggy_echo_client (u_short port_num, const
char s) int handle socket (PF_UNIX,
SOCK_DGRAM, 0) write (handle, s, strlen (s)
1) sockaddr_in s_addr memset (s_addr,
0, sizeof s_addr) s_addr.sin_family
AF_INET s_addr.sin_port htons (port_num)
connect (handle, (sockaddr ) s_addr, sizeof
s_addr)
Operations called in wrong order
63
The ACE_SOCK_Connector Class

• Class Capabilities
• ACE_SOCK_Connector is factory that establishes a
  new endpoint of communication actively provides
  capabilities to
• Initiate a connection with a peer acceptor then
  to initialize an ACE_SOCK_Stream object after the
  connection is established
• Initiate connections in either a blocking,
  nonblocking, or timed manner

• Use C traits to support generic programming
  techniques that enable wholesale replacement of
  IPC functionality

64
Sidebar Traits for ACE Wrapper Facades (1/2)

• ACE uses the C generic programming idiom to
  define combine a set of characteristics to
  alter the behavior of a template class
• In C, the typedef typename language feature
  is used to define a trait
• A trait provides a convenient way to associate
  related types, values, functions with template
  parameter type without requiring that they be
  defined as members of the type
• Traits are used extensively in the C Standard
  Template Library (STL)

65
Sidebar Traits for ACE Wrapper Facades (2/2)

• ACE Socket wrapper facades use traits to define
  the following associations
• PEER_ADDR this trait defines the ACE_INET_Addr
  class associated with the ACE Socket Wrapper
  Façade
• PEER_STREAM this trait defines the
  ACE_SOCK_Stream data transfer class associated
  with the ACE_SOCK_Acceptor ACE_SOCK_Connector
  factories

class ACE_TLI_Connector public typedef
ACE_INET_Addr PEER_ADDR typedef
ACE_TLI_Stream PEER_STREAM // ...
class ACE_SOCK_Connector public typedef
ACE_INET_Addr PEER_ADDR typedef
ACE_SOCK_Stream PEER_STREAM // ...
66
Using the ACE_SOCK_Connector (1/3)

• This example shows how the ACE_SOCK_Connector can
  be used to connect a client application to a Web
  server

int main (int argc, char argv)
const char pathname argc gt 1 ?
argv1 /index.html" const char
server_hostname argc gt 2 ? argv2
www.dre.vanderbilt.edu" typedef
ACE_SOCK_Connector CONNECTOR CONNECTOR
connector CONNECTORPEER_STREAM peer
CONNECTORPEER_ADDR peer_addr if
(peer_addr.set (80, server_hostname) -1)
return 1 else if (connector.connect (peer,
peer_addr) -1)
return 1

• Instantiate the connector, data transfer,
  address objects

• Block until connection established or connection
  request failure

67
Using the ACE_SOCK_Connector (2/3)
// Designate a nonblocking connect. if
(connector.connect (peer,
peer_addr,
ACE_Time_Valuezero) -1) if (errno
EWOULDBLOCK) // Do some other work ...
// Now, try to complete connection
establishment, // but don't block if it
isn't complete yet. if (connector.complete
(peer, 0,
ACE_Time_Valuezero) -1)

• Perform a non-blocking connect

• If connection not established, do other work
  try again without blocking

// Designate a timed connect. ACE_Time_Value
timeout (10) // 10 second timeout. if
(connector.connect (peer,
peer_addr, timeout)
-1) if (errno ETIME) //
Timeout, do something else

• Perform a timed connect e.g., 10 seconds in this
  case

68
Using the ACE_SOCK_Connector (3/3)

• The ACE_SOCK_Connector can be passed the
  following values to control its timeout behavior

69
The ACE_SOCK_Stream Class (1/2)

• Motivation
• Developers can misuse sockets in ways that can't
  be detected during compilation
• An ACE_SOCK_Stream object can't be used in any
  role other than data transfer without violating
  its (statically type-checked) interface

int buggy_echo_server (u_short port_num)
sockaddr_in s_addr int acceptor socket
(PF_UNIX, SOCK_DGRAM, 0) s_addr.sin_family
AF_INET s_addr.sin_port port_num
s_addr.sin_addr.s_addr INADDR_ANY bind
(acceptor, (sockaddr ) s_addr, sizeof s_addr)
int handle accept (acceptor, 0, 0) for ()
char bufBUFSIZ ssize_t n read
(acceptor, buf, sizeof buf) if (n lt 0)
break write (handle, buf, n)
Reading from wrong handle
70
The ACE_SOCK_Stream Class (2/2)

• Class Capabilities
• Encapsulates data transfer mechanisms supported
  by data-mode sockets to provide the following
  capabilities

• Support for sending receiving up to n bytes or
  exactly n bytes
• Support for scatter-read, which populate
  multiple caller-supplied buffers instead of a
  single contiguous buffer

• Support for gather-write'' operations, which
  transmit the contents of multiple noncontiguous
  data buffers in a single operation
• Support for blocking, nonblocking, timed I/O
  operations
• Support for generic programming techniques that
  enable the wholesale replacement of functionality
  via C parameterized types

71
Using the ACE_SOCK_Stream (1/2)

• This example shows how an ACE_SOCK_Stream can be
  used to send receive data to from a Web server

• // ...Connection code from example in Section
  3.5 omitted...
• char bufBUFSIZ
• iovec iov3
• iov0.iov_base (char ) "GET "
• iov0.iov_len 4 // Length of "GET ".
• iov1.iov_base (char ) pathname
• iov1.iov_len strlen (pathname)
• iov2.iov_base (char ) " HTTP/1.0\r\n\r\n"
• iov2.iov_len 13 // Length of "
  HTTP/1.0\r\n\r\n"
• if (peer.sendv_n (iov, 3) -1)
• return 1
• for (ssize_t n (n peer.recv (buf, sizeof
  buf)) gt 0 )
• ACEwrite_n (ACE_STDOUT, buf, n)
• return peer.close () -1 ? 1 0

• Initialize the iovec vector for scatter-read
  gather-write I/O

• Perform blocking gather-write on ACE_SOCK_Stream

• Perform blocking read on ACE_SOCK_Stream

72
Using the ACE_SOCK_Stream (2/2)

• Blocking non-blocking I/O semantics can be
  controlled via the ACE_SOCK_STREAM enable()
  disable() methods, e.g.,
• peer.enable (ACE_NONBLOCK) // enables non
  blocking
• peer.disable (ACE_NONBLOCK) // disable non
  blocking
• If the I/O operation blocks, it returns a -1
  errno is set to EWOULDBLOCK
• I/O operations can involve timeouts, e.g.,
• ACE_Time_Value timeout (10) // 10 second timeout
• If (peer.sendv_n (iov, 3, timeout) -1)
• // check if errno is set to ETIME,
• // which indicates a timeout
• // similarly use timeout for receiving data

73
Sidebar Working with ( Around) Nagles Algorithm

• Nagles Algorithm
• Problem Need to tackle the send-side silly
  window syndrome, where small data payloads, such
  as a keystroke, result in transmissions of large
  packets causing unnecessary waste of network
  resources congestion
• Solution The OS kernel buffers a of
  small-sized application messages concatenates
  them into a larger size packet that can then be
  transmitted
• Consequences Although network congestion is
  minimized, it can lead to higher unpredictable
  latencies, as well as lower throughput
• Controlling Nagles Algorithm via ACE
• Use the set_option() method of the ACE_SOCK class
  e.g.,
• int nodelay 1 // Disable Nagles algorithm
• ACE_SOCK_Stream option_setter (handle)
• if (-1 option_setter.set_option
  (ACE_IPPROTO_TCP,
• TCP_NODELAY,
• nodelay,
• sizeof
  (nodelay)))
• ...

74
The ACE_SOCK_Acceptor Class (1/2)

• Motivation
• The C functions in the Socket API are weakly
  typed, which makes it easy to apply them
  incorrectly in ways that cant be detected until
  run-time
• The ACE_SOCK_Acceptor class ensures type errors
  are detected at compile-time

int buggy_echo_server (u_short port_num)
sockaddr_in s_addr int acceptor socket
(PF_UNIX, SOCK_DGRAM, 0) s_addr.sin_family
AF_INET s_addr.sin_port port_num
s_addr.sin_addr.s_addr INADDR_ANY bind
(acceptor, (sockaddr ) s_addr, sizeof s_addr)
int handle accept (acceptor, 0, 0) for ()
char bufBUFSIZ ssize_t n read
(acceptor, buf, sizeof buf) if (n lt 0)
break write (handle, buf, n)
Reading from wrong handle
75
The ACE_SOCK_Acceptor Class (2/2)

• Class Capabilities
• This class is a factory that establishes a new
  endpoint of communication passively provides
  the following capabilities

• It accepts a connection from a peer connector
  then initializes an ACE_SOCK_Stream object after
  the connection is established
• Connections can be accepted in either a blocking,
  nonblocking, or timed manner
• C traits are used to support generic
  programming techniques that enable the wholesale
  replacement of functionality via C
  parameterized types

76
Using the ACE_SOCK_Acceptor

• This example shows how an ACE_SOCK_Acceptor
  ACE_SOCK_Stream can be used to accept connections
  send/receive data to/from a web client

extern char get_url_pathname (ACE_SOCK_Stream
) int main () typedef ACE_SOCK_Acceptor
ACCEPTOR ACCEPTORPEER_ADDR server_addr
ACCEPTOR acceptor ACCEPTORPEER_STREAM
peer if (server_addr.set (80) -1) return
1 if (acceptor.open (server_addr) -1)
return 1 for () if (acceptor.accept
(peer) -1) return 1 peer.disable
(ACE_NONBLOCK) // Ensure blocking ltsend_ngt.
ACE_Auto_Array_Ptrltchargt pathname
(get_url_pathname (peer)) ACE_Mem_Map
mapped_file (pathname.get ()) if
(peer.send_n (mapped_file.addr (),
mapped_file.size ()) -1) return 1
peer.close () return acceptor.close ()
-1 ? 1 0

• Instantiate the acceptor, data transfer,
  address objects

• Initialize a passive mode endpoint to listen for
  connections on port 80

• Accept a new connection

• Send the requested data

• Close the connection to the sender

• Stop receiving any connections

77
Sidebar The ACE_Mem_Map Class

• Memory Mapped Files
• Many modern operating systems provide a mechanism
  for mapping a files contents directly into a
  processs virtual address space
• This memory-mapped file mechanism can be read
  from or written to directly by referencing the
  virtual memory
• e.g., via pointers instead of using less
  efficient I/O functions
• The file manager defers all read/write operations
  to the virtual memory manager
• Contents of memory mapped files can be shared by
  multiple processes on the same machine
• It can also be used to provide a persistent
  backing store

• ACE_Mem_Map Class
• A wrapper façade that encapsulates the memory
  mapped file system mechanisms on different
  operating systems
• Relieves application developers from having to
  manually perform bookkeeping tasks
• e.g., explicitly opening files or determining
  their lengths
• The ACE_Mem_Map class offers multiple
  constructors with several signature variants

78
The ACE_Message_Block Class (1/2)
MESSAGES BUFFERED AWAITING PROCESSING
MESSAGES BUFFERED FOR TRANSMISSION
MESSAGES IN TRANSIT

• Motivation
• Many networked applications require a means to
  manipulate messages efficiently, e.g.

• Storing messages in buffers as they are received
  from the network or from other processes
• Adding/removing headers/trailers from messages as
  they pass through a user-level protocol stack
• Fragmenting/reassembling messages to fit into
  network MTUs
• Storing messages in buffers for transmission or
  retransmission
• Reordering messages that were received
  out-of-sequence

79
The ACE_Message_Block Class (2/2)

• Class Capabilities
• This class is a composite that enables efficient
  manipulation of messages via the following
  operations
• Each ACE_Message_Block contains a pointer to a
  reference-counted ACE_Data_Block which in turn
  points to the actual data associated with a
  message

• It allows multiple messages to be chained
  together into a composite message
• It allows multiple messages to be joined together
  to form an ACE_Message_Queue
• It treats synchronization memory management
  properties as aspects

80
Allocators ACE_Message_Block

• You can set new allocation strategies for
• Allocating ACE_Message_Block objects
• Allocating ACE_Data_Block objects referred to by
  an ACE_Message_Block
• Allocating the memory chunk controlled by
  ACE_Data_Block
• They all default to ACE_Allocatorinstance(),
  which defaults to ACE_New_Allocator, i.e., C
  operator new
• Any class conforming to the ACE_Allocator
  interface may be used, including
• ACE_New_Allocator uses C operator new
• ACE_Static_Allocator uses a fixed pool of
  fixed-size blocks
• ACE_Cached_Allocator uses operator new, but
  caches blocks
• ACE_Allocator_Adapter adapts other pools, such
  as shared memory, to ACE_Allocator interface

81
Two Kinds of Message Blocks

• Composite messages contain multiple
  ACE_Message_Blocks
• These blocks are linked together in accordance
  with the Composite pattern
• Composite messages often consist of a control
  message that contains bookkeeping information
• e.g., destination addresses, followed by one or
  more data messages that contain the actual
  contents of the message
• ACE_Data_Blocks can be referenced counted

• Simple messages contain a one ACE_Message_Block
• An ACE_Message_Block points to an ACE_Data_Block
• An ACE_Data_Block points to the actual data
  payload

82
ACE_Message_Block Areas

• There are a number of size-related methods to
  obtain these values
• length() Amount of data between rd_ptr()
  wr_ptr()
• size() Total amount of useable allocated area
• capacity() Total amount of allocated area
  (usually same as size(), only different if size
  is decreased after creating the block)
• space() Free space after the wr_ptr() how many
  bytes can be added

wr_ptr()
end()
base()
rd_ptr()
length()
space()
size()
capacity()
83
Using the ACE_Message_Block (1/2)

• The following program reads all data from
  standard input into a singly linked list of
  dynamically allocated ACE_Message_Blocks
• These ACE_Message_Blocks are chained together by
  their continuation pointers

• Allocate an ACE_Message_Block whose payload is of
  size BUFSIZ

int main (int argc, char argv)
ACE_Message_Block head new ACE_Message_Block
(BUFSIZ) ACE_Message_Block mblk head
for (size_t recvd 0 recvd 0) ssize_t
nbytes ACEread_n (ACE_STDIN,
mblk-gtwr_ptr (),
mblk-gtsize (), recvd) if (nbytes lt 0)
break // Break out at EOF or error.
mblk-gtwr_ptr (recvd)

• Read data from standard input into the message
  block starting at write pointer (wr_ptr ())

• Advance write pointer by the number of bytes read
  to end of buffer

84
Using the ACE_Message_Block (2/2)

• Allocate a new ACE_Message_Block of size BUFSIZ
  chain it to the previous one at the end of the
  list

mblk-gtcont (new ACE_Message_Block (BUFSIZ))
mblk mblk-gtcont () // Print the
contents of the list to the standard output.
for (mblk head mblk ! 0 mblk mblk-gtcont
()) ACEwrite_n (ACE_STDOUT, mblk-gtrd_ptr
(), mblk-gtlength ()) head-gtrelease ()
// Release all the memory in the chain. return
0

• Advance mblk to point to the newly allocated
  ACE_Message_Block

• For every message block, print mblk-gtlength()
  amount of contents starting at the read pointer
  (rd_ptr ())
• Can also use ACEwrite_n (head) to write entire
  chain

85
ACE CDR Streams

• Motivation
• Networked applications that send/receive messages
  often require support for

class ACE_Log_Record ... private ACE_UINT
type_ ACE_UINT pid_ ACE_Time_Value
timestamp_ char msg_data_ACE_MAXLOGMSGLEN

• Linearization
• To handle the conversion of richly typed data
  to/from raw memory buffers
• (De)marshaling
• To interoperate with heterogeneous compiler
  alignments hardware instructions with different
  byte-orders

86
ACE CDR Streams (contd)

• The ACE_OutputCDR ACE_InputCDR classes provide
  a highly optimized, portable, convenient means
  to marshal demarshal data

• These classes use the standard OMG CORBA Common
  Data Representation (CDR)
• Unlike blindly always sending in network byte
  order, CDR uses a receiver makes right approach

• ACE_OutputCDR creates a CDR buffer from a data
  structure (marshaling)

• ACE_InputCDR extracts data from a CDR buffer
  (demarshaling)

87
The ACE_OutputCDR ACE_InputCDR Classes

• Class Capabilities
• ACE_OutputCDR ACE_InputCDR support the
  following features
• Operations to (de)marshal types
• Primitive types, e.g., booleans 16-, 32-,
  64-bit integers 8-bit octets single double
  precision floating point numbers characters
  strings
• Arrays of primitive types
• The insertion (ltlt) extraction (gtgt) operators
  can marshal demarshal primitive types, using
  the same syntax as the C iostream components
• ACE_Message_Block chains are used internally to
  minimize mem copies

• Use CORBA CDR alignment byte-ordering rules to
  avoid memory copying byte-swapping operations,
  respectively
• Provide optimized byte swapping code that uses
  inline assembly language instructions for common
  hardware platforms (such as Intel x86) standard
  hton() ntoh() macros/functions on other
  platforms
• Support zero copy marshaling demarshaling of
  octet buffers
• Users can define custom character set translators
  for platforms that do not use ASCII or Unicode as
  their native character sets

88
Sidebar Log Record Message Structure

• This example uses a 8-byte, CDR encoded header
  followed by the payload
• Header includes byte order, payload length,
  other fields

ACE_Log_Record is a type that ACE uses internally
to keep track of the fields in a log record
class ACE_Log_Record private ACE_UINT
type_ ACE_UINT pid_ ACE_Time_Value
timestamp_ char msg_data_ // Defaults to
ACE_MAXLOGMSGLEN public ACE_UINT type ()
const ACE_UINT pid () const const
ACE_Time_Value timestamp () const const char
msg_data () const
89
Using ACE_OutputCDR

• We show the ACE CDR insertion (operatorltlt)
  extraction (operatorgtgt) operators for
  ACE_Log_Record that's used by client application
  logging server

int operatorltlt (ACE_OutputCDR cdr,
const ACE_Log_Record log_record) size_t
msglen log_re