Building High Throughput, Multithreaded Servers in C

1
Building High Throughput, Multi-threaded Servers
in C/.NET

• David Buksbaum

2
Who Am I?

• Wall Street Wage Slave
• Developer I still write code
• Tech-Lead My bugs are harder to find
• Pointy Haired Manager I live by the KISS
  principle

3
What have I built?

• User Time to Near Real Time Systems
• User Time event time is based on user
  interaction
• Near Real Time event timing is as fast as
  possible without guarantee
• Server to Server
• Client - Server
• Small Scale 1-25 concurrent users
• Large Scale 250 concurrent users
• Distributed Systems (now called SOA)

4
Agenda

• C Crash Course
• Application Servers
• Concepts
• Connection Stack
• Conclusion
• Further Concepts

5
C Crash Course

• external

6
Application Servers

• What I mean by App Servers
• A stateful server to service client requests for
  high-level business functions in near user time
• RRE Messaging Model (request, response, event)
• Middle Tier Intra-Net Systems
• May contain 1 or more logical servers
• What I do not mean by App Servers
• Not the back-end infrastructure systems
• Broadcast Servers (pricing, curve servers, etc)
• Bus Servers (messaging platforms like Jabber)
• Database Servers

7
Concepts

• Endian-ness
• Does not matter unless your network is exactly ½
  little and ½ big
• Pick the model used more and make the other
  convert
• Only the smaller group will have the performance
  hit
• Most clients to your servers will be
  little-endian
• Many middle-tier systems are Intel based
• Solaris is being moved further back or out due to
  costs
• Typical configurations will be WinTel clients
  with either WinTel or LinTel servers

8
Concepts cont.

• Threading
• Your server functionality will determine
  threading or not
• Guideline If you threads on WinTel, then thread
  count target should be (2 CPUs) 1
• Explained in detail by Jeffery Richter in
  "Programming Server Side Applications for Windows
  2000"
• Exceptions Lots of I/O bound threads for
  example, so use your judgment
• Burst servers should be guided towards single
  threaded per connection (min concurrency)
• Servers with minimal state, fast response time,
  and no persistent connections
• HTTP 1.0 defaulted to a burst server (no
  keepalives)
• What is a burst server? echo, time, etc
• Pattern UDP or short lived TCP

9
Concepts cont.

• Threading cont.
• Stateful servers need threads so use them
• State operations are slower than network
  operations, so minimize it in the network thread
• If you use threads, use thread pools
• Creating threads are expensive, so dont do it
• .NET has a built in thread pool
• System.Threading.ThreadPool
• ThreadPool.QueueUserWorkItem(new
  WaitCallback(myThreadFunc), myContextObject)
• In large systems, use multiple specialized pools
• Different Priorities, Order Guarantees, etc

10
Concepts cont.

• Queues
• Rule 1 If you use threads you use queues
• Queues break performance coupling between two
  independent tasks
• Task 1 Read / Write Data From / To Network
• Task 2 Do Something with the Data
• Queues allow for complex tasks to be sub-divided
  into simpler sub-tasks
• These sub-tasks can be worked on concurrently
  (parallelism or grids) or synchronously
• Reusability of sub-tasks between different parent
  tasks
• eg fetch from database, calculate distance
  between two points on a map, execute a trade
• Simpler code is easier to build, debug, maintain

11
Concepts cont.

• Synchronicity
• Blocking Synchronous I/O
• Read or Write w/o timeouts
• Select w/o timeouts
• Non-Blocking Synchronous I/O
• Read or Write w/ timeouts
• Select w/ timeouts
• Polling
• Asynchronous I/O
• BeginReceive or BeginSend
• I/O Completion Ports Overlapped I/O

12
Concepts cont.

• Synchronicity cont.
• Which do you use?
• Rule 2 Never, Ever, Use Blocking I/O
• Server Threading model defines synchronicity
  strategy
• Thread per connection ? Non-Blocking Synchronous
  I/O
• Thread pools ? Asynchronous I/O
• Clients can usually use Non-Blocking Synchronous
  I/O in a dedicated thread

13
Concepts cont.

• State
• Definition
• Data and Methods encapsulating a logic block
  related information
• An App Server has different types of state
• Server State
• State managed on a per logical server basis
• Used anywhere the server needs to self administer
  itself
• Usually the container of all other state in the
  server
• Connection State
• State managed on a per user connection basis
• Used only in a context where the connection needs
  to be known
• Reading and Writing to the network
• Verifying credentials

14
Concepts cont.

• State cont.
• Business State
• Contains all the cached data used to support the
  business functionality
• Provides access to all non-cached data
  transparently
• eg BusinessState.GetAllAccounts() may go to the
  cache or some external store. Follow good OO
  procedures here.
• State Rules
• Minimize surface area (less public methods), but
  no side effects
• Everything public MUST be reentrant or
  thread-safe
• Encapsulate no public variables
• Design for threading

15
Concepts cont.

• State cont.
• Rule 3 State will dominate server design
  performance
• Questions to ask when defining state
• Who needs access to it?
• When do they need access to it?
• Special Cases frequent pervasive
• Logon, Logoff
• Credentials Check

16
Concepts cont.

• State cont.
• Techniques
• Synchronized Collections
• Equal punishment to readers writers
• Reader / Writer Locks
• Favors reads over writes
• External Data Store
• Push the locking burden on an expert
• Embedded
• All state is passed between the client and server
  in every message
• eg URLs, Cookies, etc
• Tokens
• A token for looking up the state is passed
  between the client server in every message
• Usually used to add state to a stateless design
  (eg HTTP)
• Hybrids
• Combinations of the above
• eg Tokens, Embedded, External are used by many
  web servers

17
Concepts cont.

• Performance
• User Time vs. Actual Time
• User Time is the time between the initiation of a
  process by the user on the client and when the
  result of that action is visible to the user
• Actual Time is the actual time it takes to
  perform the given action on the server from
  receipt of a request to the sending of a response
• Throughput of the server is often defined in
  terms of actual time (messages per second)
• Rule 4a The user does not care about Throughput
• Rule 4b User Time is all that matters in
  interactive systems

18
Concepts cont.

• Performance cont.
• Users vs. Actions
• How many concurrent users?
• How many concurrent actions?
• Similar to Time vs. Memory tradeoff
• eq 1 user with 100 actions or 100 users with 1
  action

19
Concepts cont.

• Case Example Massive Multiplayer Online
  Role-Playing Games (MMORPG) Everquest 2
• 2500 concurrent users per server
• Each user requires significant state management
• Location in the game world
• Character statistics
• Inventory
• Relationships to other characters, non-player
  characters, organizations
• State is a hybrid combining tokens external
  data store
• User time is between 11 140 ms, with the bulk
  of the users being at around 100 ms derived
  from frame rates
• Actual time peak is 25000 / second (2500 10)
• Assumes all 2500 are not just concurrent, but
  active

20
Questions?
21
Architecture _at_ 50,000 Feet
User Time
Queue
Task Thread Pool
P1
P2
P3
Network Thread Pool
Network
Actual Time
Initial Response Time

• P1, P2 and P3 represent the areas that have the
  possibility for high contention due to locking or
  congestion
• A fast initial response time, followed by
  incremental updates (streaming or events), can
  give the illusion of faster user time

22
Protocol Stack _at_ 50,000 Feet

• Wire
• TCP, UDP, or other network protocol
• Transport
• Envelope to encapsulate the application message
• Fields
• Length, Checksum, Flags, Data
• Application
• Message ID
• Message Data

Application
Transport
Wire
23
Wire Protocols

• UDP
• Not reliable
• Small blocks of data
• Connectionless
• TCP
• Reasonably reliable
• Any reasonable amount of data
• No information about content

24
Wire Protocols cont.

• Message Queues
• Queue to Queue communications
• Each end point has a queue
• Messages are moved from one queue to another
• A centralized queue can have 1 or more listeners
  in non-exclusive mode
• A centralized queue in exclusive mode can have
  auto-failover
• Reliability level is controllable
• Products
• JMS Java Messaging Service
• .NET bindings exists
• Derivative works like Tibcos EMS is a very
  performant version
• Deployable on top of most network infrastructures
• MSMQ
• MS has .NET bindings in the System.Messaging
  namespace
• Not available on all Windows platforms by default
• Many organizations have restricted this product
  due to tight integration with the Windows platform

25
Wire Protocols cont.

• Message Queues cont.
• Alternative Systems
• Book Message Passing Server Internals by Bill
  Blunden
• Contains the design and implementation of
  Bluebox an enterprise ready messaging queuing
  system in Java
• Others
• Tibco Rendezvous
• Basically a serialized hash table (key, item)
• Jabber
• Java framework originally geared for Instant
  Messaging, but now supporting robust bus based
  communications
• Still XML based, so not a great fit for high
  performance

26
Wire Protocols cont.

• Why not UDP?
• Packet size limitations
• Connectionless
• Reliability
• Why not TCP?
• Reasonable reliability is not always good enough
• Performance in broadcast scenarios (1 to many
  listeners)

27
Wire Protocols cont.

• Why messaging systems?
• Reliable recoverable queues are great for when
  the message absolutely must get to its
  destination
• Trading systems
• Banking systems
• Transactional
• Queues can be used as the back bone of a system
  that support distributed functionality with
  distributed transactions (XA)
• Your app sever can participate in a transaction
  that spans all systems between the client and the
  database

28
Wire Protocols cont.

• Why messaging systems? cont.
• Support tools
• Many messaging systems provide rich tools for
  sniffing and/or capturing the messages
• Captured messages can be used for playback
• Robust infrastructure
• Messaging systems have been built by networking
  experts to accommodate a number of different
  infrastructures, topologies, and geographical
  idiosyncrasies
• You have someone to scream at when it goes wrong
• Rule 5 If you can, use a commercial transport

29
Transport Protocol

• Why do you need a protocol between the
  application and the wire?
• Wire Insulation The wire may change
• TCP ? Message Queues
• UDP ? TCP
• The application protocol may not play nicely into
  a wire protocol
• MB sized files
• Streaming data
• XML
• Variable length records

30
Transport Protocol cont.

• Remember our goal at the wire?
• Get the data off the wire and onto the queue as
  fast as possible
• We need information as we get the data off the
  wire to speed up processing
• Length of the total message
• Message data
• Smallest transport protocol (4 bytes)
• 4 byte length n byte data block

31
Transport Protocol cont.

• What else do we need?
• We have
• Length
• Data
• We need (loosely used)
• Message type information
• Message ID
• Sequencing information
• Beginning
• Middle
• End
• Flags or Options

32
Transport Protocol cont.

• DIME Lite
• Protocol created by Microsoft to allow for easier
  transmission of binary data over SOAP
• Can be used independently of SOAP as a
  unidirectional transport block

33
Transport Protocol cont.
34
Application Protocol

• What is it?
• It is anything meaningful to your servers
  business logic
• Hash Tables
• Schemas can be used to define required keys
• eg Default schema may mandate key 100 in all
  messages
• Key 100 might be the schema id for the rest of
  the message
• Allows for complex record based pseudo-objects
  with mandatory, optional, default fields
• Compatible cross language / platform
• Efficient
• Client objects can wrap hash table

35
Application Protocol cont.

• What is it? cont.
• Serialized Objects
• Complex message object or hierarchy of objects
• Very rich distributed object functionality
• Limited to .NET environment and possible subset
  of languages within the .NET world
• eg Java RMI, some transport for .NET Remoting
• SOAP, Raw XML, or other text based transport
• Schema or well defined set of tag / value pairs
• String based
• Changes require new memory block for the whole
  message since strings are immutable
• Slow to parse (350 / sec) validating, msxml
• Even the fastest is less than 1000 / sec

36
Application Protocol cont.

• What is it? cont.
• Hybrids
• Serialized objects encoded in hash tables
• Flexibility of hash tables
• Expandable through new keys
• Cross platform / language support
• Versionable objects
• Main limitation is the primitive types supported
• Requires well known schemas

37
Complete Example
Simple RPC Client / Server
38
Questions?
39
Further Thoughts

• Lock / Wait Free Data Structures
• Used in graphics pipelines and media streaming
• Reduces contention
• Aimed at small of writers and any number of
  readers (where you would use Reader / Writer
  locks)
• Protocols
• Current trends
• Ignore wire performance and go for B2B protocols
  (SOAP, et al)
• Text based protocols, such as ASCII encoded key /
  value pairs
• Serialization
• No simple way to work cross language / platform
• Primitives are not always portable
• Most research is focused on B2B

40
Further Thoughts cont.

• Memory Pools
• Pre-allocate pools of objects that can are reused
  rather than freed

41
Rules Summary

• Rule 1 If you use threads you use queues
• Rule 2 Never, Ever, Use Blocking I/O
• Rule 3 State will dominate server design
  performance
• Rule 4a The user does not care about Throughput
• Rule 4b User Time is all that matters in
  interactive systems
• Rule 5 If you can, use a commercial transport

42
References Links

• HTTP 1.0 RFC 1945
• http//www.ietf.org/rfc/rfc1945.txt
• HTTP 1.1 RFC 2616
• http//www.ietf.org/rfc/rfc2616.txt
• Echo Protocol RFC 862
• http//ftp.rfc-editor.org/in-notes/rfc862.txt
• Time Protocol RFC 868
• http//ftp.rfc-editor.org/in-notes/rfc868.txt
• Superseded by Network Time Protocol (NTP)
• http//en.wikipedia.org/wiki/Network_time_protocol

43
References Links cont.

• Endianness
• http//en.wikipedia.org/wiki/Endian
• .NET Thread Pools
• MSDN Docs for ThreadPool Class
• http//msdn.microsoft.com/library/default.asp?url
  /library/en-us/cpref/html/frlrfsystemthreadingthre
  adpoolclasstopic.asp
• Stephen Toubs Managed Thread Pool
• http//www.gotdotnet.com/Community/UserSamples/Det
  ails.aspx?SampleGuidbf59c98e-d708-4f8e-9795-8bae1
  825c3b6
• Enhanced version of Stephen Toubs Thread Pool
• http//www.codeproject.com/csharp/SmartThreadPool.
  asp

44
References Links cont.

• Overlapped I/O
• http//msdn.microsoft.com/library/default.asp?url
  /library/en-us/dllproc/base/synchronization_and_ov
  erlapped_input_and_output.asp
• I/O Completion Ports
• http//msdn.microsoft.com/library/default.asp?url
  /library/en-us/fileio/fs/i_o_completion_ports.asp

45
References Links cont.

• Message Queues
• Message Passing Server Internals by Bill Blunden
• http//www.amazon.com/gp/product/0071416382/103-63
  38215-6207008?vglancen283155
• Jabber
• http//www.jabber.org/
• DIME
• http//msdn.microsoft.com/library/en-us/dnglobspec
  /html/draft-nielsen-dime-02.txt

46

• Lock / Wait Free Data Structures
• http//www.audiomulch.com/rossb/code/lockfree/
• http//appcore.home.comcast.net/

47
Extra Slides
48
UDP Echo Server the shell

• using System
• using System.Text
• using System.Net
• using System.Net.Sockets
• namespace EchoServer
• class Program
• static void Main(string args)
• try
• . . .
• catch(Exception x)
• Console.WriteLine(x.ToString())

49
UDP Echo Server the code

• try
• // create a udp server on port 8666
• UdpClient server new UdpClient(8666)
• // create an end point that maps to any ip and
  any port
• IPEndPoint endPoint new IPEndPoint(IPAddress.A
  ny, 0)
• // receive data from our end point
• byte data server.Receive(ref endPoint)
• // convert from bytes to string
• string msg Encoding.ASCII.GetString(data)
• // display the message and who it is from
• Console.WriteLine(
• "Received message from 0 gt 1",
• endPoint.ToString(), msg)

50
UDP Echo Client in Java

• public static void main(String args)
• try
• // set up our locals
• InetAddress host InetAddress.getByNam
  e("127.0.0.1")
• int port 8666
• String msg "Hello World from
  Java"
• // get the bytes for string
• byte data msg.getBytes()
• // create a udp client
• DatagramSocket client new DatagramSocket()
• // create our data packet
• DatagramPacket packet new
  DatagramPacket(data, data.length, host, port)
• // send the data
• client.send(packet)

51
UDP Echo Client in C

• static void Main(string args)
• // set up locals
• string hostname "127.0.0.1"
• int port 8666
• string msg "Hello World from C"
• try
• // get ascii bytes for our message
• byte data Encoding.ASCII.GetBytes(ms
  g)
• // create a udp client
• UdpClient client new UdpClient()
• // send the bytes to a udp listener
• client.Send(data, data.Length, hostname,
  port)
• catch(Exception x)
• Console.WriteLine(x.ToString())