io multiplexing

1
i/o multiplexing

• On adding a poll() method to our character-mode
  device-driver for an 82573L network controller

2
An application idea

• We want to create an application program that
  would allow two users to conduct an on-line
  chat-session while working from two separate
  nodes on our anchor cluster
• Whatever either user decides to type will be
  displayed on the both users screens
• To keep their two input-streams visually
  separated we want to use two windows

3
netchat.cpp
Whatever a user types will be displayed in the
lower window on that users screen -- and in
the upper window on the other users screen
Hello, Susan
Hi, Paul
Hi, Paul _
Hello, Susan _
anchor01
anchor02
4
Multiplexed input-streams

• In order to implement the foregoing idea, its
  necessary for the netchat program to accept
  input from TWO hardware devices
• the keyboard (i.e., standard input device)
• the ethernet controller (i.e., /dev/nic)
• Such a situation is a fairly common one in
  UNIX/Linux application programming, and is
  referred to as i/o multiplexing

5
Device-drivers role

• Special software support is needed within each of
  the device-drivers in order for any
  application-program to do i/o multiplexing in a
  way that avoids wasting of CPU time
• (For an excellent discussion of the various
  approaches that can be taken to deal with this
  basic programming issue, see Richard Stevens
  Advanced Programming in the UNIX Environment,
  Chapter 12)

6
The basic problem

• Normally when an application reads from a
  device-file, that process will sleep until some
  data is available from that device
• So if data becomes available on another device,
  it will not get processed because the application
  is blocked from being given any CPU time by the
  OS scheduler
• This would spoil our netchat application

7
read() causes blocking
netchat application
terminal
Ethernet controller
read
write
read
write
Whichever device the application attempts to
read from, it will get blocked until that
device has some data to deliver
8
Do multiprocessing?

• One idea for getting around this blocking
  problem would be to just use the fork()
  system-call to create a separate process for
  reading from the different device-files
• Each process can sleep, and whichever process
  receives any new data will be awakened and
  scheduled for execution
• No changes needed to device-driver code

9
Different processes do read()
netchat parent- process
write
read
terminal
Ethernet controller
netchat child-process
read
write
Using multiple processes can overcome the
blocking-read problem, but complicates the
code for program termination
10
Non-blocking read

• It is possible for the application to request
  non-blocking read-operations i.e., any
  read() calls will immediately return with 0 as
  return-value in case no data is available
• The standard-input device-driver already has
  support for this non-blocking option, and it can
  be easily added to the read() function in
  network controllers driver

11
Code-modification
ssize_t my_read( struct file file, char buf,
size_t len, loff_t pos ) static int rxhead
0 // in case no new data has been received,
then either // return immediately if
non-blocking mode is in effect // or else sleep
until some new data arrives (or until // the
user hits ltCONTROLgt-C to cancel execution) if
( rxhead ioread32( io E1000_RDH ) if (
file-gtf_flags O_NONBLOCK ) return 0 if (
wait_event_interruptible( wq_rx, rxring !
ioread32( io E1000_RDH ) ) return
EINTR
12
Uses busy-waiting loop
netchat application
terminal
read
write
Ethernet controller
write
read
Using the nonblocking-read option overcomes
the problem of a sleeping task, but it
wastefully consumes the CPU time
13
The elegant solution

• The select() system-call provides a very
  general scheme for doing i/o-multiplexing in a
  manner that avoids wasting CPU time or making the
  program-code complicated
• But it does require adding an extra driver
  method the so-called poll() function

14
The select() arguments

• Using select() requires an application to setup
  an fd_set object, which defines the set of
  file-descriptors whose activity needs to be
  monitored by the Linux kernel (in our netchat
  application this would be just the two
  device-files the console keyboard and the
  gigabit ethernet network controller)
• This fd_set object becomes an argument

15
Using select() in netchat
int kbd STDIN_FILENO // keyboard ID
int aux open( /dev/nic, O_RDWR ) //
device-file ID fd_set permset // create an
fd_set object FD_ZERO( permset ) //
initialize it to empty FD_SET( kbd, permset
) // add keyboard to set FD_SET( aux, permset
) // and add the nic to set while
(1) fd_set readset permset if ( select(
1aux, readset, NULL, NULL, NULL ) lt 0 )
break if ( FD_ISSET( kbd, readset ) ) /
process keyboard input / if ( FD_ISSET(
aux, readset ) ) / process network input /

16
How it works

• The readset argument to the select()
  system-call lets the kernel know which
  device-drivers should have their poll() method
  invoked
• Then each device-drivers poll() method will
  perform a test to determine if any new data is
  ready to be read from that device
• So the application calls read() only when a
  device is ready with data immediately

17
struct file_operations

• We need to include the function-pointer to our
  implementation for the poll() method

struct file_operations my_fops
owner THIS_MODULE, read my_read,
write my_write, ioctl my_ioctl, pol
l my_poll,
18
Our drivers poll() method

• Linux provides helper-functions to do most of the
  supporting work for use of select()

include ltlinux/poll.hgt // for the poll_wait()
helper-function unsigned int my_poll( struct
file file, struct poll_table_struct wait )
unsigned int mask 0 poll_wait( file,
wq_recv, wait ) if ( (ioread32( io
E1000_RDH ) ! rxhead ) mask POLLIN
POLLRDNORM return mask
19
The ncurses library

• Our netchat application uses the Linux
  implementation for the UNIX curses API
• This function-library includes routines that not
  only can manage the consoles cursor, but also
  implements raw keyboard input (for
  instantaneous keystroke processing) and allows
  drawing of window-borders

20
Compiling netchat

• The g compiler can compile-and-link in a
  single command

g netchat.cpp -l ncurses -o netchat
The C source-file
Name of the library
Lowercase letter L (for library to link with)
Name for the compilers output-file
21
Try it out

• We can try running the netchat program on any
  pair of our anchor-cluster stations
• Of course, we will first need to install our
  nicpoll.ko device-driver module on each of
  those anchor machines
• Then a user at either machine can type in any
  messages, and every character typed will be
  visible immediately on both screens

22
But

• There is a slight problem with using the present
  version of our nicpoll.c module
• All of the network packets are broadcast to
  every station on the anchor cluster!
• So any third-party who is simultaneously trying
  to use our network will be receiving all the
  packets that are being broadcast

23
In-class exercise

• Based on what you have learned from our past
  lessons, and from your reading of the Intel Open
  Source Software Developers Manual for its PCIe
  GbE Controllers, can you propose any ideas for
  using the NICs hardware capabilities so as to
  allow a pair of anchor-stations to use netchat
  without any other anchor-stations being involved?