Accessing the NIC

1
Accessing the NIC

• A look at the mechanisms that software can use to
  interact with our 82573L network interface

2
Typical NIC hardware

main memory
packet
nic
TX FIFO
transceiver
buffer
LAN cable
B U S
RX FIFO
CPU
3
No way to communicate
SOFTWARE
Linux operating system kernel (no knowledge of
the NIC)
These components lack a way to interact
HARDWARE
Network interface controller (no knowledge of the
OS)
4
Role for a device driver
SOFTWARE
Linux operating system kernel (no knowledge of
the NIC)
device driver module (knows about both the OS
and the NIC)
HARDWARE
Network interface controller (no knowledge of the
OS)
5
Three x86 address-spaces
accessed using a large variety of processor
instructions (mov, add, or, shr, push, etc.) and
virtual-to-physical address-translation
memory space (4GB)
accessed only by using the processors
special in and out instructions (without
any translation of port-addresses)
PCI configuration space (16MB)
i/o space (64KB)
i/o-ports 0x0CF8-0x0CFF dedicated to accessing
PCI Configuration Space
6
Interface to PCI Configuration Space
PCI Configuration Space Address Port (32-bits)
31 23
16 15 11 10 8 7
2 0
reserved
E N
bus (8-bits)
device (5-bits)
doubleword (6-bits)
function (3-bits)
00
CONFADD ( 0x0CF8)
Enable Configuration Space Mapping (1yes, 0no)
PCI Configuration Space Data Port (32-bits)
31

0
CONFDAT ( 0x0CFC)
7
82573L

• Two mechanisms for accessing the NIC
• I/O space (allows booting over the network)
• Memory-mapped I/O (available after booting)
• Both mechanisms require probing for PCI
  Configuration Space information (for I/O
  port-number or memory-mapped address)
• Probing requires knowing the devices IDs (the
  VENDOR_ID and the DEVICE_ID)

8
Our NICs ID numbers

• The VENDOR_ID for Intel Corporation
• 0x8086 (famous chip-number for IBM-PCs)
• The DEVICE_ID for Intels 82573L NIC
• 0x109A (found in Intels documentation p.141)

9
NICs access mechanisms

• The two ways for accessing our network
  controllers control and status registers are
  explained in Chapter 13 of the Intel Open Source
  Programmers Reference
• This Chapter also includes a detailed list of the
  names and functional descriptions for the various
  network interface registers
• All are accessed as 32-bit doubewords

10
PCI Configuration Space
A non-volatile parameter-storage area for each
PCI device-function
PCI Configuration Space Header (16 doublewords
fixed format)
PCI Configuration Space Body (48 doublewords
variable format)
64 doublewords
11
PCI Configuration Header
16 doublewords
31
0
31
0
Dwords
Status Register
Command Register
Device ID
Vendor ID
1 - 0
BIST
Cache Line Size
Class Code Class/SubClass/ProgIF
Revision ID
Latency Timer
Header Type
3 - 2
Base Address 0
Base Address 1
5 - 4
Base Address 2
Base Address 3
7 - 6
Base Address 4
Base Address 5
9 - 8
Subsystem Device ID
Subsystem Vendor ID
CardBus CIS Pointer
11 - 10
reserved
capabilities pointer
Expansion ROM Base Address
13 - 12
Minimum Grant
Interrupt Pin
reserved
Interrupt Line
Maximum Latency
15 - 14
12
Interface to PCI Configuration Space
PCI Configuration Space Address Port (32-bits)
31 23
16 15 11 10 8 7
2 0
reserved
E N
bus (8-bits)
device (5-bits)
doubleword (6-bits)
function (3-bits)
00
CONFADD ( 0x0CF8)
Enable Configuration Space Mapping (1yes, 0no)
PCI Configuration Space Data Port (32-bits)
31

0
CONFDAT ( 0x0CFC)
13
Reading PCI Configuration Data

• Step one Output the desired longwords address
  (bus, device, function, and dword) with bit 31
  set to 1 (to enable access) to the
  Configuration-Space Address-Port
• Step two Read the designated data from the
  Configuration-Space Data-Port

read the PCI Header-Type field (byte 2 of dword
3) for bus0, device0, function0 movl 0x800000
0C, eax setup address in EAX movw 0x0CF8,
dx setup port-number in DX outl eax,
dx output address to port mov 0x0CFC,
dx setup port-number in DX inl dx, eax
input configuration longword shr 16, eax
shift word 2 into AL register movb al,
header_type store Header Type in variable
14
Demo Program

• We created a short Linux utility that searches
  for and reports all of your systems PCI devices
• Its named pciprobe.cpp on our CS686 website
• It uses some C macros that expand to Intel
  input/output instructions -- which normally are
  privileged instructions that a Linux
  application-program is not allowed to execute
  (segfault!)
• Our system administrator (Alex Fedosov) has
  created a utility (named iopl3) that will allow
  your command-shell to acquire I/O privileges

15
Example network interface

• We identify the network interface controller in
  our classroom PCs by class-code 0x02
• The subclass-code 0x00 is for ethernet
• We can identify the NIC from its VENDOR and
  DEVICE identification-numbers
• VENDOR_ID 0x8086 (for Intel Corporation)
• DEVICE_ID 0x109A (for 82573L controller)
• You can use the grep command to search for
  these numbers in this header-file
• lt/usr/src/linux/include/linux/pci_ids.hgt

16
The NICs PCI resources
16 doublewords
31
0
31
0
Dwords
Status Register
Command Register
DeviceID 0x109A
VendorID 0x8086
1 - 0
BIST
Cache Line Size
Class Code Class/SubClass/ProgIF
Revision ID
Latency Timer
Header Type
3 - 2
Base Address 0
Base Address 1
5 - 4
Base Address 2
Base Address 3
7 - 6
Base Address 4
Base Address 5
9 - 8
Subsystem Device ID
Subsystem Vendor ID
CardBus CIS Pointer
11 - 10
reserved
capabilities pointer
Expansion ROM Base Address
13 - 12
Minimum Grant
Interrupt Pin
reserved
Interrupt Line
Maximum Latency
15 - 14
17
Linux PCI helper-functions
include ltlinux/pci.hgt struct pci_dev devp
unsigned int mmio_base unsigned int mmio_size
void io devp pci_get_device( VENDOR_ID,
DEVICE_ID, NULL ) if ( devp NULL ) return
ENODEV mmio_base pci_resource_start( devp,
0 ) mmio_size pci_resource_len( devp, 0
) io ioremap_nocache( mmio_base, iomm_size
) if ( io NULL ) return ENOSPC
18
Mechanisms compared
Each NIC register has its own address in
memory (allows one-step access)
kernel memory-space
NIC i/o-memory
io
user memory-space
Access to all of the NICs registers
is muliplexed through a pair of
I/O-ports (requires multiple instructions)
addr
data
CPUs virtual address-space
CPUs I/O address-space
19
C language hides complexity

• For the multiplexed i/o-space access

// Your module uses just one C statement to
input a NIC register-value int device_status
inl( ioport 0x0008 ) // but the C
compiler translates this statement into SIX
cpu-instructions mov ioport, dx mov 8,
eax out eax, dx add 4, dx in dx,
eax mov eax, device_status
20
Seeing through the C

• For the i/o-memory mapped access

// Your module uses just one C statement to
fetch a NIC register-value int device_status
ioread32( io 0x0008 ) // but the C
compiler translates this statement into three
cpu-instructions mov io, esi mov 8(esi),
eax mov eax, device_status
21
Courses continuing theme is
Using the computer to study the computer
22
TimeStamp Counter

• The x86 processor has a 64-bit register named the
  TimeStamp Counter (TSC)
• It continuously increments each cpu cycle (so
  with a 2-GHz processor, it increments two-billion
  times every second)
• It can be read at any time using a special
  processor instruction (named RDTSC) its value
  appears in the EDXEAX registers

23
Using CLI and STI

• For doing accurate timing measurements, your
  module can temporarily disable your CPUs
  response to interrupt requests

Basic steps for performing an uninterrupted
elapsed time measurement step 1 Turn off
interrupts (using cli) step 2 Read the
TimeStamp Counter step 3 Perform a NIC
register access step 4 Read the TimeStamp
Counter step 5 Turn on interrupts (using
sti) step 6 Subtract start-time from
finish-time
24
In-class exercise

• Look at our timing.c demo module for an
  inline assembly language example using the
  rdtsc, cli and sti instructions
• Add some extra code of your own, and do an
  additional timing measurement, so you can compare
  the execution-times for the NICs two
  register-access mechanisms
• How much faster is memory-mapped I/O?