Accessing network hardware

1
Accessing network hardware

• The Network Interface Controllers are part of a
  larger scheme used in modern PCs for device
  control

2
Memory-mapped I/O

• Non-Intel processor architectures typically will
  dedicate a set of memory-addresses to performing
  communication with hardware devices which are
  installed in the system
• Such architectures allow software to read
  device-status or write device-commands using
  ordinary CPU instruction-opcodes, such as mov,
  test, add, xor, et cetra

3
I/O ports

• But Intels original x86 architecture used a
  different approach in which a separate set of
  addresses and special instructions are employed
  for accessing system devices
• The device addresses are called I/O ports, and
  the opcodes are named in and out
• The in instruction reads a devices status
• The out instruction writes device command

4
Range of I/O ports

• For devices integrated into a PC systems
  motherboard, the port-numbers used were 8-bit
  values and could be specified within an I/O
  instruction, as immediate data
• Example in 0x64, al read port 0x64
• For optional devices plugged into slots on the
  motherboard, a port-number could be a 16-bit
  value, located in the DX register
• Example in dx, al indirect address

5
Two separate address-spaces
accessible by using a wide variety of
general-purpose arithmetical and logical
instructions
memory address-space (4GB)
accessed only by using the special in and
out instructions
I/O address-space (64KB)
6
Early PCs

• Peripheral devices in the early PCs used fixed
  i/o-ports and fixed memory-addresses, e.g.
• Video memory address-range 0xA0000-0xBFFFF
• Programmable timer i/o-ports 0x40-0x43
• Keyboard and mouse i/o-ports 0x60-0x64
• Real-Time Clocks i/o-ports 0x70-0x71
• Hard Disk controllers i/o-ports 0x01F0-01F7
• Graphics controllers i/o-ports 0x03C0-0x3CF
• Serial-port controllers i/o-ports 0x03F8-0x03FF
• Parallel-port controllers i/o-ports
  0x0378-0x037A

7
The PCs evolution

• It became clear in the 1990s that there would be
  contention among equipment vendors for fixed
  resource-addresses, which of course were in
  limited supply
• Among the goals that motivated the PCI
  Specification was the creation of a more flexible
  scheme for allocating addresses that future
  peripheral devices could use

8
PCI Configuration Space
A non-volatile parameter-storage area for each
PCI device-function
PCI Configuration Space Header (16 doublewords
fixed format)
PCI Configuration Space Body (48 doublewords
variable format)
64 doublewords
9
PCI Configuration Header
16 doublewords
31
0
31
0
Dwords
Status Register
Command Register
Device ID
Vendor ID
1 - 0
BIST
Cache Line Size
Class Code Class/SubClass/ProgIF
Revision ID
Latency Timer
Header Type
3 - 2
Base Address 0
Base Address 1
5 - 4
Base Address 2
Base Address 3
7 - 6
Base Address 4
Base Address 5
9 - 8
Subsystem Device ID
Subsystem Vendor ID
CardBus CIS Pointer
11 - 10
reserved
capabilities pointer
Expansion ROM Base Address
13 - 12
Minimum Grant
Interrupt Pin
reserved
Interrupt Line
Maximum Latency
15 - 14
10
Three x86 address-spaces
accessed using a large variety of processor
instructions (mov, add, or, shr, push, etc.) and
virtual-to-physical address-translation
memory space (4GB)
accessed only by using the processors
special in and out instructions (without
any translation of port-addresses)
PCI configuration space (16MB)
i/o space (64KB)
i/o-ports 0x0CF8-0x0CFF dedicated to accessing
PCI Configuration Space
11
Interface to PCI Configuration Space
PCI Configuration Space Address Port (32-bits)
31 23
16 15 11 10 8 7
2 0
reserved
E N
bus (8-bits)
device (5-bits)
doubleword (6-bits)
function (3-bits)
00
CONFADD ( 0x0CF8)
Enable Configuration Space Mapping (1yes, 0no)
PCI Configuration Space Data Port (32-bits)
31

0
CONFDAT ( 0x0CFC)
12
Reading PCI Configuration Data

• Step one Output the desired longwords address
  (bus, device, function, and dword) with bit 31
  set to 1 (to enable access) to the
  Configuration-Space Address-Port
• Step two Read the designated data from the
  Configuration-Space Data-Port

read the PCI Header-Type field (byte 2 of dword
3) for bus0, device0, function0 movl 0x800000
0C, eax setup address in EAX movw 0x0CF8,
dx setup port-number in DX outl eax,
dx output address to port mov 0x0CFC,
dx setup port-number in DX inl dx, eax
input configuration longword shr 16, eax
shift word 2 into AL register movb al,
header_type store Header Type in variable
13
Demo Program

• We created a short Linux utility that searches
  for and reports all of your systems PCI devices
• Its named pciprobe.cpp on our class website
• It uses some C macros that expand to Intel
  input/output instructions -- which normally are
  privileged instructions that a Linux
  application-program is not allowed to execute
  (segfault!)
• But using sudo you can escalate the
  privilege-level at which this utility-program
  will be run
• sudo ./pciprobe

14
Example network interface

• We can identify the network interface controllers
  in our classrooms PCs by class-code 0x02
• Then the subclass-code 0x00 is for Ethernet
• We can identify the NIC from its VENDOR and
  DEVICE identification-numbers
• VENDOR_ID 0x8086 // for Intel Corporation
• DEVICE_ID 0x109A // for 82573L controller
• Our second demo-program (i.e, 82573pci.cpp)
  shows this nics full PCI Configuration Space

15
Typical NIC

main memory
packet
nic
TX FIFO
transceiver
buffer
LAN cable
B U S
RX FIFO
CPU
16
Intels Pro1000 nics

• These network controllers implement a large
  number of 32-bit device registers
• Two ways are implemented for accessing these
  numerous registers
• Using a small range of I/O port-addresses
• Using a large range of memory-mapped
  register-addresses

17
PCI Base-Address registers

• The key to accessing our NICs various registers
  is provided by values we find in certain PCI
  Configuration Space locations

32-bit Base-Address Register layout for I/O space
port-address
1
R S V
unused
32-bit Base-Address Register layout for Memory
space
memory-address
0
P R E F
R S V
0032-bit decode, 01reserved, 1064-bit decode,
11reserved
18
Output from 83573pci.cpp
executed on anchor01
BAR0 (at 0x10) 0xE8200000 (Physical
memory-address for NICs mapped
register-bank) BAR1 (at 0x14) 0x00000000 BAR2
(at 0x18) 0x00005001 (I/O port-address for NICs
multiplexing register-bank) BAR3 (at 0x1C)
0x00000000 BAR4 (at 0x20) 0x00000000 BAR5 (at
0x24) 0x00000000
19
mapped advantage

• For high-performance (and thread-safe) access
  to the NICs bank of registers, the memory-mapped
  option offers very clear advantages over
  multiplexed I/O access
• Access is faster to memory than to I/O ports
• Access to memory is atomic, whereas the
  multiplexed access via I/O ports is a 2-step
  operation, giving rise to race conditions

20
I/O advantage

• But during the systems startup process (before
  the Operating System is loaded), theres no
  alternative but to use the I/O access-method,
  because the memory-mapped registers are at high
  addresses which the processor cannot yet reach
• For example, in cases where an OS needs to be
  loaded via a BOOTP network-server

21
Clarification

• Dont confuse PCI Configuration Space registers
  with our NICs device registers!

CPUs memory address-space
Systems PCI Configuration Space
CPUs I/O-port address-space
NICs PCI Configuration Space registers
BAR2
BAR0
Network Interface Controller
NIC device-registers
window
22
In-class exercise

• Copy our 82573pci.cpp source-module to your own
  directory, then rename it as audio.cpp
• Your assignment is to modify it so that it will
  display the PCI Configuration Space registers in
  our classroom workstations sound-card (a
  multimedia class device)