A

1

• A
• Smart Port Card
• Tutorial
• ---
• Software
• John DeHart
• Washington University
• jdd_at_arl.wustl.edu
• http//www.arl.wustl.edu/jdd

2
Software Overview

• SPC Specific Kernel Modifications
• Building a Kernel
• config
• compile
• Loading Kernel onto SPC to execute it
• download via APIC
• Initiating Kernel execution
• boot loader
• Configuring Devices
• APIC

3
SPC Kernel Modifications

• Kernel Configuration File
• Memory Disk
• Serial Console
• Most device drivers unneeded (e.g no Ethernet)
• Source Code Changes
• PCI Device Configuration ./dev/pci/if_apic_pci.c
• APIC Specific configuration handled by driver
• Change some APIC defaults ./dev/ic/apic.c
• made switch port the default
• No HLT instruction locore.s
• Kernel Memory Page Allocation
• ./arch/i386/include/pmap.new.h
• RTC ./arch/i386/isa/clock.c, ./ufs/ffs/ffs_vfsops
  .c
• We still had to remove some RTC code...

4
Steps to an Executing Kernel on an SPC

• Configure Kernel
• Start with a typical NetBSD kernel config file
• Add SPC specific option
• options SPC
• add memory disk (filesystem will reside in
  kernel!)
• options MEMORY_DISK
• options MINIROOTSIZE
• serial console port
• options CONSDEVNAME
• options CONS_OVERRIDE
• remove extra devices (Ethernet, mouse, )

5
Steps to an Executing Kernel on an SPC

• Filesystem
• we use a vnode disk device vnd(4)
• disk device in a regular file
• pseudo-device vnd
• must be included in your kernel where you are
  building the FS
• Does NOT need to be included in your kernel for
  the SPC
• build filesystem
• dd(1) create the file
• vnconfig(8) - configure vnode pseudo disk as a
  regular file
• disklabel(8) - label it
• newfs(8) - construct a new file system
• mount(8)/umount(8) mount/unmount filesystem
• mknod(8) - make device files
• populate regular files from an existing system

6
Steps to an Executing Kernel on an SPC

• Compile Kernel
• Make a symbol only version of kernel for /netbsd
• ps, netstat, use symbols from /netbsd
• real kernel with file system CANNOT reside in
  itself!
• Add File System to Kernel
• mdsetimage(8)
• Create an image version of Kernel
• sets up bss section and symbol table
• Kernel bss file is ready to download!

7
Steps to an Executing Kernel on an SPC

• Reset SPC
• Download Boot Loader
• via APIC control cells
• loaded at physical memory location 0x0
• Flip bit at location 0xFFE00 to start boot loader
• configure NorthBridge (before kernel download!!)
• cache, memory size, memory type, timing, etc
• wait for kernel download to complete
• looks for location 0x3FC to change from 0 to 1
• set up parameters on stack and jump to kernel
• kernel located at 0x100000
• Download kernel bss file
• Flip bit at location 0x3FC - kernel starts
  booting...

8
The Download Process
0x0
Bootloader
RESET
0x03FC
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 1
Download Program
0x0FFE00
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 1
0x0100000
Kernel
0xFFFFFFF0
Boot ROM
9
Host APIC to SPC Control Connections
Example
0/41
0/42
0/33 (0x21)
0/0x0321
10
APIC Driver at SPC Boot Time

• PCI Configuration Space (sys/dev/pci/if_apic_pci.c
  )
• Base memory address
• APIC Physical memory in SPC starts at 0xF0000000
• Memory mapping options
• Memory Enable
• Master Enable
• Write Invalidate Enable
• IRQ
• Driver normally would read this from PCI config
  space
• APIC statically assigned IRQ 5 by System FPGA
• For consistency, driver writes 5 to Interrupt
  Line Register

11
PCI Configuration Space

• Configuration Space Header

12
Defining a Route on APIC Network Interface

• Configure the APIC
• ifconfig apic0 inet 192.168.10.1 netmask
  0xffffff00
• network interface apic0
• our IP address 192.168.10.1
• Configure a VCI
• atm_ifconfig apic0 0xc5 open -aal5 -llc
  -besteffort -maxqueuebufs 100 -inport 0
  -outports 1
• network interface apic0
• vci 0xc5
• besteffort (vs. paced or lowdelay)
• Tx buffer queue limit for this VCI maxqueuebufs
  100
• Configure the route
• route add -iface 192.168.10.5 -link
  apic00.0.0.c5
• destination 192.168.10.5
• network interface apic0
• vci 0xc5

13
atm_ifconfig

• atm_ifconfig
• Usage atm_ifconfig ltinterfacegt ltvpivcigt
  openclosemodifygetall options...
• Options are
• -aal0 -aal5
• -llc -nollc ltprotocolgt use LLC/SNAP?
• protocol is inet ratm value
  of ethernet type
• -loopback
• -inport ltinput_port_numbergt
• -outports ltbitvector_of_output_portsgt
• bitvector 1 gt port 0, 2 gt port
  1, 3 gt both
• -lowdelay -paced ltrategt -besteffort
• -maxqueuebufs ltmax_descriptors_in_tx_queue
  gt
• ltrategt is in Kbits/sec

14
APIC Configuration
ControlProcessor
Switch Fabric
Sys.FPGA
32-64 MB
APIC
NorthBridge
. . .
SPC
TI
15
Defining APIC Ports for Routing
Switch
INPORT OUTPORTs 0 1
0 2
1 1
1 2
0 3
1 3

• atm_ifconfig apic0 0xca open -aal5 -llc
  -besteffort -inport INPORT -outports OUTPORTs

16
Caveats

• You cannot change the IRQ assignments
• They are fixed in the System FPGA
• And there should be no reason to change them
• HLT instruction in locore.s commented out
• PCI bus may be 25MHz or 33 MHz
• Adjustment of NetBSD clock may be needed
• Memory/Filesystem tradeoffs
• we have not pushed the limits yet
• Understand APIC port numbering
• SPC needs to be reset before EACH download