A

1

• A
• Smart Port Card
• Tutorial
• ---
• Software
• John DeHart
• Washington University
• jdd_at_arl.wustl.edu
• http//www.arl.wustl.edu/jdd

2
Software Overview

• No MSR details included in this talk
• That will come in the MSR Tutorial
• SPC Specific Kernel Modifications
• Building a Kernel
• config
• compile
• Loading Kernel onto SPC to execute it
• download via APIC
• Initiating Kernel execution
• boot loader
• Configuring Devices
• APIC

3
SPC Kernel Modifications

• Kernel Configuration File
• Memory Disk
• Serial Console
• Most device drivers unneeded (e.g no Ethernet)
• Source Code Changes
• PCI Device Configuration ./dev/pci/if_apic_pci.c
• APIC Specific configuration handled by driver
• Change some APIC defaults ./dev/ic/apic.c
• made switch port the default
• No HLT instruction locore.s
• Kernel Memory Page Allocation
• ./arch/i386/include/pmap.new.h
• RTC ./arch/i386/isa/clock.c, ./ufs/ffs/ffs_vfsops
  .c
• We still had to remove some RTC code...

4
Steps to an Executing Kernel on an SPC

• Configure Kernel
• Start with a typical NetBSD kernel config file
• Add SPC specific option
• options SPC
• add memory disk (filesystem will reside in
  kernel!)
• options MEMORY_DISK
• options MINIROOTSIZE
• serial console port
• options CONSDEVNAME
• options CONS_OVERRIDE
• remove extra devices (Ethernet, mouse, )

5
Steps to an Executing Kernel on an SPC

• Filesystem
• we use a vnode disk device vnd(4)
• disk device in a regular file
• pseudo-device vnd
• must be included in your kernel where you are
  building the FS
• Does NOT need to be included in your kernel for
  the SPC
• build filesystem
• dd(1) create the file
• vnconfig(8) - configure vnode pseudo disk as a
  regular file
• disklabel(8) - label it
• newfs(8) - construct a new file system
• mount(8)/umount(8) mount/unmount filesystem
• mknod(8) - make device files
• populate regular files from an existing system

6
Steps to an Executing Kernel on an SPC

• Compile Kernel
• Make a symbol only version of kernel for /netbsd
• ps, netstat, use symbols from /netbsd
• real kernel with file system CANNOT reside in
  itself!
• Download kernel and File System separately
• see following slides on AAL5download
• Or, Download as one file
• Add File System to Kernel
• mdsetimage(8)
• Create an image version of Kernel
• sets up bss section and symbol table
• Kernel bss file is ready to download!

7
Steps to an Executing Kernel on an SPC

• Reset SPC
• Download Boot Loader
• via APIC control cells
• loaded at physical memory location 0x0
• Flip bit at location 0xFFE00 to start boot loader
• configure NorthBridge (before kernel download!!)
• cache, memory size, memory type, timing, etc
• wait for kernel download to complete
• looks for location 0x3FC to change from 0 to 1
• set up parameters on stack and jump to kernel
• kernel located at 0x100000
• Download kernel bss file
• Flip bit at location 0x3FC - kernel starts
  booting...

This is the OLD WAY Please stop using
download_APIC It is slow and painful
8
AAL5_Download

• Replaces old download_APIC utility
• Can shotgun download to lt 8 SPCs at once
• Puts exactly same kernel on all 8
• This works fine for MSR where we want same kernel
• May not be good for all uses
• Shotgun number controlled from command line
• Can download kernel and file system separately
• no need to load file system into kernel before
  download
• still need to load symbol version of kernel into
  file system before downloading
• Takes about 1 minute for the whole download and
  boot process.

9
AAL5_Download For SPC

• Implemented and runs in Stages
• Stage1 uses APIC Control cells to
• reset the SPC(s)
• download Stage2 to run on the SPC(s)
• Stage2 (boot.out) runs on the SPC
• runs SPC boot code to configure CPU and
  Northbridge
• configures APIC on SPC to use AAL5 descriptors
• sends pings back to CP, waiting for Stage3
• receives AAL5 frames from Stage3 and sends
  ACK/NACK
• Stage3 runs on PC
• syncs with Stage2
• downloads the kernel or FS via 36KB AAL5 frames
• retransmits if any errors

10
AAL5_Download For SPC-II

• We are still working out the final details
• We are starting to test the first batch of
  SPC-IIs
• Implemented and runs in Stages
• Stage2 (boot.out) runs on the SPC-II
• Already loaded on the IDE Disk on Module.
• no need to download it
• runs SPC boot code to configure CPU and
  Northbridge
• configures APIC on SPC to use AAL5 descriptors
• sends pings back to CP, waiting for Stage3
• receives AAL5 frames from Stage3 and sends
  ACK/NACK
• Stage3 runs on PC
• syncs with Stage2
• downloads the kernel or FS via 36KB AAL5 frames
• retransmits if any errors

11
Simple View
Stage1
PC
SPC-I
VCI0x321
Stage1
APIC
VCI21
VCI21
VCI25
Stage3
VCI23
Stage2
VCI24
12
Sequential View
SPC Memory
PC
SPC
0x0
Boot and Stage2
Stage1
APIC
RESET
0x0FFE00
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 1
Stage3
Stage2
Kernel and Filesystem
0x0100000
0xFFFFFFF0
Boot ROM
13
Source and Executables

• source location wu_arl/utilities/SPC/AAL5Download
  
• binary location
• wu_arl/utilities/bin/OSTYPE/Stage1
• wu_arl/utilities/bin/OSTYPE/Stage3
• wu_arl/utilities/bin/OSTYPE/AAL5_download.sh
• wu_arl/utilities/bin/OSTYPE/boot.out
• wu_arl/utilities/bin/OSTYPE/FindNM

14
Usage

• Individual command line commands
• ./Stage1 shotgun 8 svci 21 rvci 21 dvci 23
  uvci 24 cvci 25
• ./Stage3 shotgun 8 native netbsd.MSR both 24
• ./Stage3 shotgun 8 bss SPC24MB.fs address
  0x0196434 both 24 -go
• AAL5download.sh script
• ./AAL5download 8 netbsd.MSR SCP24MB.fs 21 21 23
  24 25
• Use from MSR Configuration Module
• system(msrDownload.sh)

15
Special Usage Note on Linux

• AAL5download.sh script
• needs to be able to determine the start location
  for the File System
• NetBSD uses nm(1)
• Linux nm(1) wont work on a NetBSD binary
• We have written a version of nm specifically for
  AAL5_download.sh FindNM
• FindNM is used automatically when
  AAL5_download.sh is run on Linux

16
AAL5_download.sh Usage

• gt ./AAL5_download.sh
• Usage ./AAL5_download.sh ltnetbsd_filegt
  ltfilesystem_filegt ltstart_vcigt ltsecond_vcigt
  ltthird_vcigtltfourth_vcigt ltfifth_vcigt

17
Stage1 Usage

• gt ./Stage1
• ./Stage1 vpi/vci data load info
• ./Stage1 -svpi (VPI for local sending of
  AAL5 control cells (0))
• ./Stage1 -svci (VCI for local sending of
  AAL5 control cells)
• ./Stage1 -rvpi (VPI for local reading of
  AAL5 control cells (0))
• ./Stage1 -rvci (VCI for local reading of
  AAL5 control cells)
• ./Stage1 -cvpi (VPI for remote CC writes of
  AAL5 control cells (0))
• ./Stage1 -cvci (VCI for remote CC writes of
  AAL5 control cells)
• ./Stage1 -dvpi (VPI for local write of AAL5
  data frames (0))
• ./Stage1 -dvci (VCI for local write of AAL5
  data frames)
• ./Stage1 -uvpi (VPI for local read of AAL5
  data frames (0))
• ./Stage1 -uvci (VCI for local read of AAL5
  data frames)
• ./Stage1 -resp (Response port of AAL5
  control cells)
• ./Stage1 -start (Initial VCI sequence (41,
  42, 43, 44))
• ./Stage1 -watch 0x (Monitor location for
  changes)
• ./Stage1 -apic (Linux Apic Id (0 lt X lt
  3))
• ./Stage1 -abit (Alternating bit setting)
• ./Stage1 -file ltpathgt (Name of boot loader
  (boot.out))
• ./Stage1 -swap (use htonl() on output)

18
Stage3 Usage

• gt ./Stage3 -help
• -switches (-help) args
• -bss ltfilegt Load .bss format file
• -native ltfilegt Load native uncompressed file
• -strip Remove Symbols from kernel
• -debug print lots of stuff
• -verbose Dump Elf headers
• -elf Elf format execution
• -rvpi ltVPIgt Load VPI number
• -rvci ltVCIgt Load VCI number
• -svpi ltVPIgt Load VCI number
• -svci ltVCIgt Load VCI number
• -both ltVCIgt Set both VCI numbers
• -address ltPhysAddrgt Load file at ltaddrgt
• -start ltPhysAddrgt Start file at ltaddrgt
• -go Start kernel
• -shotgun ltcountgt Multicast loading 8X

19
Host APIC to SPC Control Connections
Example
0/41
0/42
0/33 (0x21)
0/0x0321
20
APIC Driver at SPC Boot Time

• PCI Configuration Space (sys/dev/pci/if_apic_pci.c
  )
• Base memory address
• APIC Physical memory in SPC starts at 0xF0000000
• Memory mapping options
• Memory Enable
• Master Enable
• Write Invalidate Enable
• IRQ
• Driver normally would read this from PCI config
  space
• APIC statically assigned IRQ 5 by System FPGA
• For consistency, driver writes 5 to Interrupt
  Line Register

21
PCI Configuration Space

• Configuration Space Header

22
Defining a Route on APIC Network Interface

• NOTE With the MSR you do not use ifconfig and
  atm_ifconfig
• the MSR uses a different APIC driver then plain
  NetBSD for the SPC
• Configure the APIC
• ifconfig apic0 inet 192.168.10.1 netmask
  0xffffff00
• network interface apic0
• our IP address 192.168.10.1
• Configure a VCI
• atm_ifconfig apic0 0xc5 open -aal5 -llc
  -besteffort -maxqueuebufs 100 -inport FIBER
  -outports FIBER
• network interface apic0
• vci 0xc5
• besteffort (vs. paced or lowdelay)
• Tx buffer queue limit for this VCI maxqueuebufs
  100
• Configure the route
• route add -iface 192.168.10.5 -link
  apic00.0.0.c5
• destination 192.168.10.5
• network interface apic0
• vci 0xc5

23
atm_ifconfig

• atm_ifconfig
• Usage atm_ifconfig ltinterfacegt ltvpivcigt
  openclosemodifygetall options...
• Options are
• -aal0 -aal5
• -llc -nollc ltprotocolgt use LLC/SNAP?
• protocol is inet ratm value
  of ethernet type
• -loopback
• -inport ltinput_port_numbergt
• -outports ltbitvector_of_output_portsgt
• bitvector 1 gt port 0, 2 gt port
  1, 3 gt both
• -lowdelay -paced ltrategt -besteffort
• -maxqueuebufs ltmax_descriptors_in_tx_queue
  gt
• ltrategt is in Kbits/sec

Changes HERE -inport and -outport values have chan
ged.
24
APIC Configuration
ControlProcessor
Switch Fabric
Sys.FPGA
32-64 MB
APIC
NorthBridge
. . .
SPC
TI
25
Defining APIC Ports for Routing
Switch
Sys.FPGA
32-64 MB
INPORT OUTPORTs LINK LINK
APIC
NorthBridge
TI
Link

• atm_ifconfig apic0 0xca open -aal5 -llc
  -besteffort -inport INPORT -outports OUTPORTs

26
Defining APIC Ports for Routing
Switch
Sys.FPGA
32-64 MB
INPORT OUTPORTs LINK SWITCH
APIC
NorthBridge
TI
Link
atm_ifconfig apic0 0xca open -aal5 -llc
-besteffort -inport INPORT -outports OUTPORTs
27
Defining APIC Ports for Routing
Switch
INPORT OUTPORTs SWITCH LINK
Sys.FPGA
32-64 MB
APIC
NorthBridge
TI
Link
atm_ifconfig apic0 0xca open -aal5 -llc
-besteffort -inport INPORT -outports OUTPORTs
28
Defining APIC Ports for Routing
Switch
INPORT OUTPORTs SWITCH SWITCH
Sys.FPGA
32-64 MB
APIC
NorthBridge
TI
Link
atm_ifconfig apic0 0xca open -aal5 -llc
-besteffort -inport INPORT -outports OUTPORTs
29
Defining APIC Ports for Routing
Switch
INPORT OUTPORTs LINK BOTH
Sys.FPGA
32-64 MB
APIC
NorthBridge
TI
Link
atm_ifconfig apic0 0xca open -aal5 -llc
-besteffort -inport INPORT -outports OUTPORTs
30
Defining APIC Ports for Routing
Switch
INPORT OUTPORTs SWITCH BOTH
Sys.FPGA
32-64 MB
APIC
NorthBridge
TI
Link
atm_ifconfig apic0 0xca open -aal5 -llc
-besteffort -inport INPORT -outports OUTPORTs
31
Caveats

• You cannot change the SPC-I IRQ assignments
• They are fixed in the System FPGA
• And there should be no reason to change them
• HLT instruction in locore.s commented out
• remnant of when we were having INTEL problems
• we can leave it this way
• gives a slight performance improvement
• SPC needs to be reset before EACH download