GLAST Proposal Review

1
The Cluster Interconnect Module (CIM)
Networking RCEs RCE Training Workshop Matt
Weaver, weaver_at_slac.stanford.edu 15 June, 2009
2
Outline

• ATCA Networks
• Zone 2 Base Fabric
• Redundancy traded for multiplicity/bandwidth
• CIM Hardware Design
• Switch ASIC Properties
• Switch mapping to Zone 2 and Zone 3
• Rear Transition Module (RTM)
• Switch Host Processor
• CIM Software
• Development environment
• Protocols
• CIM Usage
• ATCA Hub slot mapping to RCEs
• Front Panel Features

2
The Cluster Interconnect Module (CIM)
3
ATCA Networks
4
(No Transcript)
5
(No Transcript)
6
CIM Hardware Design
7
Switch ASIC Properties

• The current generation Cluster Interconnect
  Fabric is based upon the Fulcrum Micro FM2224
  ASIC
• 24-port Ethernet Layer 2 switch
• Each port 1Gb / 2.5Gb / 10Gb capable
• Fully provisioned, non-blocking, shared memory
• Low-latency cut-through operation (200ns)
• Pause frame flow control
• 1.5W per channel
• Hardware MAC address lookup, static dynamic
• Supports trapping of link protocol messages

Compatible with RCE MGT serial 1 (2.5)Gb
7
The Cluster Interconnect Module (CIM)
8
(No Transcript)
9
Rear Transition Module (RTM)

• Bulk cabling to external network is in the rear
• Technology choice is largely independent of
  front-board design
• CX4 electrical connections
• Inexpensive, Passive
• short range (15m)
• XFP optical transceivers
• Pluggable transceivers
• Short range or long range (250-1000)
• 1W / channel
• Total external bandwidth limited by panel real
  estate
• 8 XFPs or 8 CX4
• High density optical technology becoming
  available (20-40)

9
The Cluster Interconnect Module (CIM)
10
Switch Host Processor

• One host processor for managing both network
  switches
• Implemented in Xilinx Virtex-4 Embedded PPC405 (a
  la RCE)
• 128MB DRAM
• 32MB flash memory file system
• FM2224 management bus interface
• Logical CPU port with DMA Engine on each network
• XFP and BaseT PHY management bus masters
• IPM Controller for ATCA electronic-keying

10
The Cluster Interconnect Module (CIM)
11
CIM Software
12
Development Environment

• GNU C cross-platform programming
• RTEMS 4.9
• Same as RCE
• Similar hardware
• Inherits as much core software and tools as
  possible
• Most likely developments will be to implement new
  management protocols or enhance monitoring

12
The Cluster Interconnect Module (CIM)
13
Management Protocols

• IGMP snooping (multicast control) implemented
• Limit forwarding of multicast data to only
  interested ports
• Lessens burden on resource limited NICs
• IPMI electronic-keying implemented
• Interfaces with ATCA Shelf Manager to
  achievesafe integration with commercial board
  Fabric channels
• Large network management protocols not yet
  implemented
• Desirable for large sets of switches RSTP, LACP
  
• Not difficult, just hasnt been a development
  priority

13
The Cluster Interconnect Module (CIM)
14
User Configuration

• DHCP Vendor Encapsulated Options
• IP address assignment of both network host ports
• Switch channel configurations (in absence of
  Shelf Mgmt)
• VLAN port assignments
• Static MAC address assignments
• Terminal -gt Flash memory
• Manual changes applied via the shell
• Accumulated configuration is written to flash
  memory

14
The Cluster Interconnect Module (CIM)
15
CIM Usage
16
(No Transcript)
17
Front Panel Features Rotary Switch and Reset
Button

• 16-position rotary switch
• Configuration selection
• Read following reset
• Hard reset button

17
The Cluster Interconnect Module (CIM)
18
Front Panel Features Auxiliary Ports
FabricNetwork
BaseNetwork
RJ45
XFP

• XFP and RJ45 external connectivity to each
  network
• Convenience for test-stand setups

18
The Cluster Interconnect Module (CIM)
19
Rear Panel Ports
FabricP3-7
BaseP3-5
Example XFP RTM

• High throughput external connectivity
• 5 x10Gb (Fabric), 3 x10Gb (Base)

19
The Cluster Interconnect Module (CIM)
20
Front Panel Features Multi Function Display

• CIM following reset
• DHCP attempting DHCP
• B - Base channels status
• F - Fabric channels status

Zone 3
H-A
H-C
H-B
P3
P5
P6
P7
P4
Front Panel
P1
P2
link up
enabled
link up
enabled
Fabric or Base
Slot 1/2
Slot 14
Zone 2
Slot 3
Slot 4
Slot 5
20
The Cluster Interconnect Module (CIM)
21
Summary

• CIM is designed to make full use of ATCA provided
  bandwidth
• Dual 10Gb and dual 1Gb channels to each slot
• Several external 10Gb channels (6/4)
• Fully provisioned packet switching fabric
• CIM Software
• Same development environment as RCE
• Customizable for large scale aggregation

21
The Cluster Interconnect Module (CIM)