MSR Command Cell Facility

1
MSR Command Cell Facility

• Fred Kuhns
• Applied Research Laboratory

2
Highlights

• Overview
• High level description - Application Layer
• MSR Command Interface Overview
• Cell format and field definitions
• Example

3
Definitions

• Session Open connection between the CP and a
  specific SPC. Intended to represent open
  connections and command state
• Transaction Represent a complete command. A
  transaction terminates with either an EOF is
  received by the CP or and error occurs.
• EOF End of File is returned to CP with the last
  bit of command data is returned or in response to
  a Cancel message (or an error occurs)

4
Overview - Cmd Interface on CP

• Synchronous Request/Response protocol
• Timeout can be specified as well as the number of
  retries - Per session option
• Essentially provides a reliable service
• Issue if no reply, cmd/reply msg lost in port,
  channel or CP. Retries may be a bad thing.
• Address - MSR Port and Command
• ltMSR_Port, MSR_Commandgt
• Message destination - Callback function within
  the Ports kernel (implements command)

5
Command Interface on CP

• Types of messages
• New Command, Get Next set of reply data Command,
  Cancel Command
• Error Reply, EOF Reply, Continued Reply
• Message Identifiers - Only requires a sequence
  number initialized to 0 for each New Command
• One sending entity on CP,
• One outstanding command for each port,
• Ports send exactly one reply msg per command msg,
• Command must fit within one cell,
• Replies may span multiple cells.

6
Command Interface on Port

• Callback function registered with MSR kernel and
  called under 3 cases
• New Command
• Flags Command Sequence 0 Length valid
  bytes in buffer Buffer application data
• Next Command
• Flags Command Next Sequence previous1
  Length valid bytes in buffer Buffer
  application data
• Cancel Command
• Flags Command Cancel Sequence previous1
  Length 0 Buffer contains no valid data

7
Command Interface on Port

• Callback function must
• Read from/Write to supplied buffer
• Set length Bytes written to buffer (in/out
  param)
• Indicate if an error occurred (return -1)
• Whether more data exists (return 0 gt EOF, return
  gt 0 gt Not EOF, return lt 0 gt ERROR EOF)
• Framework
• generates reply message using same Command value
  and Sequence number.
• sets flags indicating status (EOF, Error etc)

8
Failure Modes

• Library support for lost messages
• if (timeout gt 0, Replies gt 0), then CP API
  library will re-send with RETRY flag set.
• if (timeout gt 0, Replies 0 or all replies
  failed), then API library returns error to
  application
• If (timeout 0 - No Timeout), then send
  operation blocks indefinitely.
• Lost Command message -
• if (timeout gt 0 and retries gt 0), CP resends
  command same sequence number but RETRY flag set.
  Command buffer and flags passed to callback fn.

9
Failure Modes

• Lost Reply message,
• if no retries, Any issues?
• if retries then CP resends
• New Command - Port knows this is a duplicate
  command (RETRY flag). Application responsible for
  handling retries. If an issue can use unique
  message ids. Extreme case use a history (last
  reply message).
• Next Command - Port receives Command w/Sequence gt
  0, w/RETRY flag. Passed to application which
  chooses the correct course of action. The intent
  is to ensure there are no holes in the reply data
  received by the CP.
• Cancel message - same as Next command.

10
Possible Enhancements

• Support asynchronous messaging
• Multiple outstanding commands per port
• Asynchronous I/O on CP
• Speed up boot process and dynamic configuration
• Facilitates implementing port monitoring (ping or
  heartbeat) for fault detection and recovery.
• two methods for reporting results
• upcall - function registered by application is
  called when results arrive
• poll - application periodically polls library for
  results.
• Support Broadcast and/or Multicast

11
MSR Command Layer

• Simple messaging facility optimized for MSR.
• Command message (CP sends)
• Sent by CP to a specific MSR port (unicast)
• Must fit within one AAL0 cell.
• Message header, includes
• protocol version
• Command
• Sequence number
• flags
• Application data follows header
• Library implements Request/Reply protocol.

12
MSR Command Layer

• Reply Message (Port sends)
• Port must send reply message in response to a
  Command message.
• Reply message Header
• version and sequence number same as command msg.
• Includes application data and flags indicating if
  command was successful and if more data exists
  (EOF).
• Application registers command specific callback
  function at port.
• Callback function must conform to specified
  interface.

13
MSR Command Overvew

• Command Protocol description
• Control Processor sends command messages to a
  specific port and expects to receive a reply
  message indicating either Success or Failure.
  This is termed a Command Cycle.
• There is the notion of a Command Transaction
  which may include one or more command cycles. A
  command transaction is terminated when the target
  (port) responds with a reply msg containing an EOF

14
MSR Command Overvew

• Command Protocol description, continued
• CP processing of Reply msg, depends on EOF flag
• If EOF is set then no further reply data is
  available and the command transaction is closed.
• If EOF is not set then there is remaining data
  and the command transaction is still open.
• If remaining data (Not EOF), then CP must follow
  with a either a Next or Cancel command message.
• Sequence number indicates the chunk of data to
  be returned.
• Command indicates the messages destination
• sequence number previous 1

15
APIC Cell Format

• Cell payload contains the MSR Command
• Command header is 4 Bytes, leaving 44 Bytes for
  sub-commands and data.

16
ATM/APIC Header

• pin (Ports-In) - Port cell arrived
• Tx not used (set to 000b)
• Rx port cell arrived on (is the below correct?)
• 001 Port 0, 010 Port 1, 100 Port 2, etc.
• pout (Ports-Out) - Set of output ports.
• Tx Command library sets
• 001 Fiber/Link, 010 Ribbon/Switch, 011 Both
• 101 Loopback MV0, 110 Loopback MV1
• Rx Set by VCXT, see pin above.

17
ATM/APIC Cell Format

• cc (Control Cell Indicator) - Not used, set to
  0b
• ld (Low Delay) - Not used, set to 0b.
• Should we use low delay?
• cid (Connection Identifier) - set to vci value.
• gfc (Generic Flow Control) - set to 0000b.

18
ATM/APIC Cell Format

• vpi (Virtual Path Identifier) - Set to 0x0.
• vci (Virtual Circuit Identifier) - Equal to cid.
• See presentation on MSR configurations for a
  complete list of VCI assignments.
• pti (Payload Type) - Set to 000b (data cell)
• cl (Cell Loss Priority) - Set to 0b (High
  Priority)

19
MSR Command Header
ver
length
command/status
sequence number
flags

• Version (2 bits) - Protocol version. Allows for
  at most 4 versions. Current version set to 0.
• field width was a trade off with the length
  field.
• Length (6 bits) - Number of valid data bytes.
• 0 lt Length lt 44, so 6 bits sufficient.
• This field is indirectly set by the application
  or command implementation. The CP library and
  kernel interfaces allow for applications to pass
  a buffer pointer and indicate the number of valid
  data bytes.

20
MSR Command Header
ver
length
command/status
sequence number
flags

• Command/Status (8 Bits) CP inserts command value,
  SPC/port inserts status information.
• Valid Commands are listed in SYS/msr/msr_ctl.h,
  also see MSR/utils/command/.c,h
• Library API on CP accepts Command as argument.
  implementation in kernel - array of function
  pointers, uses Command as index
• Reply msg Status indicating success or an error
  code (Upcall, ATM, Cmd Invalid, Cmd Not
  Implemented, or Other Cmd Error).

21
Current Commands

• Current set of commands
• set_debug/get_debug - set debug flags/masks.
• port_init - set local port number.
• policy - Manage MSR policy object. Currently
  sets the Policy control flags (Gate 0 enable)
• dq - Preshanth?
• fipl -Ed?
• rp_pcu - extensive set of sub-commands.
• rp_inst - send message to plugin instance
• rp_class - send message to plugin base class

22
MSR DEBUG Command

• Dynamic setting of debug mask - affects what
  messages are sent to syslogd
• set_debug or get_debug command
• Valid debug categories/modules
• apic, ipfwd, iprx, iptx, mem, dq, stats, ctl,
  conf, kern, natm, pcu, plugin, gate
• Valid debug levels - 0 - 255
• predefined verbose, warning, error, critical
• Interface in MSR kernel

MSR_DEBUG((MSR_DEBUG_ltcategorygtMSR_DEBUG_LEVEL_ltl
evelgt, format, args))
23
MSR RP_PCU Command

• rp_pcu Command sub-commands
• addfltr/remfltr - add/remove pkt filter at gate x
• flist - port prints current gate x filter list
• bind/unbind - bind/unbind instance to fltr/gate
  combo
• create - create plugin instance
• free - remove plugin instance
• clist - port prints current plugin class list
• ilist - port prints current instance list
• load - load plugin - not implemented
• null - no-op. Can be used as a ping operation

24
MSR Command Header
ver
length
command/status
sequence number
flags

• Sequence Number (8 Bits) - Is of primary use by
  the applications.
• When command message first sent, sequence 0.
• If the reply does not include an EOF flag, then
  CP increments sequence by one for each subsequent
  command message.
• When EOF is received the Command Transaction is
  complete and the sequence number is reset to 0.

25
MSR Command Header
ver
length
command/status
sequence number
flags

• Flags (8 bits) - Bit field, valid flags are
• Invalid - flag 0, should not occur
• CMD - cell contains a valid command from CP
• REPLY - cell contains reply from Port
• ERROR - Reply only, error processing on Port
• EOF - No reply data remains, end of cmd
  transaction
• NEXT - get next set of reply data
• CANCEL - cancel current cmd transaction
• RETRY - set if cp resend a command after it was
  lost

26
CP Library API

• Library API for application on CP,
• int sendcmd(int sid, int cmd, char data, int
  flags, int dlen)
• sid session id,
• cmd - Command to execute on port
• data buffer pointer,
• flags
• RETRY (reply timeout),
• CANCEL (cancel current command),
• Next (get next set of reply data)

27
MSR Kernel API

• MSR kernel interface SYS/msr/msr_ctl.h,c
• Callback function signature
• msr_ctl_ltcmdgt (void buf, u_int8_t flags,
  u_int8_t seq, u_int8_t dlen)
• buf command buffer w/application data,
• flags
• CMD,
• NEXT,
• RETRY or
• CANCEL,
• seq sequence number indicating reply data set,
  and
• dlen is input/output parameter, data length in
  bytes.

28
Kernel State Diagram
Next
Retry
Command
Command
Command
Cancel
EOF
Proto Error
Closed
Idle
Proto Error
Retry
29
CP Library State Diagram
Retry
Next
Result of a timeout
Wait (for reply)
Protocol Error
Command
EOF
Open Session
Closed
Idle
Protocol Error
30
Example Sending Cmd to Port
Lookup sub-command perform function call then
report results
wugs
192.168.200.X
192.168.204.X
P0
P4
SPC/FPX
SPC/FPX
Next/Prev Hop
Next/Prev Hop
msr_ctl
DQ
DQ
192.168.205.X
192.168.201.X
P1
P5
SPC/FPX
SPC/FPX
reply() plugin instance created Status, Instance
ID
Next/Prev Hop
Next/Prev Hop
DQ
DQ
sendcmd() create plugin instance port id
0, PluginID 200
192.168.202.X
192.168.206.X
P2
P6
Report command completion status to application.
SPC/FPX
SPC/FPX
Next/Prev Hop
Next/Prev Hop
DQ
DQ
192.168.202.2
192.168.203.X
192.168.207.X
P3
P7
SPC/FPX
SPC/FPX
Next/Prev Hop
CP
DQ
DQ
192.168.203.2