Implementing PCE Express HotPlug

1
Implementing PCE Express Hot-Plug

• Carl Jackson
• Hewlett-Packard

2
Session Outline

• PCI Hot-Plug Background
• PCI Express Hot-Plug features
• Native Hot-Plug elements
• Hot-Plug events
• Hot-Plug capabilities, control, and status
  registers
  
• Wake-up and PME
• Reset
• Application to specific form factors
• PCI Express Card electromechanical
• Server I/O module
• Implementation issues

3
Session Goals

• Attendees should leave this session with the
  following
  
• A better understanding of the features and
  functionality of PCI Express Hot-Plug
  
• Hot-Plug design considerations for PCI Express
  Card and Server I/O module form factors
  
• Awareness of various implementation issues
• Knowledge of where to find additional resources
  to ensure a successful design

4
PCI Hot-Plug Background

• What is PCI Hot-Plug?
• Mechanism and methodology to permit I/O adapter
  insertion and removal without requiring system
  downtime or interruption
  
• Places burden on platform HW, minimal impact to
  I/O adapter
  
• Platform must electrically isolate slot and
  remove its power
  
• 1997 PCI Hot-Plug 1.0 introduced
• Defined basic platform, card, and software
  requirements to support hot-plug
  
• User interface left fairly flexible
• No software visible register set defined
  implementation specific
  
• 2001 PCI Standard Hot-Plug Controller (SHPC)
  1.0 and PCI H-P 1.1
  
• Tightened user interface requirements
• Defined standard register set and functional
  model
  
• Permitted OSVs to provide hot-plug support
  without platform-specific software

5
PCI Express Hot-Plug

• PCI Express builds on PCI Hot-Plug and SHPC
  technologies
  
• Integrates standard register set into PCI Express
  Capability Structure in Configuration Space of
  Downstream Ports implementing slots
  
• No memory-mapped or indirect access required
• Defines native toolbox of mechanisms Permits
  PCI Express I/O adapter form factors to support
  hot-plug using self-consistent infrastructure
  
• Form factor specifications define which
  mechanisms are required, optional, or not
  applicable for that form factor
  
• Currently leveraged by the following PCI Express
  form factors
  
• PCI Express Card Electromechanical (CEM)
  Specification, Rev 1.0a
  
• Server / Workstation Module (SIOM)
  Electromechanical Specification (under
  development by PCI-SIG. Rev 0.5 available for
  review)
• ExpressCard Standard, Release 1.0 (developed by
  PCMCIA)

6
PCI Express Topology

• Hot-Plug slots may be sourced from Root Ports and
  from Downstream Ports of switches
  
• Ports appear to software as PCI-to-PCI Bridges
• Each port implementing a slot contains its own
  hot-plug register set in Bridge Configuration
  Space
  
• Hot-Plug registers report presence or absence of
  defined hot-plug mechanisms to software

CPU
Root Complex
Memory
Root Port
Slot 1
Switch
Slot 2
Slot 3
Switch Down-stream Port
7
PCI Express Hot-Plug Elements

• Power indicator Green
• Indicates power state of the slot On or Off
• Indicates power state is transitioning Blink
• Adapter insertion and removal only permitted when
  indicator is Off
  
• Attention indicator Amber or yellow
• Indicates an operational problem exists On
• Also used to visually identify a particular slot
  Blink
  
• Attention button Initiates hot-add and/or
  hot-remove operations
  
• Press again within 5 seconds to cancel
• Hot-Plug signaling messages (in-band) Permit
  electrical control of indicators and attention
  button by device on adapter

8
PCI Express Hot-Plug Elements (2)

• Presence Detect Reports the presence or absence
  of an adapterin a slot
  
• Both pin-based and in-band mechanisms supported
• Manually-operated Retention Latch (MRL) Rigidly
  holds adapter in its slot
  
• Provides mechanical stability for cable
  attach/detach, etc.
  
• MRL Sensor Reports the position of the MRL
• Confirms card is properly installed
• Used to control Vaux, side-band signals as needed
  by form factor
  
• Triggers slot power removal if opened
  unexpectedly
  
• Electromechanical Interlock Physical locking
  mechanism
  
• Prevents adapter removal while power is on
• Optionally provides physical security for adapter

9
PCI Express Hot-Plug Elements (3)

• Power controller Controls main power for an
  adapter
  
• Also reports main and auxiliary power faults
• Slot numbering Uniquely identifies a slot
  within a system
  
• Combination of PCI Express Physical Slot Number
  and optional Chassis Number (as defined in the
  PCI Bridge spec)
  
• Software user interface OSV specific
• Provides method to monitor slot status
• Alternate means to initiate hot-plug operations

10
Hot-Plug Events

• Hot-plug events reported via Slot Status Register
  fields
  
• Slot Events
• Attention button pressed
• Power fault detected
• MRL sensor changed
• Presence detect changed
• Command completed event
• Indicates hot-plug controller ready to accept new
  command
  
• Does NOT guarantee all actions for previous
  command have completed
  
• Hot-plug events may trigger software
  notification
  
• Global enable plus individual enables per event
• Generates a system interrupt for OS with native
  hot-plug support? Redirect to a GPE for ACPI
  based legacy OS
  
• Hot-plug events may also initiate platform wakeup

11
Slot Capabilities Register

• Required in Downstream Ports (switch and root)
  that implement slots
  
• Indicates support for specific hot-plug elements
• Specifies maximum power dissipation for slot (in
  watts)
  
• Write to this register sends Set_Slot_Power_Limit
  message to adapter
  
• Hot-Plug ECN will add Electromechanical Interlock
  Present

12
Slot Control And Status Registers

• Required in Downstream Ports that implement
  slots
  
• Slot Control Register
• Controls specific hot-plug elements
• Individual and global enables for software
  notification
  
• Hot-Plug ECN will add Electromechanical Interlock
  Control
  
• Slot Status Register
• Reports specific hot-plug eventsClear status bit
  by writing 1
  
• Reports hot-plug element state
• Hot-Plug ECN will add Electromechanical Interlock
  State

13
Device Capabilities Register

• Located in Upstream Port of device
• Reports slot power limit last captured by device
• Form factor specification defines default slot
  power limit
  
• Adapter must not exceed default unless higher
  value received via Set_Slot_Power_Limit message
  
• Indicates presence of indicators and attention
  button electrically controlled by adapter
  
• Support requires use of in-band Hot-Plug
  Signaling Messages Link must be up
  
• Hot-Plug ECN proposes to make support for these
  messages optional

Not the complete register
14
Platform Wakeup And PME

• PCI PME Used both to wake system and to
  request power management state change
  
• PCI Express approach Break functionality into
  two separate components while maintaining
  compatibility with PCI-PM
  
• Platform wakeup via WAKE sideband signal or
  in-band Beacon
  
• PM state change request via in-band PM_PME
  message
  
• Dont signal Wakeup to OS
• Make Wakeup visible to platform power controller
  only
  
• Must not be combined with PCI PME
• System and adapter support for Wakeup is
  optional
  
• Requires Vaux enabled for components that
  participate in Wakeup
  
• Includes components that propagate Beacon
• PCI Express devices must support PCI PM
  Capability
  
• Minimally support D0 and D3 states

15
Platform Wakeup And PME (2)

• PM_PME message sent by device requesting PM state
  change
  
• Follows platform wakeup when requesting D3 ? D0
  transition
  
• Link must be active (L0 state) to transmit
  PM_PME
  
• PM_PME message routed upstream to Root Complex
• TLP header uniquely identifies source via
  Requester ID
  
• Exception PCI devices behind a bridge may not be
  uniquely identified
  
• Root Complex stores PM_PME Requester ID in Root
  Status Register and sets PME Status bit
  
• Generates a system interrupt (if enabled) for OS
  with native PME support? Redirect to a GPE for
  ACPI based legacy OS
  
• Deadlock avoidance requires Root Complex to
  discard PM_PME messages if buffer space
  overflows
  
• Initiator re-sends PM_PME if unserviced after 100
  msec timeout

16
I/O Adapter Reset

• PCI Express Cold, Warm, and Hot Reset
• Cold Reset Fundamental Reset accompanying
  application of power Generally signaled via
  PERST
  
• Warm Reset Fundamental Reset without power
  removal and re-application
  
• Hot Reset Signaled in-band (physical layer)
• All device state must be re-initialized by
  Fundamental Reset
  
• Exception Specified Power Management context
  when Vaux is enabled
  
• Sticky registers excluded from Hot reset
• E.g., many Advanced Error Reporting registers
• I/O adapters with on-board voltage regulation
  must be properly reset with PERST assertion
  
• On-board voltages may not decay sufficiently to
  trigger self-detected reset when slot power is
  cycled

17
PCI Express Card (CEM) Form Factor

• Mechanically similar to traditional PCI cards
• 3 power rails 12V and 3.3V (required) plus
  3.3Vaux (optional)
  
• 3 power consumption limits

• x1 cards default to 10W max
• Long cards can consume up to 25W with Slot Power
  Limit increase
  
• X16 cards default to 25W max
• Graphics cards can consume up to 75W with Slot
  Power Limit increase

18
PCI Express Card (CEM) Form Factor (2)

• Power Indicator, Attention Indicator, and
  Attention Button located on chassis adjacent to
  each slot
  
• Electrically controlled by associated Root or
  Switch Downstream Port
  
• Platform hot-plug power control circuitry
  required for each slot
  
• Removes power from slot for card insertion /
  extraction
  
• Provides in-rush control during power-on
• Max card capacitance spec bounds voltage ramp
• Monitors power status reports under-voltage and
  over-current faults
  
• 3.3Vaux must be disconnected when MRL opened
• Detected by MRL Sensor
• Also disconnect specified side-band signals
• Unexpected MRL opening also triggers main power
  removal
  
• Protect platform and adapter from damage

19
PCI Express Card (CEM) Form Factor (3)

• Presence detect pulls down hot-plug controller
  input upon card insertion
  
• Connector pin location, shorter length help
  ensure card fully inserted
  
• Multiple PRSNT2 slot connections supports
  up-plugging
  
• E.g., x1 card in a x4 slot

20
Server I/O Module (SIOM) Form Factor

• Under development by PCI SIGSpec release
  anticipated this fall
  
• Single axis insertion / extraction
• Simplifies hot-plug
• Single and double module widths
• Supports both front / back and bottom / top
  cooling airflow
  
• 2 power rails 12V main power plus 3.3Vaux
  (both required)
  
• 2 power consumption ratings
• 25W for single wide module
• 50W for double wide module
• Single wide supports x8 lane width Double wide
  supports x16

21
Server I/O Module (SIOM) Form Factor (2)

• Power Indicator, Attention Indicator, and
  Attention Button located on module faceplate
  
• Electrically controlled by the Root Port or
  Switch Downstream Port
  
• Requires sideband signals on connector
• Slot power (12V 3.3Vaux) always available at
  slot
  
• Staggered pins enable 3.3Vaux consumption on
  module insertion
  
• PWREN from hot-plug controller enables bulk
  power consumption
  
• Module provides bulk power conversion
• On-board power monitoring reports power good,
  fault status
  
• Presence detect also reported using staggered
  pins
  
• WAKE support required (signal inverted from CEM
  usage)
  
• SMBus support with Vital Product Data EEPROM
  required
  
• Optional internal storage interface supports SAS,
  SATA

22
Server I/O Module (SIOM) Form Factor (3)

• Example Hot-Plug Interface Implementation

23
Recent Implementation Issues

• Command completed Hot-plug controller ready for
  next command
  
• Does not mean action from previous command is
  complete
  
• Example Indicator message sent to adapter, not
  LED state changed
  
• Also applies to power controller commands
• Example problem areas for software
• When does PERST deassert? Timing delays depend
  on thisE.g., 100 mSec delay to first config
  access
  
• When is slot power off? Need for Power Indicator
  (software controlled)Also for On ? Off ? On
  power transitions
  
• Will be addressed in Hot-Plug ECN
• Link status Link speed, width fields only
  defined when link is up
  
• Will be addressed in Surprise Down Error and Link
  Up Additions ECN
  
• Electromechanical interlock Position
  persistence after shutdown
  
• Requires position sensor, use control bit write
  to toggle state
  
• Will be addressed in Hot-Plug ECN

24
Call To Action

• Ensure your platform and/or add-in card(s) meet
  the requirements of the PCI Express Base spec and
  applicable errata
  
• Watch for Surprise Down Error and Link Up
  Additions ECN and upcoming Hot-Plug ECN
  
• Consider each form factor when planning your
  designsCommon mechanisms make support
  straightforward
  
• Provide feedback on the SIOM draft spec see
  PCI-SIG website
  
• Take advantage of PCI-SIGs training, support,
  and compliance (plugfest) programs

25
Additional Resources

• PCI-SIG Member Services
• PCI, PCI-X, and PCI Express Specifications
• Developers conferences, compliance workshops,
  training
  
• Technical support via email techsupp _at_
  pcisig.com
  
• Early access to draft specifications, errata, and
  ECNs
  
• See http//www.pcisig.com for information
• Other Resources
• Unofficial email list pci-sig _at_ znyx.com
• PCI Hot-Plug Applications DesignAlan Goodrum,
  Annabooks 1998

26
(No Transcript)