IP Storage: Best Practices

1
IP Storage Best Practices

• Marc Staimer, President CDS
• Dragon Slayer Consulting
• marcstaimer_at_earthlink.net
• 503-579-3763

2
Agenda

• IP Storage Level Setting
• File vs. Block Storage
• FCIP, iFCP iSCSI
• Fiction Facts about iSCSI
• Storage Replication over WANs
• Considerations for Designing IP Storage Networks
• Questions

3
IP Storage Level Setting

• There are 3 types
• Block Storage
• iSCSI
• File Storage
• NAS
• Storage over WAN for business continuity Block
• iSCSI
• FCIP
• iFCP

4
IP Storage Level Setting

• iSCSI Storage
• Block-based external storage on Ethernet
• Vs. SCSI, USB, 1394, or FC
• NAS or Network Attached Storage
• File-based storage NFS CIFS
• No different than any other file server
• Requires block storage behind it

5
By show of hands, what is acceptable packet loss
for IP Block Storage on a LAN/WAN?
Hosted by

1. 1
2. 5
3. 10
4. 0

6
Acceptable Packet Loss for IP Block Storage on a
LAN/WAN

• The Answer is

0
7
Block vs. File Storage

• IP block data is unlike any other IP data
• Overwhelms most current LAN/WAN environments
• Incredible amounts of traffic
• Tolerates ZERO packet loss
• Very low latency
• File storage specialized file server
• NFS CIFS
• Higher prioritization is required depending on
  app
• Volume of data may overwhelm untuned LAN/WAN

8
FCIP, iFCP, iSCSI

• FCIP
• Fibre Channel tunneled in TCP/IP
• IP transport between FC switches
• iFCP
• IP header put on Fibre Channel frames for routing
• IP connection services for FC devices
• iSCSI
• SCSI-3 mapped to TCP

9
FCIP

• Pt-to-Pt Becomes One FC SAN
• Disruptions pass SAN-to-SAN
• Large FSPF database
• PSS between SAN sites
• Gateways between fabrics (blades or boxes)

10
iFCP

• Pt-to-Multi-Pt
• Device specific passing only the data that is
  required
• Devices can appear in multiple individual SANs
• The SANs themselves remain independent

11
Parable
12
THERE AIN'T NO FREE LUNCH!
13
iSCSI Ethernet-Based SAN

• The Hype
• Block storage on Ethernet
• Leverage current infrastructure investment
  knowledge base
• Lowers cost
• Eliminates headaches
• Ubiquitous
• Makes FC another Ethernet Road Kill

14
iSCSI Defined

• IETF standard protocol
• Establishes manages connections
• Carries storage (SCSI) blocks
• From initiators to storage targets
• Encapsulates SCSI blocks in TCP/IP
• Then tunneled in Ethernet
• iSCSI is to Ethernet as FCP is to Fibre Channel
• Network application
• One infrastructure for LANs, NAS, SANs

Ethernet Frame
TCP/IP Packet
SCSI-3
15
iSCSI Applications

• NAS/SAN combined storage units
• Entry level SANs
• Limited budget SANs

16
SAN Benefit Assertions of iSCSI

• Reduced costs
• Professional services, implementation, management
  IT Staff time
• Reduced complexity
• Reduced Management
• Increased Interoperability
• Elimination of Multiple Networks
• Unlimited SAN Distance
• Equal or Better Performance

17
Fiction Facts about iSCSI

• Fiction
• Cost is lots lt Fibre Channel (FC)
• Complexity is much lt than FC
• Uses current infrastructure
• Requires no storage knowledge
• Is as fast as FC
• Will replace FC in Enterprise
• Easier to manage than FC
• Eliminates SAN distance limits
• Latency (delay) is not an issue

• Facts
• Known technology
• Costs are relatively lt FC
• Cycles or hardware
• Doesnt require special HW
• But benefits from it
• Latency (Delay) matters
• Cant be gt 1 millisecond
• Deterministic routing
• Doesnt require any-to-any

TRUTH!
18
Rating the iSCSI Value Props

• Reduces or Eliminates SAN Professional Services
• Lowers SAN Hardware Costs
• Simplifies SAN Management
• Eliminates Interoperability Issues
• Converges SAN/LAN/MAN/WAN Fabric Infrastructure
• Extends SANs over unlimited distances
• Equal or better performance than FC SANs

19
iSCSI Reality Check

• There are some real cost benefits for
• NAS/SAN on the same fabric infrastructure
• SANs that dont need the performance of FC
• Entry SANs that may not even need GigE and TOEs
• Hype overshadows reality
• GigE NICs with iSCSI and TOEs cost same as FC
  HBAs
• FC ports GigE ports on server motherboards
• Makes port cost differences higher for GigE
  w/TOEs
• Very low cost simplified FC switches
• Have erased much of the infrastructure HW
  differences

20
Sample iSCSI Vendors

• Storage NICs
• QLogic
• Intel
• Alacritech
• Adaptec
• Emulex

• Switches
• Cisco
• Extreme
• Foundry
• Enterasys
• Nortel
• Lucent
• 3Com

• Gateways
• Cisco
• McDATA

• Silicon
• Adaptec
• Alacritech
• Intel
• Siliquent
• QLogic

21
By show of hands, is latency (delay) important to
iSCSI block storage?
Hosted by
22
Is Latency Important to IP Block Storage?

• The Answer is
• Yes, for the most part
• It also depends on application

23
By show of hands, who believes that TOEs iSOEs
are an iSCSI block storage requirement?
Hosted by
24
Are TOEs iSOEs an iSCSI block storage
requirement?

• The Answer is Not necessarily

25
Spectrum of iSCSI Adapter Solutions
26
iSCSI No TOE

• Definition
• Std Ethernet NIC
• TCP/IP iSCSI
• Host-based in drivers
• Who
• Microsoft Cisco
• Advantages
• Lowest cost
• NICs available today
• Easy integration with OS
• Disadvantages
• Lowest performance
• High CPU load
• High interrupts
• Once/packet
• Many/ TCP segment

27
iSCSI Little TOE

• Definition
• NIC w/limited TOE
• Packets in order no frags
• Out of order etc. go to OS
• Who
• Alacritech
• Advantages
• Relatively low cost
• Small layout (low profile card)
• Good throughput w/pristine Ethernet
• Disadvantages
• Out-of-order frags lt performance
• Interrupts Once/TCP segment
• Many/IO
• OS interface challenges

28
iSCSI Firmware TOE iSOE

• Definition
• TCP/IP iSCSI firmware
• On-board processors
• Who
• Adaptec, Intel, Emulex, QLogic
• Advantages
• Flexibility to change code
• Low CPU load
• Low interrupt load lt 1/IO
• Disadvantages
• No 10Gb scaling
• Performance
• Power, size

29
iSCSI Hardware TOE iSOE

• Definition
• Hardware ASIC
• TCP/IP bulk data path
• iSCSI digest (CRC)
• iSCSI in processors
• Who
• QLogic, iReady
• Advantages
• Flexibility to change iSCSI code
• Low CPU load
• Low interrupt load lt 1/IO
• Performance, scaling to 10G
• Disadvantages
• Complex chip
• Lack flexibility to change TCP code

30
A TOEs Impact on iSCSI
31
By show of hands, who believes that iSCSI allows
block storage to go unlimited distance?
Hosted by
32
Will iSCSI allow block storage to go unlimited
distance?

• The Answer is Yes No
• Latency is the limiting factor
• Application dependent
• Transactions cannot exceed 1ms one way (100
  miles)
• Asynch replication is not distance dependent

The speed of light, is not just a limit, its a
law.
33
Key Block IP Storage Issues

• Distance
• Latency
• WAN bandwidth utilization of IP
• Security
• Encryption
• Access
• Performance
• Must be to, or gt than current expectations

34
Storage Replication over WANs

• Issues
• Good Citizen on Shared TCP/IP WANs
• Filling the pipe gt 50
• End-to-end throughput
• Compression
• TCP latency

35
Changing Paradigm for Asynch Storage Replication

• Native Storage GigE interfaces emerging
• EMC Symm5 and DMX are available today
• EMC CLARiiON in development
• Hitachi developing GigE for Lightning and Thunder
  
• Software Replication Apps over native IP
• Leverages IP WAN already in place
• Eliminates SAN gateway requirement (FCIP or iFCP)
• Significant lt cost Mirror/Replication apps

36
High Speed TCP/IP Data Transport Challenges

• Optimized for
• Small payloads relatively short distances
• Employs inefficient
• Error recovery session management techniques
• Delivers poor bandwidth utilization
• For most high performance applications
• Usually lt 30 efficiency at extended distances
• Even less as distance and bit errors increase

37
Cost of Inefficiency

• Higher Bandwidth Cost
• Despite lt costs, high speed (DS3, OC3, etc)
  circuits expensive
• DSC survey of 200 end-users
• BW 50-70 of storage replication costs
• Operational Inefficiencies
• Cant complete within time window delaying
  production ops
• Explosion in data exacerbates the problem
• Current specialized equipment separate
  networks
• Cant fully leveraging IP infrastructure gt costs

38
Native GigE Replication SRDF

• Adaptive Copy SNAP/Asynch
• Performance degrades starting at 300 miles
• At 500 miles performance degradation is
  noticeable significant

39
Native GigE Replication SRDF RFC 3135

• Adaptive Copy SNAP/Asynch
• RFC 3135 TCP/IP Performance Enhancing Proxy
• Up to satelite distances (46K miles roundtrip)
• 90 bandwidth utilization (T1/E1, DS3, OC3,
  OC12)
• Plus 2 to 4 to 1 compression
• Who
• NetEx (HyperIP), Expand, NetCera, Digital Fountain

Ethernet LAN/WAN Switches
40
EMC SRDF Replication over WANs Replication
Methodologies Illustrated
41
Considerations for Designing IP Storage Networks

• Separate LAN fabric
• Minimally, separate VLAN
• Layer 2 switching
• Best latency for Ethernet switching
• Nothing less than GigE
• Understand LUNs
• Mapping and Masking

42
Questions?