Systems and Operations

1
Systems and Operations

• CCDs Presentation
• December 12, 2007

2
(No Transcript)
3
Services Server Farm Systems Administration
Storage Farm EZ-Backup VMware (In
Development)
4
Server Farm

• Manager Rick Polcaro
• Staff Peggy Roberts
• Randy Smith

5
Server Farm

• The CIT Server Farm provides a secure environment
  for housing departmental servers. There are more
  than 500 servers in the facility today. The
  service includes 24 hour system monitoring and
  reboot, network connectivity, and uninterruptible
  power with generator backup. Servers are mounted
  in CIT provided racks. It is the customers
  responsibility to provide hardware and software
  maintenance.

6
Server Farm History

• The server farm began in CCC in the mid-1980s
  with a few Unix systems for data warehousing and
  research.
• Our first departmental user was Dining. CISER
  soon followed.
• Major CIT servers into the 1990s
• Instruct Machines
• EZ-Remote servers
• Postoffice machines
• P2K Servers
• Library Servers
• Server farm moved from CCC to Rhodes Hall in 1997.

7
Server Farm Facilities

• Raised Floor in Rhodes and CCC.
• Emergency power and generator backups.
• Standard electrical power consists of 120/208
  vac, with 30 amp three prong receptacle. Special
  requests are directly billed.
• Physical security to authorized personnel only,
  with security cameras.
• Basic support, via the NOC, for 24/7 monitoring
  and basic troubleshooting, rebooting, and system
  administrator notification.

8
Server Farm Facilities

• Costs
• Space is charged by the U, an industry standard
  measurement in a 19 rack 1.75 vertical inches.
• The monthly server charge (server Us
  7.60)
• There will be a one time installation charge of
  118 per server
• The cost for a network connection(s) and NUBB
  will be charged directly to the users University
  account by CIT NCS.

9
Server Farm Growth
10
Server Farm Customers

• Distribution (number of servers)
• CIT Owned 456
• Lab of O 17
• Library 29
• Comp Sci 14
• CCE 10
• Others 60 (representing 29 depts)
• Total 586 (12/07)

11
Questions

• Information / Contact
• Server Farm http//www.cit.cornell.edu/services/s
  erverfarm
• serverfarm_at_cornell.edu
• ???

12
Systems Administration Support

• Manager Mariann Carpenter
• Staff Muhammad (Moe) Arif
• Mary Cronk
• Doug Flanagan
• Glen Hoffman
• Kent Ross
• David Shirk
• Thomas Walden
• Solomon Welch

13
Systems Support

• Systems Support provides systems administration
  and systems programming support for projects that
  span the 400 servers and 75 terabytes of storage
  located in our machine rooms.
• The Systems Administration team provides the
  hardware and operating system support for CITs
  servers and storage assets.
• Systems Engineering provides and enhances tools
  for our production servers and services, such as
  NetVigil, as well as specific support for the
  DNS/DHCP service and its ancillary systems.

14
Client Systems Services

• Systems Administration Services
• Quote, OS Install, OS Maintenance, Backup
  Configuration, Userid Maintenance, Application
  setup assistance, Retirement
• The following operating systems are currently
  supported
• SUN Solaris
• IBM AIX (no new installations)
• Windows 2000 (no new installations)
• Windows 2003 (standard and enterprise editions)
• RedHat Linux Enterprise AS 4.0
• Installation into the Server farm facility.
• Monitoring with 24/7 Support as required.

15
Client Systems Activities

• Routine Administration.
• Coordination of hardware maintenance contracts.
• Lifecycle management of servers, working with our
  customers.
• Automation projects to leverage our staff
  investment.

16
Systems Administration Demographics

• We support the OS on approximately 400 servers.
• Distribution by Operating System
• 74 Solaris
• 1 AIX
• 25 Wintel, Linux

17
Questions

• Information / Contact
• Mariann Carpenter 255-7707
• mgc1_at_cornell.edu
• ???

18
Storage Farm Service

• CITs Storage Farm service provides storage
  capacity, connectivity, and management services
  to servers housed in CITs Server Farm and
  managed by CITs Systems Administration Support
  group.

19
Storage Farm Staffing

• Manager of Storage Services
• Paul Zarnowski
• Manager of Systems Administration
• Mariann Carpenter
• Storage Engineers
• David Shirk
• Kent Ross

20
Storage Farm Historical Growth

21
Storage Farm Efficiencies

• Centralized Management
• Small set of devices to support maintain
• Common management tools
• Fewer resource baskets to manage
• Centralized Procurement Process
• Fewer purchases (hardware, maintenance, etc)
• Streamlined process
• Higher Storage Utilization

22
Storage Farm Advantages

• Easy to Use
• Ease of storage allocation
• Faster storage allocation
• Maintenance handled by Storage Farm staff
• Technology upgrades
• Financial Advantages
• No up-front capital expense or depreciation
• Grow storage as needed (pay as you go)
• Risk reduction
• Lower overall costs (considering all components)

23
Storage Farm Components

• Storage Connectivity
• Storage Tiers
• Storage Management

24
Storage FarmConnectivity Options

• Fibre Channel Storage Area Network (FC-SAN)
• Single-attached, to one fabric (red or green)
• Redundancy for Storage Array connectivity only
• Dual-attached, for higher availability (red AND
  green)
• Redundancy for both Array and Server connectivity

25
Storage Farm Services Storage Tiers
26
Storage Farm ServicesManagement Services

• Storage Network connection setup
• LUN allocations expansions
• LUN tier migration
• Performance Capacity planning
• Storage device firmware upgrades
• Maintenance procedures
• 24x7 Health monitoring of storage
  subsystem(out-board from servers FC HBA)
• Troubleshooting of storage hardware software
• Procurement
• Life-cycle upgrades

LUN Logical Unit Number (aka logical disk)
27
Recent Storage RFP Goals
(Spring, 2007)

• Improved Storage Management
• Storage hot spots
• LUN resizing effort
• Reporting
• Cheaper Storage Connectivity
• iSCSI support
• Lower Storage Costs
• Maintenance
• Right-placing data
• Decrease head room costs

iSCSI SCSI over TCP/IP
28
Storage RFP Process

• Storage RFP Review Committee
• Paul Zarnowski, David Shirk, Kent Ross, Don
  MacLeod, Tony Damiani, Ken Friedman, Mike
  Hojnowski, Brian Messenger
• 15 Vendor Proposals reviewed
• 8 On-site Vendor Presentations
• 4 Finalists
• Compellent vetted selected

29
Storage Center
Scalable Enterprise-class SAN solutionwith
block-level intelligence

• Stable performance
• All I/O spread balanced across many disk
  spindles
• No more hot spots
• Reduce capital expenditures
• Thin provisioning
• Automated Tiered Storage
• Supports any open-systems server without agents
• Connects via FC, iSCSI or both
• Scales from 500 GB to over 300 TB
• Cut operating expenses
• Disk virtualization simplifies administration
• Eliminate the need for 3rd party software
• Enhanced data availability
• Unlimited snapshots without full copies
• Replication flexibility for low-cost DR

30
Automated Tiered Storage
Automatic block movement to lowest cost storage
Reducing cost of storing data

• Problem
• 70-80 of data is inactive
• Manual classification of data
• Difficult to migrate data between tiers
• Add-on software cost and complexity
• Solution Data Progression
• Automatic data classification and migration
• Move blocks within files
• Bi-directional data movement
• Move snapshots to lower tiers
• Benefits
• Lower drive costs by up to 74
• Fewer FC drives and better performance

Compellent Solution
31
Thin Provisioning
Thin provisioning lowers disk cost and improves
performance
Traditional Allocation

• Problem
• Disk acquisition cost
• Expanding and shrinking volumes
• 40-60 allocated but unused space
• Solution Dynamic Capacity
• Create any size volume
• Allocate as usual, utilize on write
• Keep unused disk space available
• Constantly monitor storage usage
• Benefits
• Cut disk acquisition costs in half
• Delay future purchases of disk
• Never take system offline
• Performance gains - only manage actual data
  (written blocks)

CRM
Database
E-mail
Available Disk Space 0 GB

• Unused
• Allocation

• Unused
• Allocation

• Unused
• Allocation

Data
Data
Data
Actual data 1.2 TB Physical storage consumed
3.5 TB
Thin Provisioned Allocation
Available Disk Space 2.3TB
CRM

• Unused
• Allocation

Database
E-mail

• Unused
• Allocation

Data
Data
Data
Actual data 1.2 TB Consumed storage 1.2 TB
32
Storage Farm Roadmap

• SAN Fabric simplification
• Elimination of older Brocade switches
• iSCSI connectivity
• Initially deploy within Server Farm environment
• Dual path FC / iSCSI connectivity
• Windows initially
• Improved performance
• Higher spindle count
• Snapshot capabilities

33
Potential Future EnhancementsRecovery Options
Technology
Snapshot
Synchronous
Asynchronous
Data Instant Replay
Remote Instant Replay Synchronous
Remote Instant Replay Asynchronous
Local (Rhodes)
Campus (CCC)
Wide Area (Weill)
Distance
34
Storage FarmPotential Future Enhancements

• Tier 4 storage service
• Very large capacity
• Low use
• Low management
• Long-term commitment
• Low cost
• Network-Attached Storage (NAS) services
• File-level storage (e.g., CIFS, NFS)
• iSCSI storage for non-managed servers

35
Storage Farm Rates
36

• Questions ???
• Contact
• Paul Zarnowski
• psz1_at_cornell.edu
• Flash demo of Automated Tiered Storage and Data
  Progression
• http//www.compellent.com/products/demo/demo_ats.h
  tml

37
EZ-Backup is . . .

• A Centralized Backup solution that offers
• Network-based
• Automated backups
• Automated management of backup data
• User-driven restore
• Off-site backups (but still on-Campus)
• Same solution used to back up CITs server farm
• Available to all Cornell departments

38
EZ-Backup Staff

• EZ-Backup Support Team
• Robert Talda
• David Beardsley
• Randy Barron
• Joanne Button
• Michelle Mogil
• Ron Seccia
• and many others throughout CIT
• Paul Zarnowski (Service Manager)

39
Distributed Support Model

• EZ-Backups Distributed Support Model focuses
  expertise where its most beneficial
• Department Support Staff
• In best position to understand and address user
  needs
• Central (CIT)
• In best position to support backup technology
• Provide tools to, and support, Department Support
  Staff
• Vendor
• Available for escalated support, when needed

40
EZ-Backup Supported Platforms
Available for most platforms in use at Cornell.

• Operating Systems
• MacOS
• Leopard support imminent
• Windows
• Including Vista
• Unix (most flavors)
• Linux (RH, SuSE)
• Sun Solaris
• AIX
• Others
• Netware

• Apps / Databases
• Oracle
• Microsoft SQL Server
• Microsoft Exchange
• Microsoft SharePoint
• Others

41
Tivoli Storage Manager

• EZ-Backup utilizes IBMs Tivoli Storage Manager
  software (TSM)
• TSM is one of the largest enterprise-class
  backup/archive storage products available on the
  market
• TSM is very scalable, and can grow to meet our
  expanding storage requirements
• TSM is extremely flexible it can probably do
  what you need
• Flexibility comes with complexity If you need
  help figuring out how to do something, ask us!

42
EZ-Backup Growth(14 years)
43
EZ-Backup Recent Changes

• New Client Support
• Windows Vista
• Macintosh Leopard (imminent)
• New TSM Client Software
• New Version 5.4
• Older versions upgraded to latest patch
  levels(for older OSes)
• Recommend upgrade to latest software levels
• Merging of CTC TSM service into EZ-Backup
• 2nd tape library will give EZ-Backup two-site
  capability

44
EZ-Backup Less-recent Changes

• Dynamic Sub-file backup
• Very useful for backing up over slower network
  connections, such as Dial-up, DSL, RoadRunner
• Journal-Based Backup
• Speeds backups for large fileservers w/ low
  change rate
• Ability to delete individual backup files
• Include/Exclude Preview Capabilities
• Open File Support
• Encryption Enhancements
• Server Split
• CIT On-Site Solutions support available
• Available on Windows clients only

45
EZ-Backup Roadmap

• Upgrade EZ-Backup/TSM Storage Subsystem
• RFP Goals
• Technology refresh
• Lower storage costs
• Provide faster restores for servers with high
  object count
• Consider use of low-cost disk storage and data
  de-duplication
• Training Classes
• Off-site Capability (Weill Medical Center)
• Pending funding decision
• Enhanced Reporting Management Tools

46
Current Rates (per Month effective July, 2007)
47
EZ-Backup Pricing History
48
EZ-Backup

• Questions ???
• Contact
• Paul Zarnowski
• psz1_at_cornell.edu
• More information is available at
• http//ezbackup.cornell.edu

49
Virtual OS Hosting Project

• Mike Hojnowski
• Manager, Special Projects

50
Project Deliverables

• Provide Windows hosts under VMware supported by
  SO Sysadmins. Note that Linux is formally not
  in scope at this time.
• Develop a financial model to support to support
  the service for CIT internal customers.
• Develop process improvements to minimize the
  effort involved in creating virtual instances.
• Implement the system in a way that facilitates
  Emergency Preparedness.
• Document this service.

51
Current Environment
Prod
Test
Dev
52
Current Environment

• Storage Compellent via SAN
• Thin Provisioning
• Fast full copy, and Snap copy capabilities
• Automated Tiered Storage
• Production Servers (3)
• IBM 3650
• 2-Socket, 4-Core, 2.66 Ghz
• 24G Ram
• 4 Gbit NICs
• 2 SAN HBAs
• Expected capacity 50-60 Virtual Machines

53
Project Status

• Project Plan approved 10/05/07.
• Project implementation has been repeatedly
  delayed, due to the Exchange project.
• Staff resources are freeing up, and were ramping
  up our efforts.

54
Timeline

• Development
• 5/1/07 12/31/07 (Lengthened due to resource
  constraints, cutting the early adopters phase
  shorter than originally planned).
• Early Adopters
• 1/1/08 2/28/08
• Full Production
• 3/1/08

55
Issues

• SAN failover
• Presently we dont replicate storage from Rhodes
  to CCC.
• We wont have full building failure protection
  on go-live.
• We will address this with either storage
  technology, or new features in a coming release
  of VMware after go-live.

56
Futures

• Linux (First half 2008)
• Solaris (First half 2008)
• Virtual Co-lo as a Designated Service? (FY09?)