ISTORE:%20An%20Introspective%20Storage%20Architecture%20for%20Network%20Service%20Applications

1
ISTORE An Introspective Storage Architecture for
Network Service Applications

• Aaron Brown, David Oppenheimer, Kimberly Keeton,
  Randi Thomas, Jim Beck, John Kubiatowicz, and
  David Patterson
• http//iram.cs.berkeley.edu/istore
• 1999 Winter IRAM Retreat

2
Agenda

• Overview of ISTORE Motivation and Architecture
• Hardware Details and Prototype Plans
• Software Architecture
• Discussion and Feedback

3
Motivation

• Emerging paradigm of network-based services
• today e-commerce, online database services,
  search engines, and web servers
• tomorrow above, plus thin-client/PDA
  infrastructure support
• These services have unique demands
• vast storage needs (large operational and
  historical DSS databases)
• high access rates
• mix of transactional (ACID) and non-transactional
  accesses
• easy incremental scaling of storage and
  processing
• minimal administration burden
• easy setup, scaling, maintenance, repair, tuning,
  ...
• automatic dynamic adaptation to failures,
  workload changes, ...

4
Scaleable Storage Appliances

• Best architecture to meet these demands is a
  scaleable self-maintaining storage appliance
• appliance a single-function device optimized to
  provide one application or service
• consists of software and hardware constructed
  with that service in mind
• interface to WAN/LAN via standard high-level
  protocols
• internal implementation hidden
• storage appliance integrates an application with
  its data storage in one unit
• scaleable appliance incrementally scaleable with
  automatic integration of new resources (e.g.,
  disks)
• self-maintaining appliance runs autonomously,
  without human intervention, once initially
  configured
• Examples e-commerce, decision-support DB, PDA
  proxy, video server, or online voting appliances

5
Todays Appliances

• Some storage appliances exist today
• Network Appliance NFS and proxy servers
• SNAP! Server file server
• Qube web server
• SonicWall firewall
• ...
• But none are truly self-maintaining, and most are
  not scaleable

6
Why Arent There Appliances That Match Our
Criteria?

• 1) Self-maintenance is tricky and
  application-dependent
• cant rely on human administrators to configure,
  monitor, and tune system for its application
• failure management
• devices must fail fast without interrupting
  service
• failures should not require immediate human
  intervention
• performance management
• system must adapt to changes in workload or
  access patterns
• system upgrades and scaling
• new hardware resources must be automatically
  incorporated without interruption of service
• new devices should be used immediately to improve
  performance or repair prior failures

7
More Appliance Challenges

• 2) Appliance designs are inextricably tied to
  single applications
• each service or application requires a
  custom-designed appliance
• new systems software must be written, tested,
  debugged
• often can share hardware, but entire system must
  be retested
• especially difficult to design new
  self-maintaining appliances
• performance and manageability suffer when an
  appliance is used for other than its intended
  purpose
• e.g., if workload doesn't quite match appliance
  design goals
• example using a file-server appliance as the
  storage subsystem for a database application

8
The ISTORE SolutionAn Introspective
Meta-Appliance

• A meta-appliance is a generic hardware and
  software platform that can be specialized into a
  single-application, focused appliance
• An introspective system uses hardware and
  software monitoring to tune itself and to achieve
  self-maintenance
• The ISTORE meta-appliance is a hardware/software
  architecture for building scaleable
  self-maintaining storage appliances

9
ISTORE Hardware Innovations

• 1) ISTORE uses intelligent hardware
• ISTORE is a shared-nothing distributed system of
  intelligent device bricks interconnected by an
  intelligent chassis

CPU, memory, NI
IntelligentChassis
Device
IntelligentDevice Brick

• intelligent device brick a device (e.g., a disk)
  plus a fast embedded CPU, memory, and NI (e.g.,
  an IRAM)
• intelligent chassis scaleable, redundant, fast
  network UPS

10
Benefits of ISTOREs Hardware Architecture

• Decentralized processing (shared-nothing)
• system can withstand partial failure
• Intelligent devices monitor their own health,
  test themselves, manage failures, collect
  application-specified performance data, and
  execute application worker code
• provides the foundation for self-maintenance and
  self-tuning
• Plug play, hot-swappable bricks ease
  configuration, scaling
• hardware is specialized/scaled by selecting an
  appropriate collection of devices
• disks, DRAMs, WAN/LAN interfaces, ...

11
ISTORE Software Innovations

• 2) ISTOREs system software is designed to
  deliver self-maintenance and allow for easy
  specialization
• layered mechanism libraries facilitate
  introspection, self-maintenance
• extensibility via domain-specific languages
  (DSLs) allows customization of the runtime system
  with application-specific policies
• Benefits of ISTOREs software architecture
• appliance designer inherits functionality of
  supplied base mechanisms for monitoring and
  adaptation
• easy for designer to customize mechanisms and
  define adaptation policies by writing
  domain-specific code

12
ISTORE and IRAM

• ISTORE relies on intelligent devices
• IRAM is an easy way to add intelligence to a
  device
• embedded, low-power CPU meets size and power
  constraints
• integrated DRAM reduces chip count
• fast network interface (serial lines) meets
  connectivity needs
• Initial ISTORE prototype wont use IRAM
• will use collection of commodity components that
  approximate IRAM functionality, not size/power
• but results will be applicable to IRAM-based
  systems
• success with ISTORE will encourage integration of
  IRAM-like processors into standard devices

13
Agenda

• Overview of ISTORE Motivation and Architecture
• Hardware Details and Prototype Plans
• Software Runtime Architecture
• Discussion and Feedback

14
Backup Slides
15
Related Work

• ISTORE extends several recent research efforts
• Intelligent/Active Disks, NASD (UCB, UCSB, CMU)
• used as a component in ISTORE ISTORE adds a
  global runtime system, introspection, and
  self-maintenance
• network service appliances (NetApp, Snap!, Qube,
  Swarm, ...)
• ISTORE adds the idea of a meta-appliance
  self-maintenance
• self-maintaining systems (Sun Project StoreX,
  Tandem, ...)
• extensible operating systems (SPIN, Exokernel,
  VINO, Scout,...)
• domain-specific languages (OGI Microlanguages,
  Prolac, VHDL,...)
• adaptive systems (HP AutoRAID, MS AutoAdmin, MS
  Millennium)
• plug-and-play system construction (Sun Jini, PC
  PlugPlay, ...)
• ISTORE leverages Berkeleys experience with
  reliable, large, distributed storage systems and
  applications
• RAID, Tertiary Disk, XFS, Sprite, NOW,
  PostGres/GiST, ...