Reliable Adaptive Distributed Systems

1
Reliable Adaptive Distributed Systems

• Armando Fox, Michael Jordan, Randy H. Katz, David
  Patterson, George Necula, Ion Stoica, Doug Tygar

2
Motivations and Observations

• Existing systems
• Fragile, easily broken, poor reliability/security
• Overwhelming configuration complexity,
  infrequently correctly performed, yielding lack
  of dependability, significant vulnerabilities
• Magic bullets?
• Statistical learning theory foundation for
  algorithms that observe/predict future behaviors
• Verification technologycheck for correct
  behavior, reveal vulnerabilities, automatically
  generate behaviors with desirable properties
• Programmable network elementsactive code
  inserted into network, provide observation/enforce
  ment points without access to user end systems

3
The Team and Approach

• Team
• Statistical Learning Theory (SLT) (Michael
  Jordan)
• Network Services/Protocols (Armando Fox, Randy
  Katz, David Patterson, Ion Stoica)
• Verification Methods for program, network,
  security behaviors (George Necula, Ion Stoica,
  Doug Tygar).
• Approach comprehensive distributed system
  architecture
• SLT building block/practical components for
  distributed system observation, coordination,
  inference, correction, and evolution of behaviors
• Network behaviors and how they reveal operation
  of higher-level network applications
• Key enabler embed observational and inference
  means at strategic points in the network,
  avoiding modification of end hosts or apps (aka
  knowledge plane)
• Apps web services, intrusion detection, storage
  access, etc.
• Improvements
• Dependability enhanced by monitoring network
  state, rapidly detecting behavioral changes
  (e.g., failures), configuring new resources in
  response
• Security enhanced thru more rapid discovery of
  and response to attacks

4
Block the in-coming attack Contain outgoing
attacks
Secure the edge network
Patch here once
Network is patched Block things rather
thanupgrade software! Not the end
systems Network is smart Not the end
systems Fragile to change, upgrade
Old System
Old System
Old System
Redundancy Isolation/Containment Heterogeneity
Old System
Too difficult to apply patches here
5
Make-A-Difference Technologies

• Statistical Learning Theory (Michael Jordan)
• Toolbox for the design and analysis of adaptive
  systems
• New and scaled-up algorithms for classification,
  diagnosis, prediction, novelty detection, outlier
  detection, quantile estimation, density
  estimation, feature selection, variable
  selection, response surface optimization,
  sequential decision-making
• kernel machine functional analysis plus convex
  optimization, yielding generalized inner product
  to measure similarities among data point pairs
• novelty detection/quantile estimation problems
  given cloud of data in feature space, place
  boundary so as to guarantee only small fraction
  falls outside (second-order cone programconvex
  optimization with efficient solution methods)
• Challenge make these algorithms work on-line
  while embedding them within network and
  distributed systems architectures

6
Make-A-DifferenceTechnologies

• New Approach to Run-Time Error Handling (George
  Necula)
• Program-level error-handling is difficult
• Repetitive, hard to maintain and specify
• Standard run-time approach is abort or ignore!
• Neculas approach
• Add explicit support at the programming language
  level
• Make compensations and interface obligations
• First-class citizens (like objects)
• With Static and Dynamic checking
• Handles tricky features (e.g., loops, the heap)
• Still provides strong guarantees
• Like sagas or compensating transactions
• Standard transactions not appropriate for this use

7
Implementation Platform based on Programmable
Networks
Tag Mem
Rules Programs
8
Short Statements

• Patterson/Fox Dependability of Distributed
  Systems
• Stoica Trust in Distributed Systems through
  Protocol Verification
• Tygar Security in Distributed Systems