Resource Selector

1
Resource Selector

• Chuang Liu

2
What do we want to do?

• A smart Resource Selector

Resource requirement

App
R S
3
What do we want to do?
App
R S
These Resources seem fit your requirement best
4
Goal

• A Resource Selector for general purpose
• Matching between applications requirement and a
  set of resources
• Adaptability to dynamic status of distributed
  environment
• Support ownership of resource
• Support performance model of application
• The interface between application and RS should
  be simple

5
RS--Structure
App
RS
A subset of resource
GIIS
Resource Selector
MDS
App
MatchMaker
GRIS
Requirement
Resource
6
Challenges

• How to specify resource and request and how to
  match request with resource.
• Consistency of data in RS and system status.
• How to choose N resources from M available
  resources. (N ltM)

7
Challenges

• How to specify the request and resource and how
  to match the request with resource
• Consistency of data in RS and system status.
• How to choose N resources from M available
  resources. (N ltM)

8
ClassAdMechanism to specify resources and request

• The classad mechanism is a language for
  expressing and evaluating attribute
• A classad is a set of named expressions
• Each named expression is called an attribute
• Expression similar to those found in C/C

9
ClassAd example(resource)

• OpSysLINUX
• Name trapezius.cs.uchicago.edu
• LoadAvg 0.03
• Friends (foster, dave)
• Untrusted (evil, rival)
• Constraints !member(other.Owner, Untrusted)
  (LoadAvg lt 0.3)

10
ClassAd example(request)

• Ownerchliu
• Requirements other.LoadAvg lt 0.3
  other.opSysLINUX
• Rank 1/other.LoadAvg

11
Challenges

• How to specify the resource and request and how
  to match the request with resource.
• Consistency of data in RS and system status.
• How to choose N resources from M available
  resources. (N ltM)

12
Consistency

• Several threads in RS update information about
  system status based TTL value.
• Update information about available resource by
  access GIIS
• Update information about status of every
  resource by access GRIS or GIIS
• Tradeoff between Performance and Consistency

13
Challenges

• How to specify the resource and request and how
  to match the request with resource.
• Consistency of data in RS and system status.
• How to choose N resources from M available
  resources which fit applications requirement
  best. (N ltM)

14
Resource Selection

• How to select N resource from M available
  resources efficiently.
• How to judge which one is best among several
  matched results

15
Resource Selection

• How to select N resource from M available
  resources efficiently.
• How to judge which one is best among several
  matched results

16
Criteria to judge the desirability of resource

• Performance model
• F(resource Info, application info)
• minCPUSpeed gt 10 MIPS
• minMemSize gt 100 MB
• Rank minCPUSpeed NumOfResource
• Embed a program or function call in Classad
• ? Classad dont support function call in
  expression.

17
Resource Selection

• How to Select N resources from M available
  resources
• How to judge which one is best among several
  matched results

18
Bilateral match- Clique

• Organize all available resources into several
  cliques.

Match maker
Classads for clique
Classads for requirement
Classads for resource
19
Clique- How to organize clique

• Methods to organize clique
• Manually
• Automatically
• Pros
• Easy and useful
• Cons
• Not flexibility

20
Clique- Naive

• Naive method
• For example
• Resource a, b, c
• Cliques a, b, c, a, b, b,c, a,c, a,
  b, c
• Cons
• The number of clique is 2 to N

21
Gang Match

• Gang Matching 1? N

Match maker
Classads for requirement
Classads for resource
22
Gang match- Greedy Algorithm

• Greedy Algorithm
• Clique, candidate null
• Match the requirement with every resource
• Choose resource with highest ranking as the first
  number of clique
• For()
• If (clique match requirement) and
  (performance of application increase)
• Candidateclique
• Match the requirement with
  resource which is not in clique
• Add node with highest ranking in
  the left nodes to clique
• Else
• Return Candidate

23
Gang match- Greedy Algorithm

• Pros
• High performance
• Give pretty good optimal result to loosely
  coupled application
• Cons
• Locally optimal choice does not always lead to
  globally optimal solution

24
Gang match- Port and docking

• Ports and docking
• Ports
• Label Host1
• Requirements MemorySize gt 17.2M
• Constraint Host1.Arch "INTEL"
  Host1.OpSys "LINUX"
• Rank Host1.MIPS
• ,
• Label Host2
• Requirements MemoryReqs gt 18 M
• Constraint Host2.Arch "INTEL"
  Host2.OpSys "LINUX"
• Host1.Subnet
  Host2.Subnet
• Rank Host2.KFlops
• Rank 1 Host1.MIPS 8 Host2.Kflops

25
Gang match- Port and docking

• Pros
• Internal mechanism provided by Classad
• ? Available in Classad package
• Cons
• Match performance
• Application need to specify how many nodes it
  wanted

26
Gang match- Dynamic programming

• Limited resource use
• System administrator control how many resources
  user can use
• Users requirement
• performance gt threadhold value number of
  resource is as little as possible
• Applications requirement
• App specify how many CPU it wanted in request
• A 4 X 4 X 4 Grid calculation, numOfCPU lt 64

27
Gang match Dynamic programming

• Multi-dimensions knapsack problem
• Knapsack problem There are M items, every
  item has a Weight and a Value. Try to choose
  items from these items such that their value is
  maximum and their total weight is less than W.

28
Gang match- Dynamic programming

• Pros
• More flexible
• Always provide good resource to application
• polynomial time algorithm o(N)
• Cons
• Performance is bad than greedy algorithm

29
Project Status

• A RS is running, but not smart enough.
• Two match strateges have been implemented
• Clique
• Greedy algorithm

30
Open problem

• How well does RS work?
• Performance of application on selected resource
• Cost of RS
• Evaluate different strategy in match making

31
Welcome comment and suggestion

• Thank you!