Preference%20Elicitation%20in%20Combinatorial%20Auctions

1
Preference Elicitation in Combinatorial Auctions

• Tuomas Sandholm
• Carnegie Mellon University
• Computer Science Department
• (papers on this topic available via
  www.cs.cmu.edu/sandholm)

2
Outline

• Combinatorial auctions for multi-item auctions
• The revelation problem
• Previous approaches
• Our approach Elicitor agent
• Topological observations that motivate
  elicitation
• Different elicitation queries
• Policy dependent elicitor algorithms
• General policy independent elicitor framework
  (with data structures assimilation algorithms)
  specific elicitor algorithms
• Making the elicitor incentive compatible

3
Combinatorial auction

• Can bid on combinations of items Rassenti,Smith
  Bulfin 82...
• Bidders perspective
• Allows bidder to express what she really wants
• Avoids exposure problems
• No need for lookahead / counterspeculationing of
  items
• Auctioneers perspective
• Automated optimal bundling
• Binary winner determination problem
• Label bids as winning or losing so as to maximize
  sum of bid prices
• Each item can be allocated to at most one bid
• NP-complete Rothkopf et al 98 using Karp 72
• Inapproximable Sandholm IJCAI-99, AIJ-02 using
  Hastad 99

4
Another complex problem in combinatorial
auctions Revelation problem

• In direct-revelation mechanisms (e.g. VCG),
  bidders bid on all 2items combinations
• Need to compute the valuation for exponentially
  many combination
• Each valuation computation can be NP-complete
• For example if a carrier company bids on trucking
  tasks TRACONET Sandholm AAAI-93
• Need to communicate the bids
• Need to reveal the bids
• Loss of privacy strategic info

5
Revelation problem

• Agents need to decide what to bid on
• Waste effort on counter-speculation
• Waste effort making losing bids
• Fail to make bids that would have won
• Reduces economic efficiency revenue

6
What info is needed from an agent depends on what
others have revealed
Elicitor
Clearing algorithm
Elicitor decides what to ask next based on
answers it has received so far
Conen S. IJCAI-01 workshop on Econ. Agents,
Models Mechanisms, ACMEC-01
7
Elicitor Conen Sandholm 2001

• Have auctioneer incrementally elicit information
  from bidders
• based on the info received from bidders so far

8
Elicitation

• Goal minimize elicitation
• Regardless of computational / storage cost
• (Future work explore tradeoffs across these)
• Approach
• At each phase
• Elicitor decides what to ask (and from which
  bidder)
• Elicitor asks that and propagates the answer in
  its data structures
• Elicitor checks whether the auction can already
  be cleared optimally given the information in hand

9
Setting

• Combinatorial auction m items for sale
• Private values auction, no allocative
  externalities
• Each bidder i has value function, vi 2m ? R
• Unique valuations (to ease presentation)

10
Terminology

• (X1,...,Xbidders) is a collection
• Bundle Xi is earmarked for agent i
• An allocation is a feasible collection (i.e.,
  collection where Xis dont overlap in items)
• Objectives (1) Find Pareto efficient
  allocation(s) (2) Find social welfare
  maximizing allocation(s)

11
Outline

• Query policy dependent ( rank lattice based)
  elicitor algorithms
• Policy independent elicitor algorithms
• Note Private values model

12
Rank lattice
Bundle Ø A B AB Rank for Agent 1
4 2 3 1 Rank for Agent 2 4 3
2 1
1,1
1,2
2,1
3,1
2,2
1,3
2,3
3,2
1,4
4,1
2,4
3,3
4,2
3,4
4,3
4,4
Infeasible
Feasible
Dominated
13
A search algorithm for the rank lattice

• Algorithm PAR PAReto optimal
• OPEN ? (1,...,1)
• while OPEN ? do
• Remove(c,OPEN) SUC ? suc(c)
• if Feasible(c) then
• PAR ? PAR ? c Remove(SUC,OPEN)
• else foreach node ? SUC do
• if node ? OPEN and Undominated(node,PAR)
• then Append(node,OPEN)
• Thrm. Finds all feasible Pareto-undominated
  allocations (if bidders utility functions are
  injective)
• Welfare maximizing solution(s) can be selected as
  a post-processor by evaluating those allocations
  
• Call this hybrid algorithm MPAR (for maximizing
  PAR)

14
Value-augmented rank lattice
Bundle Ø A B AB Value for Agent 1 0
4 3 8 Value for Agent 2 0 1 6 9
17
1,1
14
13
1,2
2,1
10
12
9
3,1
2,2
1,3
8
9
2,3
3,2
1,4
4,1
2,4
3,3
4,2
3,4
4,3
4,4
15
Search algorithm family for the value-augmented
rank lattice

• Algorithm EBF Efficient Best First
• OPEN ? (1,...,1)
• loop
• if OPEN 1 then c ? combination in OPEN
• else
• M ? k ? OPEN v(k) maxnode ? OPEN v(node)
• if M ? 1 ? ?node ? M with Feasible(node) then
  return node
• else choose c ? M such that c is not dominated
  by any node ? M
• OPEN ? OPEN \ c
• if Feasible(c) then return c
• else foreach node ? suc(c) do
• if node ? OPEN then OPEN ? OPEN ? node
• From now on, assume quasilinear utility functions
  
• Thrm. Any EBF algorithm finds welfare maximizing
  allocations
• Thrm. VCG payments can be determined from the
  information already elicited

16
Rank lattice based partially-revealing VCG

• Use EBF to explore rank lattice top-down,
  best-first
• Elicitors queries (start at rank 1)
• tell me the bundle at the current rank (bundle
  query),
• tell me the value of the bundle at the current
  rank (value query), increment rank
• Natural sequence from good to bad bundles
• Elicit just the information necessary to find the
  best undominated feasible allocation
• Thrm. VCG payments can be determined from the
  information already obtained

17
Best worst case elicitation effort

• Best case rank vector (1,...,1) is feasible
• One bundle query to each agent, no value queries
• (VCG payments 0)
• Thrm. Any EBF algorithm requires at worst
  (2items bidders biddersitems)/2 1 value
  queries
• Proof idea. Upper part of the lattice is
  infeasible and not less in value than the
  solution
• Not surprising because worst-case communication
  complexity of the problem is exponential Nisan
  01

18
EBF minimizes feasibility checks

• Def An algorithm is admissible if it always
  finds a welfare maximizing allocation
• Def An algorithm is admissibly equipped if it
  only has
• value queries, and
• a feasibility function on rank vectors, and
• a successor function on rank vectors
• Thrm There is no admissible, admissibly equipped
  algorithm that requires fewer feasibility checks
  (for every problem instance) than any EBF
  algorithm

19
Partial-revelation mechanisms Theoretical results

• The EBF-based mechaAn extended EBF algorithm,
  RANK, determines an efficient allocation and VCG
  payments with no additional queries
• A RANK mechanism is incentive-compatible and
  economically efficient
• Thrm. Let B be the EBF that is used in a specific
  RANK mechanism. Then there does not exist any
  other mechanism based on an admissible,
  admissibly equipped, deterministic algorithm that
  requires fewer checks of the feasibility of nodes
  for all instances of the allocation problem

20
MPAR minimizes value queries

• Thrm. No admissible, admissibly equipped
  algorithm (that calls the valuation function for
  bundles in feasible rank vectors only) will
  require fewer value queries than MPAR

21
MPAR minimizes value queries

• Thrm. No admissible, admissibly equipped
  algorithm (that calls the valuation function for
  bundles in feasible rank vectors only) will
  require fewer value queries than MPAR
• MPAR requires at most biddersitems value queries

22
Differential-revelation

• Extension of EBF
• Information elicited differences between
  valuations
• Hides sensitive value information
• Motivation max ? vi(Xi) ? min ? vi(r-1(1))
  vi(Xi)
• Maximizing sum of value ? Minimizing difference
  between value of best ranked bundle and bundle in
  the allocation
• Thrm. Differences suffice for determining welfare
  maximizing allocations VCG payments
• 2 low-revelation incremental ex post incentive
  compatible mechanisms ...

23
Differential elicitation ...

• Questions (start at rank 1)
• tell me the bundle at the current rank
• tell me the difference in value of that bundle
  and the best bundle
• increment rank
• Natural sequence from good to bad bundles

24
Differential elicitation ...

• Variation Bitwise decrement mechanism
• Is the difference in value between the best
  bundle and the bundle at the current rank greater
  than d?
• if yes increment d, requires min. Increment
• allows establishing a bit stream (yes/no
  answers)

25
Differential-revelation Algorithm

• Like EBF algorithms, except in step 3,
  determination of the set of combinations that are
  considered for expansion
• M k?OPEN Tight(k) ? ?k ?d for all d with
  Tight(d) ? ?k lt ?d for all d with Not(Tight(d))

26
Differential-revelation Theoretical results

• Any algortihm of the modified EBF family finds a
  welfare-maximizing feasible allocation
• Given an arbitrary subset of rank lattice nodes,
  the set M is the same whether the original EBF or
  the differential-revelation EBF is used
• No additional revelation is needed to determine
  the VCG payments
• The differential-revelation mechanisms are
  incentive compatible and economically efficient

27
Policy independent elicitor algorithms
28
Some of our elicitors query types

• Order information Which bundle do you prefer, A
  or B?
• Value information What is your valuation for
  bundle A? (Answer Exact or Bounds)
• Rank information
• What is the rank of bundle b?
• What bundle is at rank x?
• Given bundle b, what is the next lower (higher)
  ranked bundle?

29
Interrogation An Example
Questions of the Auctioneer Answers of the
Agents

1. a1,a2 Give me your highest ranking bundle
2. a1,a2 Give me your next best bundle
3. a1 Give me your valuation for AB and Aa2 Give
   me your valuation for AB and B

• a1 AB, a2 AB(not feasible)
• a1 A, a2 B(feasible)
• a1 vAB8, vA4a2 vAB9, vB6

30
General Algorithmic Framework for Elicitation
Algorithm Solve(Y,G) while not Done(Y,G) do o
SelectOp(Y,G) ? Choose question I
PerformOp(o,N) ? Ask bidder G Propagate(I,G) ?
Update data structures with answer Y
Candidates(Y,G) ? Curtail set of candidate
allocations
Output Y set of optimal allocations Input Y
set of candidate allocations (some may turn
out infeasible, some suboptimal) G partially
augmented order graph
31
General Task of the Procedures

• Done checks if the topological structure has
  been sufficiently explored to exclude
  existence of better solutions
• In SelectOp, a Policy determines which questions
  to ask next
• PerformOp asks the questions and obtains
  answers
• Propagate will update the augmented order graph
• Candidates will determine a new set of potential
  solutions based on the update graph

32
(Partially) Augmented Order Graph
8
8
8
8
8
8
Ø
B
A
AB
Agent1
A
Ø
0
0
0
0
gt
Allocations
B
B
4
0
3
6
2
6
1
9
Ø
A
B
AB
Agent2
1
1
0
0
1
6
1,1
1,2
2,1
Rank
Upper Bound
3,1
2,2
1,3
1
9
2,3
3,2
1,4
1,4
AB
6
2,4
3,3
4,2
3,4
4,3
Lower Bound
4,4
Some interesting procedures for combining
different types of info
33
Storing the answer

• Interval constraint networks, 1 per agent
• Nodes store upper/lower bounds on value of
  bundle
• Edge (b,b) means vi(b) ? vi(b)
• At start create all nodes, add edges for free
  disposal

34
Constraint Network
111
1 per agent
110
101
011
100
010
001
000
35
Constraint Network
0,?
111
Upper bound
0,?
0,?
0,?
110
101
011
Lower bound
0,?
0,?
0,?
100
010
001
0
000
36
Constraint Propagation
0,?
111
vi(110)5
0,?
0,?
5
110
101
011
0,?
0,?
0,?
100
010
001
0
000
37
Constraint Propagation
0,?
111
vi(110)5
0,?
0,?
5
110
101
011
0,?
0,?
0,?
100
010
001
0
000
38
Constraint Propagation
5,?
111
vi(110)5
0,?
0,?
5
110
101
011
0,5
0,5
0,?
100
010
001
0
000
39
Constraint Propagation
5,?
111
vi(110)5
0,?
0,?
5
110
101
011
0,5
0,5
0,?
100
010
001
0
000
40
Constraint Propagation
5,?
111
vi(110)5
0,?
0,?
5
110
101
011
0,5
0,5
0,?
100
010
001
0 ,5
000
000
Additional edges from order queries
41
Constraint propagation

• Davis87 shows propagation is
• complete for assimilation (values for UB, LB are
  as tight as they can be made)
• incomplete for inference (cannot always use
  values to infer vi(b) ? vi(b))
• Need to use both values and network topology
  during inference

42
Are we done yet?

• Need to stop when enough information has been
  gathered
• Store list of possible allocations (candidates)
  C
• After each phase, eliminate allocations that
  cannot be optimal v(c) ? v(c)
• Stop when C 1

43
We present algorithms that use any combination of
value, order rank queries

• If value queries are used, all social welfare
  maximizing allocations are guaranteed to be found
• Otherwise, all Pareto efficient allocation are
  guaranteed to be found
• We propose several query policies that are geared
  toward reducing the number of queries needed

44
What to query should the elicitor ask (next) ?

• Simplest answer value query
• Ask for the value of a bundle vi(b)
• How to pick b, i?
• First try Randomly (subject to not asking
  queries whose answer can be inferred from info
  already elicited)

45
Random elicitation with value queries only

• Thrm. If the full-revelation (direct) mechanism
  makes Q value queries and the best
  value-elicitation policy makes q queries, we
  make value queries
• Proof idea We have q red balls, and the
  remaining balls are blue how many balls do we
  draw before removing all q red balls?
• Universal revelation reducer
• Is it tight? Run experiments

46
Experimental setup for all graphs in this talk

• Simulations
• Draw agents valuation functions from a random
  distribution where free disposal is honored
• Run the auction auctioneer asks queries of
  agents, agents look up answer from a file
• Each point on plots is average of 10 runs

47
Random elicitation

• Not much better than theoretical bound

queries
queries
2 agents
4 items
80
1000
60
Full revelation
100
Queries
40
10
20
1
9
2
2
3
4
5
6
3
4
5
6
7
8
10
agents
items
48
Querying random allocatable bundle-agent pairs
only

• Bundle-agent pair (b,i) is allocatable if some
  yet potentially optimal allocation allocates
  bundle b to agent i
• How to pick (b,i)?
• Pick a random allocatable one
• Asking only allocatable bundles means throwing
  out some queries
• Thrm. This restriction causes the policy to make
  at worst twice as many expected queries as the
  unrestricted random elicitor. (Tight)
• Proof idea These ignored queries are either
• Not useful to ask, or
• Useful, but we would have had low probability of
  asking it, so no big difference in expectation

49
Querying random allocatable bundle-agent pairs
only

• Much better
• Almost (items / 2) fewer queries than
  unrestricted random
• Vanishingly small fraction of all queries asked !
• Subexponential number of queries

queries
queries
80
1000
60
Full revelation
100
40
Queries
10
20
1
2
3
4
5
6
3
4
5
6
7
8
9
2
10
agents
items
50
Best value query elicitation policy so far
Focus on allocations that have highest upper
bound. Ask a (b,i) that is part of such an
allocation and among them, pick the one that
affects (via free disposal) the largest number
of bundles in such allocations.
Fraction of values queried before optimal
allocation found proven
Number of items for sale
51
Order queries

• Order query agent i, is bundle b worth more to
  you than bundle b ?
• Motivation Often easier to answer than value
  queries
• Order queries are insufficient for determining
  welfare maximizing allocations
• How to interleave order, value queries?
• How to choose i, b, b ?

52
Value and order queries

• Interleave
• 1 value query (of random allocatable agent-bundle
  pair)
• 1 order query (pick arbitrary allocatable i, b,
  b )
• To evaluate, in the graphs we have
• value query costs 1
• order query costs 0.1

53
Value and order queries

• Elicitation cost reduced compared to value
  queries only
• Cost reduction depends on relative costs of order
  value queries

54
Rank lattice based elicitation

• Go down the rank lattice in best-first order (
  EBF)
• Performance not as good as value-based why?
• nodes in rank lattice is 2bidders items
• feasible nodes is only biddersitems

queries
queries
80
1000
Full revelation
60
100
40
Queries
10
20
1
2
3
4
5
6
4
6
8
2
10
12
agents
items
55
Bound-approximation queries

• Often bidders can determine their valuations more
  precisely by allocating more time to deliberation
  S. AAAI-93, ICMAS-95, ICMAS-96, IJEC-00 Larson
  S. TARK-01, AGENTS-01 workshop, SITE-02 Parkes
  IJCAI workshop-99
• Get better bounds UBi(b) and LBi(b) with more
  time spent deliberating
• Idea dont ask for exact info if it is not
  necessary
• Query agent i, hint spend t time units
  tightening the upper (lower) bound on b
• How to choose i, b, t, UB or LB ?
• For simplicity, in the experiment graph, fix t
  0.2 time units (1 unit gives exact)

56
Bound-approx query policy
This slide is hidden later, it should replace
the next slide.

• For simplicity, fix t 0.2 units (1 unit gives
  exact)
• Can choose randomly.
• More complicated policy does slightly better
• Choose query that will change the bounds on
  allocatable bundles the most
• Dont know how much bounds will change
• Will try 3 policies
• Compute expectation (assume uniform distribution)
• Be optimistic assume most possible change
• Be pessimistic assume least possible change

57
Bound-approximation query policy

• Could choose the query randomly
• More sophisticated policy does slightly better
• Choose query that will change the bounds on
  allocatable bundles the most
• Dont know exactly how much bounds will change
• Assume all legal answers equiprobable, sample to
  get expectation

58
Bound-approximation queries

• This policy does quite well
• Future work try other related policies

queries
queries
160
1000
Full revelation
120
100
Query cost
80
10
40
1
9
2
2
3
4
5
6
3
4
5
6
7
8
10
agents
items
59
Bound-approximation a note

• To choose which query to ask, we calculated the
  expected change it makes
• But what is change from ? ?
• Policy actually is ask everyone for an UB on the
  grand bundle first
• After that, we neednt worry about ?
• Thrm. Upper bound on value of grand bundle is
  needed for all but one agent
• Thrm. With more than one bidder, eliciting the
  grand bundle from every agent cannot increase the
  length of the shortest elicitation certificate

60
Supplementing bound-approximation queries with
order queries

• Integrated as before
• Computationally more expensive

queries
queries
160
1000
Full revelation
120
Total cost
100
80
Order cost
10
Value cost
40
1
2
3
4
5
6
3
4
5
6
7
8
9
2
10
agents
items
61
A potentially better policy

• Assume auctioneer has an oracle that says which
  allocation is optimal. How to verify?
• To prove optimality, need to
• Prove sufficiently tight LB on optimal
• Prove sufficiently tight UB on all others
• Indicates a strategy when oracle is missing
• Usually ask queries that reduce UB
• But, need to sometimes raise LB

62
Incentive compatibility

• Elicitors questions leak information about
  others preferences
• Can be made ex post incentive compatible
• Ask enough questions to determine VCG prices
• Worst approach bidders1 elicitors
• Could interleave these extra questions with
  real questions
• To avoid lazyness Not necessary from an
  incentive perspective
• Agents dont have to answer the questions may
  answer questions that were not asked
• Unlike in price feedback (tatonnement)
  mechanisms Bikhchandani-Ostroy, Parkes-Ungar,
  Wurman-Wellman, Ausubel-Milgrom,
  Bikhchandani-deVries-Schummer-Vohra,
• Push-pull mechanism

63
Universal revelation reducers
64
Universal revelation reducer

• Def. A universal revelation reducer is an
  elicitor that will ask less than everything
  whenever the shortest certificate includes less
  than all queries
• Thrm Hudson Sandholm 03 No determionistic
  universal revelation reducer exists
• A randomized one exists
• (E.g., the one that asks random unknown value
  queries)

65
Elicitation where worst-case number of queries is
polynomial in items
66
Read-once valuations

• Thrm. If an agent has a read-once valuation
  function, the number of queries needed to elicit
  the function is polynomial in items
• Thrm. If an agents valuation function is
  approximable by a read-once function (with only
  MAX and PLUS nodes), elicitor finds an
  approximation in a polynomial number of queries

Zinkevich, Blum, Sandholm ACMEC-03
67
Toolbox valuations

• Items are viewed as tools
• Agent can accomplish multiple goals
• Each goal has a value requires some subset of
  tools
• Agents valuation for a package of items is the
  sum of the values of the goals that those tools
  allow the agent to accomplish
• E.g. items medical patents, goals medicines
• Thrm. If an agent has a toolbox valuation
  function, it can be elicited in O(items ?
  goals) queries

68
2-wise dependent valuations

• Thrm. If an agent has a 2-wise dependent
  valuation function, elicitor finds it in n(n1)/2
  queries
• Thrm. If an agents valuation function is
  approximately 2-wise dependent, elicitor finds an
  approximation in n(n1)/2 queries
• Thrm. Every super-additive valuation function is
  approximately 2-wise dependent
• Thrm. These results generalize to k-wise
  dependent valuations

69
Towards a broad polytime elicitor

• Thrm. If agents valuation function is in
• Read-once valuations (with SUM and MAX gates
  only)
• Toolbox valuations
• 2-wise dependent valuations
• then elicitor can learn the function using
• O(items2 items ? goals) queries

?
?
70
Combining polynomially elicitable classes

• Thrm. Conitzer, Sandholm, Santi 03 If Class C1
  is elicitable in polytime and class C2 is
  elicitable in polytime, then C1 U C2 is
  elicitable in polytime

71
Power of multi-agent elicitation

• Thrm. For some classes of valuation functions,
• eliciting the function requires an exponential
  number of queries,
• but a polynomial number of queries suffices for
  allocating the items optimally among the agents

72
Ascending combinatorial auctions
73
Demand queries

• If these were the prices, which bundle would you
  buy?
• A value query can be simulated by a polynomial
  number of demand queries
• A demand query cannot be simulated in a
  polynomial number of value queries Nisan

74
Ascending combinatorial auctions

• Increase prices until each item is demanded only
  once
• Item prices vs. bundle prices
• E.g. where there exist no appropriate item prices
• Discriminatory vs. nondiscriminatory prices

Bundle Bidder 1s valuation Bidder 2s valuation
1 0 2
2 0 2
1,2 3 2
75
Competitive equilibrium

• Def. Competitive equilibrium (CE)
• For each bidder, payoff max vi(S) pi(S), 0
• Sellers payoff maxS ? Feasibles ?i pi(S)
• Prices can be on bundles and discriminatory
• Thrm. Allocation S is supported in CE iff it is
  an efficient allocation
• Thrm Parkes 02 NisanSegal 03. In a
  combinatorial auction, the information implied by
  best-responses to some set of CE prices is
  necessary and sufficient as a certificate for the
  optimal allocation

76
Communication complexity of ascending auctions

• Exponential in items in the general case
• (like any other preference elicitation scheme)
• If items are substitutes (for each agent), then a
  Walrasian equilibrium exists,
• i.e., nondiscriminatory per-item prices suffice
  for agents to self-select the right items
• Number of queries needed to find such prices is
  polynomial in items Nisan Segal 03

77
Conclusions on preference elicitation in
combinatorial auctions

• Combinatorial auctions are desirable winner
  determination algorithms now scale to the large
• Another problem The Revelation Problem
• Valuation computation / revelation /
  communication
• Introduced an elicitor that focuses revelation
• Provably finds the welfare maximizing (or Pareto
  efficient) allocations
• Policy dependent search algorithms for
  elicitation
• Based on topological observations
• Optimally effective among admissibly equipped
  elicitors
• Eliciting value differences suffices
• Policy independent general elicitation framework
• Uses value, order rank queries (etc)
• Bound-approximation queries takes incremental
  revelation further
• Several algorithms, data structures query
  policies in the paper
• Only elicits a vanishingly small fraction of the
  valuations
• Presented a way to make the elicitor incentive
  compatible
• Yields a push-pull partial-revelation mechanism

78
Conclusions on ascending combinatorial auctions

• Demand queries (exponentially more powerful than
  value queries)
• Per-item prices vs. bundle prices
• Discriminatory vs. nondiscriminatory prices
• Exponential communication complexity, but
  polynomial in special classes (e.g., when items
  are substitutes)
• To allocate optimally, enough info has to be
  elicited to determine the minimal competitive
  equilibrium prices
• Could also use descending prices

79
Future research on multiagent preference
elicitation

• Scalable general elicitors (in queries, CPU, RAM)
• Current run-time exp in items, poly in agents
• Current space exp in items, linear in agents
• More powerful queries, e.g. side constraints
• New query policies
• New polynomially elicitable valuation classes
• Using models of how costly it is to answer
  different queries Hudson S. AMEC-02
• Decision-theoretic elicitation using priors
• Elicitors for markets beyond combinatorial
  auctions
• (Combinatorial) reverse auctions exchanges
• (Combinatorial) markets with side constraints
• (Combinatorial) markets with multiattribute
  features
• Other applications (e.g. voting Conitzer S.
  AAAI-02)