Commonsense Physical Reasoning: Boxes and Pitchers

1
Commonsense Physical Reasoning Boxes and Pitchers

• Ernest Davis
• New York University
• June 6, 2007

2
Commonsense inferences

• You can carry a collection of objects U at
  location A to location B by bringing an open box
  OB to A, loading the objects one by one into OB,
  and carrying OB to B.
• You can transfer liquid from container OS to
  container OD by pouring it i.e. lifting OS over
  OD and tilting it

3
Part of Common Sense Reasoning Enterprise

• Represent knowledge of commonsense domains and
  automate commonsense reasoning
• Potential applications to natural language
  interpretation, automated planning, vision,
  expert systems, automated tutoring, etc.

4
Commonsense and Science

• How experiment manipulation and perception at
  the human level
• relate to the underlying scientific theory.

5
Understanding variants

• What would happen if
• There were no test tubes?
• The test tubes were right side up?
• The test tubes were slanted?
• The test tubes were initially full of air?
• One of the electrodes was outside the tube?
• Both electrodes were in the same tube?

6
Variants

• Will fail if
• The cargo objects dont fit in the box
• The box is tilted too far
• Unless the box has a lid
• The box has a hole in the bottom
• Unless the hole is too small for the objects
• Some other object falls into the box
• Some agent lifts a cargo object out

7
More variants

• You can put one box inside another.
• You can make a pitcher overflow by dropping
  pebbles into it.
• You can ladle liquid out of a pail with a spoon.

8
Classic AI Representation

• (define (domain Box)
• (action load (?o ?b ?l)
• precondition (and (at ?o ?l) (at ?b ?l)
• 
  (box ?b))
• effect (and (in ?o ?b) (not (at ?o ?l))))
• (action move (?b ?l1 ?l2)
• precondition (at ?o ?l1)
• effect (and (at ?o ?l2) (not (at ?o ?l1)))))

9
Scientific Computing

• Given precise boundary conditions ---
• object shapes, material properties,
• initial state, motion of manipulated objects
• Output precise prediction.
• Rigid solid objects fine-grained time. Model
  all contact forces and impacts.
• Liquids fine-grained time and space. Model
  forces on every piece of liquid.

10
Commonsense Reasoning

• Aware of geometry
• Does not require exact spatial information
• Inconvenient /impossible to perceive or measure.
• Information from inference or natural language
• System under design / reasoning generically
• Supports alternative directions of inference
• future to past, behavior to shape, etc.
• Robust under variation of shape and motion
• Contrast Newtonian analysis.

11
Logic-Based Analysis (Hayes)

• Develop
• Ontology (microworld)
• First-order language
• Formal first-order theory (plus 2 default rules)
• Problem specification
• Model qualitative reasoning as logical inference

12
Constraints

• Qualitative reasoning Shapes, motions, number of
  cargo objects
• No derivatives
• No velocities or accelerations
• No forces or energy
• Reasoning over extended time and space
• Keep nonmonotonicity to a minimum
• Closed world assumptions in problem spec.

13
Outline

• Related work
• Pre-formal analysis
• Box microworld
• Liquids microworld
• Representation
• Future work and Conclusion

14
Related Work

• Pat Hayes Naïve Physics Manifesto, Ontology
  for Liquids
• AI Solid Objects Forbus, Faltings, Nielssen,
  Joskowicz, Shrobe Davis, Gelsey.
• AI Liquids Gardin Meltzer, Collins, Kim
• Sensorless Robotics Mason, Erdmann, LaValle
• Sci. Computing, Graphics, Robotics

15
Pre-Formal Analysis

• Modelling assumptions
• Objects are rigid. Liquids are incompressible.
• Motion is continuous.
• Large friction. Inelastic collisions. No
  viscosity, surface tension dry water.
• Objects in contact with liquids move slowly.
• Manipulation is motion of grasped object

16
Box/Cup

• Region R is boxed in state S iff
• the interior of R is connected
• no solid object is in the interior of R
• every boundary point of R below its top is a
  boundary point of some solid object

17
Boxed regions
18
Plan 1

• boxTransport(Uobjectset, OBbox,
• L1, L2 location)
• sequence(while (some object O is not in OB)
• load(O into OB)
• carry(OB from L1 to L2)
• )

19
Conditions

• In s1, uCargo and oBox are stably placed on
  oTable1
• uCargo and oBox are inside a region isolated from
  other objects.
• uCargo fits inside oBox.
• The path from oTable1 to oTable2 is isolated from
  other objects.
• oBox is kept fairly upright while being carried.
• No other agent manipulates any of the objects.

20
What can go wrong?

• Some object is screwed into the table.
• Loading one object knocks another off the table.
• The configuration in which the objects fit in the
  box is unattainable. (Note Cant assume that the
  objects stay where they were originally placed)
• The box falls over while being loaded.

21
What else can go wrong?

• Loading one object causes another to fly out of
  the box.

22
What else can go wrong?

• An object can come out of the box while it is
  being carried
• Box is tilted too far.
• Rolling object in bowl sloshes out.
• Large accelerations push object out of box.

23
Fixes

• Each cargo object is separately on the table,
  with a clear path to the box.
• Lots of excess room in box
• uCargo ?H/2D??W/2D??L/2D?
• Each object loaded to fairly low point.
• Vertical projection of inside of box inside
  convex hull of points of support on table.

24
Default rules

• By default, no object in a heap moves upward
  w.r.t. all the supports of the heap.
• If
• O is supported by OS.
• Objects in set U are in heaps supported by O.
• U remains over convex hull of points of support
  of points of O by OS
• then O does not fall over.

25
Liquid Microworld

• Kinematic constraints Liquids and solids do not
  interpenetrate, continuity, rigidity,
  incompressibility.

26
3 scenarios

• Cupped region If throughout a history a cupped
  region has greater volume than the liquid inside,
  then there is no outflow.
• If the volume of a cupped region falls below the
  liquid inside, then it overflows on top.
• Liquid flows down when it can, flowing around
  obstacles.

27
Overflow

• During overflow, the liquid rises a finite
  distance over the spout and goes out a distance
  beyond the spout. Falling down rule is
  suspended.

28
Ontology

• Real-valued time
• Euclidean space. Geometric entities as needed.
• Objects. Sets of objects.
• Chunks of liquids
• Agents.
• States
• Histories
• Fluents (Boolean and non-Boolean)
• Pseudo-objects (inside of a box center of mass)
• Plans

29
Open Box

• openBoxShape(RB,RIregion)
• openBoxShape(RB,RI) ?
• bounded(RB) bounded(RI) ec(RB,RI)
• connected(interior(RI))
• ?P P?boundary(RI) height(P) lt top(RI) ?
• P?boundary(RB)

30
Cupped region

• holds(S, cuppedRegion(R)) ?
• holds(S, openBoxShape(solidSpace,R)).

31
Liquid in cupped region

• dynamic(H) slowObjectsInContact(Q,H)
  continuous(Q,H)
• throughout(H,singleCup(Q)
• 
  fullOfLiquid(Q)) ?
• noOutflow(Q,H).
• (Note use of region-valued fluent Q great
• insight of Hayes.)

32
Default against flying out of box

• upwardMotion(O,US,H) ?
• ?OS?US,QC
• source(QC)OS coordSys(QC)
• sometime(H,zAxis(QC) z?)?
• value(end(H),zCoor(cMass(O),QC)) gt
• value(start(H),zCoor(cMass(O),QC)).

33
Default (cntd)

• anomUp(O,H) ?
• ?UH,US O?UH
• holds(start(H),heap(UH,US))
• throughout(H,isolated(UH,US,Ø))
• upwardMotion(O,US,H).
• noAnomUp(H) ?
• ?O,H1 subHistory(H1,H) ?anomUp(O,H1).
• dynamic(H) noAnomUp(H) / noAnomUp(H).

34
Feasibility of manipulation (simplified)

• kinematic(H) holds(start(H),grasps(A,O)) ?
• ?H1,H2 prefix(H1,H) duration(H1) ? 0
• sameMotionOn(H2,H1,O)
• start(H2)start(H)
• throughout(H2,grasping(A,O))
• dynamic(H2).

35
Box plan specification

• Plan1
• seq(while(unloaded ? Ø,
• seq(loadBox(a1,unloaded,
• 
  qinBox,manip1),
• waitUntil(stable(uCargo ?
• 
  oBox, oTab1)))
• carryBox(a1,oBox,qIn,uCargo,
• 
  oTab2,manip2))

36
Variants I can handle

• Boxes inside boxes
• Boxes composed of multiple parts
• Boxes with holes (milk crate)
• Boxes with lids
• Ladling liquid out of a bowl
• Raising water level by adding pebbles

37
Close variants I cant (yet) handle

• Unloading a box
• Carrying a heap of objects on a tray
• Pouring liquid onto a horizontal surface.

38
Future work

• More physical domains
• Probabilistic inference (e.g. stability of a
  heap)
• Tractable rule system
• Establish soundness w.r.t. a plausible model
• What qualitative inferences are important?

39

• David Waltz (1995)
• It was widely believed that logic could
  successfully model images and scenes, even though
  the baroque improbability of that effort should
  have long been clear to everyone who read Pat
  Hayes Naïve Physics Manifesto.

40
Conclusion

• Knowledge-based analysis using a logical language
  allows us to characterize a broader range of
  qualitative physical reasoning than any other
  known technique.