Commonsense Physical Reasoning: Boxes and Pitchers Ernest Davis New York University June 6, 2007

Commonsense inferences You can carry a collection of objects U at location A to location B by bringing an open box OB to A, loading the objects one by one into OB, and carrying OB to B. You can transfer liquid from container OS to container OD by pouring it: i.e. lifting OS over OD and tilting it

Part of Common Sense Reasoning Enterprise Represent knowledge of commonsense domains and automate commonsense reasoning Potential applications to natural language interpretation, automated planning, vision, expert systems, automated tutoring, etc.

Commonsense and Science How experiment – manipulation and perception at the human level – relate to the underlying scientific theory.

Understanding variants What would happen if: There were no test tubes? The test tubes were right side up? The test tubes were slanted? The test tubes were initially full of air? One of the electrodes was outside the tube? Both electrodes were in the same tube?

Variants Will fail if: The cargo objects don’t fit in the box The box is tilted too far –Unless the box has a lid The box has a hole in the bottom –Unless the hole is too small for the objects Some other object falls into the box Some agent lifts a cargo object out

More variants You can put one box inside another. You can make a pitcher overflow by dropping pebbles into it. You can ladle liquid out of a pail with a spoon.

Classic AI Representation (define (domain Box) (:action load (?o ?b ?l) :precondition (and (at ?o ?l) (at ?b ?l) (box ?b)) :effect (and (in ?o ?b) (not (at ?o ?l)))) (:action move (?b ?l1 ?l2) :precondition (at ?o ?l1) :effect (and (at ?o ?l2) (not (at ?o ?l1)))))

Scientific Computing Given precise boundary conditions --- object shapes, material properties, initial state, motion of manipulated objects Output precise prediction. Rigid solid objects: fine-grained time. Model all contact forces and impacts. Liquids: fine-grained time and space. Model forces on every piece of liquid.

Commonsense Reasoning Aware of geometry Does not require exact spatial information –Inconvenient /impossible to perceive or measure. –Information from inference or natural language –System under design / reasoning generically Supports alternative directions of inference –future to past, behavior to shape, etc. Robust under variation of shape and motion –Contrast Newtonian analysis.

Logic-Based Analysis (Hayes) Develop Ontology (microworld) First-order language Formal first-order theory (plus 2 default rules) Problem specification Model qualitative reasoning as logical inference

Constraints Qualitative reasoning: Shapes, motions, number of cargo objects No derivatives No velocities or accelerations No forces or energy Reasoning over extended time and space Keep nonmonotonicity to a minimum –Closed world assumptions in problem spec.

Outline Related work Pre-formal analysis: –Box microworld –Liquids microworld Representation Future work and Conclusion

Related Work Pat Hayes: “Naïve Physics Manifesto”, “Ontology for Liquids” AI Solid Objects: Forbus, Faltings, Nielssen, Joskowicz, Shrobe & Davis, Gelsey. AI Liquids: Gardin & Meltzer, Collins, Kim Sensorless Robotics: Mason, Erdmann, LaValle Sci. Computing, Graphics, Robotics

Pre-Formal Analysis Modelling assumptions: Objects are rigid. Liquids are incompressible. Motion is continuous. Large friction. Inelastic collisions. No viscosity, surface tension – “dry water”. Objects in contact with liquids move slowly. Manipulation is motion of “grasped” object

Box/Cup Region R is boxed in state S iff the interior of R is connected; no solid object is in the interior of R; every boundary point of R below its top is a boundary point of some solid object

Boxed regions

Plan 1 boxTransport(U:object[set], OB:box, L1, L2: location) sequence(while (some object O is not in OB) load(O into OB); carry(OB from L1 to L2) )

Conditions In s1, uCargo and oBox are stably placed on oTable1 uCargo and oBox are inside a region isolated from other objects. uCargo fits inside oBox. The path from oTable1 to oTable2 is isolated from other objects. oBox is kept fairly upright while being carried. No other agent manipulates any of the objects.

What can go wrong? Some object is screwed into the table. Loading one object knocks another off the table. The configuration in which the objects fit in the box is unattainable. (Note: Can’t assume that the objects stay where they were originally placed) The box falls over while being loaded.

What else can go wrong? Loading one object causes another to fly out of the box.

What else can go wrong? An object can come out of the box while it is being carried: –Box is tilted too far. –Rolling object in bowl sloshes out. –Large accelerations push object out of box.

Fixes Each cargo object is separately on the table, with a clear path to the box. Lots of excess room in box: |uCargo| ≤ ⌊H/2D⌋⌊W/2D⌋⌊L/2D⌋ Each object loaded to fairly low point. Vertical projection of inside of box inside convex hull of points of support on table.

Default rules By default, no object in a heap moves upward w.r.t. all the supports of the heap. If: –O is supported by OS. –Objects in set U are in heaps supported by O. –U remains over convex hull of points of support of points of O by OS then O does not fall over.

Liquid Microworld Kinematic constraints: Liquids and solids do not interpenetrate, continuity, rigidity, incompressibility.

3 scenarios Cupped region: If throughout a history a cupped region has greater volume than the liquid inside, then there is no outflow. If the volume of a cupped region falls below the liquid inside, then it overflows on top. Liquid flows down when it can, flowing around obstacles.

Overflow During overflow, the liquid rises a finite distance over the spout and goes out a distance beyond the spout. “Falling down” rule is suspended.

Ontology Real-valued time Euclidean space. Geometric entities as needed. Objects. Sets of objects. Chunks of liquids Agents. States Histories Fluents (Boolean and non-Boolean) Pseudo-objects (inside of a box; center of mass) Plans

Open Box openBoxShape(RB,RI:region) openBoxShape(RB,RI)  bounded(RB) ^ bounded(RI) ^ ec(RB,RI) ^ connected(interior(RI)) ^ [ ∀ P P∈boundary(RI) ^ height(P) < top(RI) ⇒ P∈boundary(RB)]

Cupped region holds(S, cuppedRegion(R))  holds(S, openBoxShape # (solidSpace,R)).

Liquid in cupped region dynamic(H) ^ slowObjectsInContact(Q,H) ^ continuous(Q,H) ^ throughout(H,singleCup # (Q) ^ # ¬ # fullOfLiquid # (Q)) ⇒ noOutflow(Q,H). (Note use of region-valued fluent Q: great insight of Hayes.)

Default against flying out of box upwardMotion(O,US,H)  ∀ OS∈US,QC [source(QC)=OS ^ coordSys(QC)^ sometime(H,zAxis * (QC) = # z̑)]⇒ value(end(H),zCoor * (cMass(O),QC)) > value(start(H),zCoor * (cMass(O),QC)).

Default (cntd) anomUp(O,H)  ∃ UH,US O∈UH ^ holds(start(H),heap(UH,US)) ^ throughout(H,isolated(UH,US,∅)) ^ upwardMotion(O,US,H). noAnomUp(H)  ∀ O,H1 subHistory(H1,H) ⇒¬anomUp(O,H1). dynamic(H) : noAnomUp(H) / noAnomUp(H).

Feasibility of manipulation (simplified) kinematic(H) ^ holds(start(H),grasps(A,O)) ⇒ ∃ H1,H2 prefix(H1,H) ^ duration(H1) ǂ 0 ^ sameMotionOn(H2,H1,{O}) ^ start(H2)=start(H) ^ throughout(H2,grasping(A,O)) ^ dynamic(H2).

Box plan specification Plan1 = seq(while(unloaded ǂ ∅, seq(loadBox(a1,unloaded, qinBox,manip1), waitUntil(stable(uCargo ⋃ {oBox, oTab1}))) carryBox(a1,oBox,qIn,uCargo, oTab2,manip2))

Variants I can handle Boxes inside boxes Boxes composed of multiple parts Boxes with holes (milk crate) Boxes with lids Ladling liquid out of a bowl Raising water level by adding pebbles

Close variants I can’t (yet) handle Unloading a box Carrying a heap of objects on a tray Pouring liquid onto a horizontal surface.

Future work More physical domains Probabilistic inference (e.g. stability of a heap) Tractable rule system Establish soundness w.r.t. a plausible model What qualitative inferences are important?

David Waltz (1995): “It was widely believed that logic could successfully model images and scenes, even though the baroque improbability of that effort should have long been clear to everyone who read Pat Hayes’ Naïve Physics Manifesto.”

Conclusion Knowledge-based analysis using a logical language allows us to characterize a broader range of qualitative physical reasoning than any other known technique.