1. Functionality of objects through observation and Interaction Ruzena Bajcsy based on Luca Bogoni’s Ph.D thesis April 2016

2. Introduction Functionality pertains to the purpose of objects. In some instances, functionality involves constancy of physical relations between objects: support,containment, stability, etc. And in others it involves interaction: cutting, piercing, sawing, etc. In order to address, the role of an object, its representation must include the objects intrinsic properties (shape, material) but also the manner in which it is used. The definition of the representation of functionality implies that the agent must apply interactive and observational procedures.

3. Introduction,cont. The representation, hence will be subjective and will depend on agent’s perceptual and interactive capabilities. What we will demonstrate via experiments in which we will use different shape of tools and material properties as defined by perceived force. Then we will observe the interaction (their success and failures). Based on those we will determine the range of applicability of tools and objects. This fact actually supports the AI claim on the desirability to perform task in given context.

4. The functional recognition The formal mechanism towards this activity we use Discrete Event System theory together with active Perception paradigm. Word of caution: Functionality is NOT a characteristic unique to a single object. A particular object may have more than one functionality. For example a fork maybe used for cutting and /or piercing. Furthermore the functional attribution of an object is context and application dependent. The functionality presented here focusses on functionalities which involves change in physical relation between the tool and the object Through interaction.

5. Discrete Event System (DES) Discrete Event System Theory provides a formalism for describing, controlling and observing of a behavior of a dynamic system. Observers as defined by Ramadge and Wonham are mostly used to provide a feedback on the behavior of the plant. However they unlike us did not used Active Observer. We introduce an Overseer whose function is to coordinate individual active observers. An Overseer fulfills the role of an observer’s supervisor – controlling, enabling and disabling events to be observed or monitored by individual agents. When the observers are active it might be necessary to change their focus of attention.

6. The Hierarchical Structure The hierarchical structure allows focusing of the individual observers on different aspects of the interaction depending on the stage of the task execution. Thus the separate subtasks, or behaviors carried out by individual observers find the interpretation in the context of the overseer. Furthermore the overseer provides for some degree of sensory fusion at the event level. In addition if the planner were interact dynamically with an overseer the changes of the focus of events to be observed could be shifted according to current requirements rather than restricted by a predefined task model.

7. The Planner By placing the planner into the task control loop a top-down control can be defined and high level reasoning capabilities about the task can be introduced. The task description is assumed provided by the planner in a form of sequence of high level description of sequence of actions and consequences with observation requirements. The sequence then is mapped to the individual robots and observers performing and monitoring the interaction.

10. Representation of Functionality We identify the following types of functionality: Intended, the objects’ functionality is defined at the design time Imposed, defines the ability of using object for a function to initially intended. Intrinsic functionality denotes functional properties which either characterize intended functionality or define in virtue of physical properties of the object.

11. Functional Knowledge

12. Components of Functionality Functionality (Object) = Structure + Context + Application Where Structure is associated with intrinsic properties of an object Context identifies and Agent, possibly a Recipient of action, and the environment in which the functionality of object is expressed. Application denotes how the object is to fulfill the desired function.

13. Representation for Functionalities of an Object The intrinsic properties: geometric, material, kinematic and dynamic List of functionalities that have been discovered to be associated with particular class of objects. The list of applications describe how the tool si to be interacted with the object. The context component defines the association of the object with the instantiated functional applications.

14. Description of Application Component Tool Application; Task Interaction; Expectation of Interaction; A list of Intrinsic properties observed to be of relevance in the execution Some perceptual probing routines to allow the investigation of intrinsic properties.

15. Object Representation

16. Three examples

17. Block diagram for the three examples

18. Explanation of the figures The scissors are shown as having three functionalities according to their importance. In the context shown the agent is a hand and the observers are identified as vision, force and tactile sensors. The geometric description is shown here as an outline of the particular tool but model chosen would depend on a particular system. Some outlined properties might have been gathered with some exploratory procedure or discovered through interaction. In the tool application –field of the description of the functionalities, the type of contacts made by the tool, with the target object defines here the major difference.

19. Shape description via super-ellipses The equation is but we also added tapering and bending

20. Examples of tools

21. Extracted super-ellipses models

22. Examples of bending and their parameters

23. DES very informal

24. Constructing an Overseer

25. Overseer and different sensors

26. Adaptive Observer Piercing involves grasping an object (tool) with the intention bringing it to contact with target object. Once the tool has been brought to contact force must be applied to enable the tool to break the surface and penetrate a target object a given distance. By requiring that one observer monitors the force transition and the other the height of the tool as it penetrates, it is possible to determine whether the interaction is successful. However if the tool employed had a joint or were flexible then the decrease in height of tool may actually be due to bending of the tool rather then penetration.

27. If the observer is able to monitor the EVOLUTION the shape and motion of parameters of the tool we can correct for this misinterpretation. It is very clear that verification and adaptation are very important part of this process.