1. http://scom.hud.ac.uk/planet/competition/ ICKEP International Competition for Knowledge Engineering in Planning Lee McCluskey PLANET Knowledge Engineering TCU

2. Contents Aims / Benefits Current IPC Problems with an ICKEP A small start

3. Current ICP - Benefits The ICP has brought benefits to the community - - focussed some researchers on technology innovation - led to a rapid development of techniques - delivered a de facto standard for communicating the dynamics of domain models - helped in the validation of planning algorithms and hence led to the sharing of benchmark domain models, tasks and planning tools.

4. Current ICP - Problems However, the ICP is controversial - it encourages rapid development - but in a narrow area The ICP has assumed that: the input to a planning engine is correct and complete - the language and encoding of a domain into the language are given. the input is in PDDL which was designed to reflect current languages and their underlying assumptions, and with the criterion of “dynamics and nothing else”. It was NOT designed with a model building method in mind OR with many ‘pragmatic’ feature which make building easier - it is more of a machine code than a language for human use!

5. Narrow views of Planning? Complete, correct, formal, Precondition-effect, Literal strips-based Model of dynamics Something Else? Plan Generator Execution, Scheduling Acquisition, Debugging, Compiling, Configuration, Modelling

6. Aim of ICKEP The aim of a KE Competition will be to promote the knowledge-based aspects of planning (to include knowledge acquisition, knowledge modelling and domain validation) by evaluating KE tools within a competitive forum. ” knowledge engineering processes support the planning process – they comprise all of the off-line, knowledge-based aspects of planning that are to do with the application being built.”

7. Possible Benefits it might address the main problem with the current competition - that, although the competition encourages rapid development, it tends to focus work narrowly. it might encourage the development and sharing of stand alone tools to help in the whole process of AI planning including domain modelling, heuristic acquisition, planner-domain matching and so forth. it might lead to some form of communication medium for knowledged-based domain models

8. Form of Current IPC  Competitors prepare before the event: a planner which can input PDDL and gives out solutions in a prescribed format.  Competitors are given at the event: domain models, tasks, in PDDL  During the event: the planners are executed with the supplied domain models and tasks.  Evaluation after the event: tools are used to rate the planners on speed, coverage, and solution quality.

9. Form of ICKEP?? But tools and methods to support knowledge acquisition and modelling … do not have standard forms of input. They may acquire knowledge from domain experts or help planning researchers debug domain models. Cannot be easily evaluated by their outputs - what is the advantage of one domain model over another? Are heterogneous - there are several types of tools performing differing functions

10. PROPOSAL: Start simple Start off with initial competition which has a very simple format, and build from there

11. Example 1 Competitors prepare before the event: two types of tool (a) one that debugs domain models (b) one that extracts heuristics from domain models. Both tools will input a certain version of PDDL and (a) will output a set of flaws in the domain model, and (b) will output a set of heuristics in a standard format, that can be used with a standard planner, to help solve plan generation problems.

12. Example 1 Competitors are given at the event: flawed domain models for (a), domain models, a planner and tasks for (b). During the event: the tools are executed with the supplied domain models and tasks. Evaluation after the event: tools are used to rate the competitors' tools for (a) percentage and type of flaws uncovered (b) quality of heuristics acquired as judged by performance improvement on a standard planner.

13. Example 2 Competitors prepare before the event: software in the form of a standalone tool or tools environment which helps in the process of knowledge engineering for planning. This could be for visualisation, knowledge acquisition, knowledge modelling, domain analysis etc. Planners/Schedulers themselves will NOT be eligible (though they may be part of an environment demonstrated, or they may be used to show the potential of a tool).

14. Example 2 Evaluation: An evaluation of each submitted software will be made by a Panel, via a demonstration. The Panel will judge the software on criteria with respect to AI Planning/Scheduling, such as the following (by no means exhaustive!): - support potential: what potential has the tool(s) in helping the process of domain acquisition, modelling, visualisation? is the scope of the tool(s) narrow or broad? will the tool make planning software more accessible or usable?

15. Example 2 Evaluation: - innovation: what is the quality of the scientific and technical innovations that underlie the software? How does it compare with KE software in other areas of AI? - build quality and interoperability: does the software appear robust? can the software be easily used with other planning software, or easily combined with third party planners? Are its interfaces well defined? - relevance: to what degree do the tool(s) address problems peculiar to KE for Planning/Scheduling?

16. Initial Working Group Prof Ruth Aylet, University of Salford, UK Dr Ronan Bartak, Charles University, Prague, Czech Republic Prof Daniel Borrajo, University Carlos III de Madrid, Spain Prof Susanne Biundo, University of Ulm, Germany Dr Christophe Doniat, Université Technologique de Troyes, France Dr Peter Jarvis, SRI International, USA Prof Lee McCluskey, University of Huddersfield, UK