Draw Your Own Story : Paper and Pencil Interactive Storytelling Edirlei Soares de Lima, Bruno Feijó, Simone Barbosa, Antonio L. Furtado, Angelo Ciarlini, and Cesar PozzerEdirlei Soares de Lima, Bruno Feijó, Simone Barbosa, Antonio L. Furtado, Angelo Ciarlini, and Cesar Pozzer J. Anacleto et al. (Eds.): ICEC 2011, LNCS 6972, pp. 1–12, 2011J. Anacleto et al. (Eds.): ICEC 2011, LNCS 6972, pp. 1–12, 2011

Outline Introduction Related Work Paper and Pencil Interactive Storytelling Story Planner Hand-Drawn Sketches Recognition Augmented Reality Dramatization Application and Evaluation Technical Evaluation User Evaluation Conclusion 2

Introduction Hand drawings as a visual complement Told by several types of media How to combine Focus on immersive interfaces → Sketch-based interfaces Storytelling system = AR + sketch-based interaction interface recognizes the sketches converts the drawings into virtual objects 3

Related Work Storytelling environment PDA 、 scanner 、 bar codes 、 large screen display Mixed reality 、 AR 、 Gaming context Hand-drawn sketches → Anoto Live Digital Pen 4

Paper and Pencil Interactive Storytelling Stories are represented in AR over the paper The world may comprise several sheets of paper Users can switch Users as gods 5

Story Planner -1 Three basic approaches : plot-based character-based hybrid approach A nondeterministic hybrid approach allow characters to try achieving goals without necessarily succeeding be specified by nondeterministic automata The story : emotional and physical states Example story : swords and dragons genre 6

Story Planner -2 Six emotions proposed by Ekman and Friesen [ calmness, anger ] [ liking, disgust ] [ confidence, fear ] [ joy, sadness ] [ cheeriness, sorrow ] [ anticipation, surprise ] The sign ( − or + ) does not mean destructive or constructive emotions Emotions can be combined to form a new emotion love = joy + liking + confidence 7

Story Planner -3 Story planner : general rules and production rules Indirectly affect the decisions, even to subvert the storyline Examples of production rules used in the prototype: Story loop : 15 seconds 、 rules are executed again 8 general rules production rules example

Hand-Drawn Sketches Recognition-1 Pre-processing phase eliminate and highlight threshold levels is important to ensure that the drawings may be identified six new images uses the rectangular border Recognition phase classified according to a predefined set of drawings support vector machine (SVM) classifier seven moment invariants 9

Hand-Drawn Sketches Recognition-2 Recognition process as follows: Extraction of the contours from each candidate drawing found in pre-processed images Extraction of the features used by the SVM classifier from each candidate drawing Filtration of the candidate drawings that have too small areas or a number of vertices outside the range defined for the known drawings Classification of the candidate drawings using the SVM classifier Analysis of the SVM output to identify the drawing recognized by the classifier Classify the hand-drawn sketches found at all segmentation levels Choose the final classification by voting 10

Augmented Reality Dramatization ARToolKit Maker image User Interaction 11

Application and Evaluation Swords and Dragons genre The virtual world is populated by three main characters: princess Marian evil dragon Draco Knight sir Brian Virtual world is composed of four places: the princess's palace the dragon forest a church the forest where the wandering hero dwells 12 six items hand mirror Sword Shield magic stone Rat poison bottle

Technical Evaluation Sketch recognition Machine-learning method the recognition rate test the performance test 250 pictures Training datasets 100~300 samples 13

User Evaluation Preliminary evaluation four male 、 two female between 20 and 26 years old With the system (S) and with a modified version (M) S → M Questionnaire 21 questions 14

Conclusion Interactive storytelling system visualize stories in augmented reality interact with virtual characters by sketching objects Open up many possibilities to explore the user creativity Improve the system facilities designing an authoring tool to simplify and unify Larger number of participants 15

Thanks for listening 16