Dimitrios Kotsalis a George Vellis a Demosthenes Akoumianakis a Jean Vanderdonckt b a Department of Informatics Engineering, Technological Education Institution of Crete, Greece b Université catholique de Louvain, Louvain School of Management, Belgium ‘Implementation-agnostic’ instantiation schemes for ubiquitous, synchronous multi-user interfaces

Research on advanced, creativity-based User Interface engineering, meaning – non-trivial case scenarios – multi-user interfaces – synchronous collaboration – distributed users – ubiquitous settings Research protocol – Joint supervision of two researchers of the istlab (Department of Informatics Engineering of TEI Crete) in PhD program at the Université catholique de Louvain, Louvain School of Management, Belgium Context of the research

Case study problem descriptions and overview of solutions (i.e., current version of the implementation) Brief and non-technical discussion of the engineering challenges and the technicalities of the work – Design-oriented inspiration – Development platforms and tools – On-going work Plan of the presentation

Two human collaborators / players – A single shared digital representation The soccer field – Two types of agents Human (collaborators) and non-human (soccer players) Human collaborators virtualized and represented by soccer players – Affordances of soccer players (inspired from semiotic engineering) Each human collaborator operates with the currently selected soccer player Each soccer player operates on a shared space inhabited by other players Each human collaborator operates through different devices (Desktop/Swing and Mobile/Android) The combined experience (i.e., game co-play) entails operations within different representations The system in its current operational settingsystem Scenario in MDSD’14 - The virtual setting

How can we design and generate non-trivial multi- user interfaces that support synchronous collaboration between distributed users in ubiquitous settings? Existing approaches – Toolkit-based programming – Model-based UI engineering – Both exhibit limitations which do not allow effective solution to the kind of problem described Research question & related works

Theory-based insights – Interaction devices versus affordances – Semiotic engineering for qualifying virtualities by type of agents and kind of operation Engineering approach – Development of an abstraction-based model anchoring interaction in to human intentions and capabilities – Specify capabilities in a model-based fashion – Extend UsiXML as needed to support the new specification Provisions for extensible interaction vocabularies – Build dedicated design tools run-time environment components Proposed approach

Current implementation of platform-agnostic UI instantiation schemesplatform-agnostic An illustration of the approach

A scheme that relies on implementation agnostic (i.e., abstract) specifications of UIs At run-time and once user and usage context parameters are discovered, the implementation agnostic spec is translated to context-specific interaction vocabularies using dedicated tools Platform-agnostic instantiation

Widget gallery

Polymorphic classification scheme for the ‘abstract button’ widget Download

Introduce new widgets as first-class design objects Widget Specification – Basic concept WSLLibraries Resources Widget Archive

Polymorphism & extensible interaction vocabularies Widget Archive

Note that polymorphism at the UI-level is a much more demanding notion than polymorphism (i.e., polymorphic method invocation) as implemented in popular Object-Oriented languages. An example of polymorphic specification of the ‘abstract SoccerField’ widget

Note that polymorphism at the UI-level is a much more demanding notion than polymorphism (i.e., polymorphic method invocation) as implemented in popular Object-Oriented languages. An example of polymorphic specification of the ‘abstract SoccerField’ widget

Run-time scenario (MDSD’14 paper)

Some technical challenges resolved Input/output techniques and event models Model View Window selectedtapped activated View Window Breakdowns  Different semantics  Implementation language/toolkit dependent

Some technical challenges resolved States and state transitions onMouseDown onMouseRelease onSelectCommand onChangePlayerCommand onTap onUntap moving stillmoving stillmoving still

Sharing schemes for Synchronous collaboration: “Common States” sharing  Not abstract enough since there may be no common states or states at all

More recent developments (THALES-MusiNet, ) Score’ interactive music metaphor: PC, expert users Tablature’ interactive music metaphor: PC, novice users The ‘circular’ interactive music metaphor: Web, novice users

Distributed Collaborative music learning User roles – Teacher (Expert) – Student (Novice) User stereotypes – Sighted – Blind Blind Platforms – PC: Java/swing – PC: JNVT2 – Web: HTML5 – Prototype Interactive metaphors – Score – Tablature – Circular Scenarios Multi-user Music Notation Lessons JNVT2

HTML5/Prototype Advanced collaborative features – Interim-feedback, Group awareness support ( Multi- user selection, Multi-user highlighting, Radar view, Film- view), Social awareness Novel features YouTube channel (“MusicNet istLab - Tei Crete”)MusicNet istLab - Tei Crete

Polymorphic specification

Current and on-going work – Support for web2 UIs – Generic support for group awareness – Run-time adaptivity and UI plasticity in distributed and ubiquitous settings – Distributed music learning Acknowledgements – The work is supported by ARCHIMEDES III, THALES – KUL for PhD dissertations (first two authors) Future work & acknowledgements

Final remark All screenshots presented were taken from running prototypes. For those interested some of them are already available in our channels on YouTube (“MusicNet istLab - Tei Crete”, iSTLab TeiCrete).MusicNet istLab - Tei CreteiSTLab TeiCrete