1. Automated Assessment of Kinesthetic Performance Simon Fothergill Ph.D. student Digital Technology Group, Computer Laboratory, University of Cambridge SeSAME Plenary Meeting, 11 th February 2010

2. Automated Assessment of Kinesthetic Performance Sense and Optimise. Feedback is fundamental pedagogical mechanism. Automate to supplement. What are they doing? What should they be doing? How? Rowing simulators.

3. Important areas of work Capturing Kinetics Performance similarity Natural expression Useful feedback

4. Capturing Kinetics : Requirements Dataset Real and uncontrived Large Representative of the performance High fidelity Synchronised Segmented Data capture system Compatible Equipment augmentation Annotation Security Portable Cheap Physically robust Extensible platform

5. Capturing Kinetics : Overview

6. Capturing Kinetics : Hardware (1) Sensors 3D position of handle 3D position of seat force applied though handle force applied though toes of each foot

7. Capturing Kinetics : Hardware (2)

9. Capturing Kinetics : Software

10. Capturing Kinetics : Operation (1) ECS (Erg Coordinate System) EMCS needs to track handle (1), seat (1) and erg position + orientation (4) WMCS currently limited to 4 LEDs Use 1 LED as a stationary point on the erg & 2 LEDs on the seat at different points in time Use PCA to extract ECS axes Erg clamped to camera rig to minimise error Two LEDS attached to seat

11. Capturing Kinetics : Operation (2) Data from one Wii controller IR camera, used in computing correspondance of LEDs between cameras End LED Handle LED Seat LEDs Server Triangulation Stereo calibration Client 4 x 2D coordinates 4 x 3D coordinates Erg calibration Label markers Transform to ECS Update ECS if necessary ECS Calibrate labeller Calibrate WMCS (openCV) Storage Calibration Live operation

12. Capturing Kinetics : Operation (3) Server (boathouse)Client Detect strokes Log data :Motion + force data, images Split data into strokes Update database Turn on/off camera Live operation Post session File server (CL) Create directories Transmit data Handle + seat coordinates, handle force, stroke boundaries Create user videos Augment and select Encode videos Record user code Data, videos, video metadata Display on GUI Create metadata

13. Capturing Kinetics : Deployment & Evaluation Technical At limit of WMCS range (accuracy and precision) WMCS won’t work in bright sunlight Hand covering LED on handle Correspondence: Unnecessary vigorous rowing upsets algorithms which could be improved (domain specific e.g. scan; generic e.g. epipolar constraints) ECS updated infrequently More force sensors on heal of feet openCV is buggy General Developed a novel and functional system and gained experience of deploying it and what is possible to achieve. It enables further useful and convincing work to be done Useful dataset, sets a benchmark Users Some people are very frightened about using it, especially as video is taken The system has a steep but short learning curve Athletes require a very simple interface. They won’t even see half the screen and definitely not read anything.

14. Demonstration using website http://www-dyn.cl.cam.ac.uk/~jsf29/

15. Supplementary Assistance for Rowing Coaching – SpARC (1) Application for real-time visual feedback

16. SpARC : Evaluation Method Method Coach athlete Record target performance Row with different feedback: none, real-time kinetics, target performance under various conditions: After 30 minutes After 5 weeks Race pace Fatigued Participants 5 rowers 2 professional GB rowing coaches Performance metrics Energy supplied to ergometer Approximate efficiency Approximate similarity to target Approximate consistency

17. SpARC : Evaluation Results The mean and standard deviation for the metrics over all the strokes of a session are given. Values are rounded to 3 significant figures. Some data was lost due to a sensor system fault. Example of how the force performance metrics changes though a session from Expt. 1 for rower 3.

18. SpARC : Evaluation Results : Does feedback help? Statistical significance Little/detrimental effect on performance immediately after rowing, (1 case where feedback helps) Quite strong correlation after prolonged solo training and during race-pace Significant correlation during fatigued rowing

19. SpARC : Conclusions and Limitations Functional application providing real-time feedback on kinetics of a rowers performance when using an ergometer System is of some use in helping rowers to maintain a consistently good technique as described by a coach, especially when the athletes are extended absence of their own coach or become fatigued. Evaluation dataset is currently small. Order of experiments is not varied. Performance metrics are only justifiable approximations, although could be included in a biomechanical model of a rowing boat.

20. Acknowledgements Andy Hopper, Rob Harle, George Colouris, Brian Jones, Sean Holden, Marcelo Pias, Salman Taherian, Andy Rice, Joe Newman, DTG, Rainbow group, Andrew Lewis, SeSAME, Computer Laboratory, Jesus College, Jesus College Boatclub, Jesus College BoatClub Trust, Cantabs boatclub Cambridge, Peter Lee & James Harris GB rowing

21. Further information HCI09 demonstration MUM09 demonstration Videolectures.net (MUM09) ISEA10 paper (submiting) S+SSPR08 paper Sourceforge StrideSense Cambridge University i-Teams

22. Questions Thank you for your attention. Comments and Questions, please!