1. Shoe-Mouse: An Integrated Intelligent Shoe ∗ Weizhong Ye, Yangsheng Xu and Ka Keung Lee Department of Automation and Computer-Aided Engineering The Chinese University of Hong Kong EB2, Shatin,CUHK,HongKong wzye@acae.cuhk.edu.hk Student ID ： M9920103 Student ： Kun-Hong Lee Adviser ： Ming-Yuan Shieh 1

2. Outline ABSTRACT INTRODUCTION HARDWARE DESIGN DESIGN OF SHOE-MOUSE BASED ON THE PLATFORM OF INTELLIGENT SHOES EXPERIMENTAL RESULTS CONCLUSION AND FUTURE WORK REFERENCES 2

3. ABSTRACT In this paper, we developed a sensor-integrated shoe as an information acquisition platform to sense the foot motion. The system is small, portable and wearable. The platform is mainly composed of four parts including a sensing module, a computing module, a wireless communication module, and a data visualization module. 3

4. ABSTRACT Based on this platform, we developed a novel input device called Shoe- Mouse, which can be used by people who have difficulties in using their hands to operate computers or devices. The platform can be also used for applications such as gait recognition, human identification, and motion monitoring. 4

5. 5 INTRODUCTION Main attributes of our system Firstly, we integrated all the sensors and circuits fully inside the shoe without adding much weight into the original shoe. It is easy to use and users will notice little difference between their normal shoes and the proposed shoes. Secondly, we built a hardware platform to collect data from the shoe. This platform is programmable, easily scalable and easy to be integrated to the other applications.

6. 6 INTRODUCTION Thirdly, we applied the system to several successful tasks based on this platform, especially the Shoe-Mouse. By using this interface, we can operate a device with our feet. This invention can be useful for the persons who have to use computers frequently but have difficulties in using their hands.

7. 7 HARDWARE DESIGN Fig. 2. Outline of the hardware design and sensor

8. 8 HARDWARE DESIGN Sensing the parameters inside the shoe This component consists of a flexible instrumented insole (in Fig. 3) that is worn inside the shoe. One important parameter the insole measures is the force applied between the foot and the shoe at some key points. These key points are under the major weight-bearing points of the foot: the big toe and the heel ( which is divided into a medial and lateral portion ).

9. 9 HARDWARE DESIGN Sensing the parameters inside the shoe Fig. 3. A flexible instrumented insole

10. 10 HARDWARE DESIGN Sensing the parameters inside the shoe The force sensors operate with a voltage source and a fixed resistor to produce a voltage that changes with the applied forces. This voltage is further processed in order to present a scaled voltage to the input of an analogto-digital converter. The digitized value of the force will be sent to the micro- controller.

11. 11 HARDWARE DESIGN Sensing the parameters inside the shoe Besides, a bend sensor is installed between the toe and the heel. This bend sensor is used to measure the degree of bend between the toe and the heel. The output of the bend the sensor also contains rich information about human motion, especially loading and uploading of feet.

12. 12 HARDWARE DESIGN Gathering information from the sensors This subsystem is mainly composed of a processor circuit board (in Fig. 4). It includes a microcomputer, an analog- to-digital converter, peripheral components, batteries, and an attached accelerometer. As can be seen from Fig. 4, the circuit board is small and it can be easily put into the heel of the shoe so that users will notice little the difference between the normal shoes and the intelligent shoes.

13. 13 HARDWARE DESIGN Gathering information from the sensors Fig. 4. Processor circuit board

14. 14 HARDWARE DESIGN Data visualization As this platform is designed for general applications, we display all parameters measured from the shoe. The host computer gets the data from the wireless receiver via RS232. Different functions for visualizing the data from different sensors are developed and compacted. We can also visualize a walking person by animation (in Fig. 5), which is mapped from different motion status of the person.

15. 15 HARDWARE DESIGN Data visualization Fig. 5. A walking person by animation

16. 16 DESIGN OF SHOE-MOUSE BASED ON THE PLATFORM OF INTELLIGENT SHOES Analysis of signals from sensors inside the shoe 6 parameters are selected and measured: 1) the force sensor installed at the toe 2) the force sensor installed at the left side of the heel 3) the force sensor installed at the right side of the heel 4) X-axis of accelerometer at the heel 5) Y-axis of accelerometer at the heel 6) degree of bend from the bend sensor

17. 17 DESIGN OF SHOE-MOUSE BASED ON THE PLATFORM OF INTELLIGENT SHOES Analysis of signals from sensors inside the shoe Fig. 6 shows the sample data taken from the intelligent shoe when a person wearing this shoe is walking at a speed around 5km/h. Fig. 6. Visulization of data

18. 18 DESIGN OF SHOE-MOUSE BASED ON THE PLATFORM OF INTELLIGENT SHOES Smoothing the curve of the data In general, errors will be unavoidably introduced into the system. There are two kinds of errors. One is the error introduced from the wireless module.The other is introduced from the abnormal contact between the shoe and the human foot, which is the output of the sensors that do not reflect the person’s intention and it should be deleted.

19. 19 DESIGN OF SHOE-MOUSE BASED ON THE PLATFORM OF INTELLIGENT SHOES Smoothing the curve of the data These two errors will cause some abnormal peaks in the output waveform. Here, exponential smoothing [7] is applied to minimize the effect of the abnormal peaks without affecting the performance.

20. 20 DESIGN OF SHOE-MOUSE BASED ON THE PLATFORM OF INTELLIGENT SHOES Smoothing the curve of the data The principle of exponential smoothing can be described as follows: St = α ∗ yt + (1 − α) ∗ (St−1 + bt−1) 0≤ α ≤ 1 (1) bt = γ ∗ (St − St−1) + (1 − γ) ∗ bt−1 0 ≤ γ ≤ 1 (2) Fig. 7. Result of exponential smoothing

21. 21 DESIGN OF SHOE-MOUSE BASED ON THE PLATFORM OF INTELLIGENT SHOES Mapping motion of foot to motion of a mouse-cursor on the screen Based on the shoe-mounted data gathering platform, we use the following mapping methods to achieve the goal: 1) Use accelerometer to produce motion of mouse cursor: The accelerometer used here (ADXL202E) can output xaxis acceleration and y-axis acceleration.

22. 22 DESIGN OF SHOE-MOUSE BASED ON THE PLATFORM OF INTELLIGENT SHOES Mapping motion of foot to motion of a mouse-cursor on the screen Fig. 8. Rectangle motion of the shoe in a plane (left) and its corresponding output of x-axis acceleration (right)

23. 23 DESIGN OF SHOE-MOUSE BASED ON THE PLATFORM OF INTELLIGENT SHOES Mapping motion of foot to motion of a mouse-cursor on the screen Fig. 9. Motion of foot and its corresponding acc output

24. 24 DESIGN OF SHOE-MOUSE BASED ON THE PLATFORM OF INTELLIGENT SHOES Mapping motion of foot to motion of a mouse-cursor on the screen 2) Use force sensors at the toe to produce single click of the left button: Force sensor at the toe is naturally mapped into a single click. When a user has clicked the shoe using his toe, a force will be produced from the force sensor installed between the toe and the shoe.

25. 25 DESIGN OF SHOE-MOUSE BASED ON THE PLATFORM OF INTELLIGENT SHOES Mapping motion of foot to motion of a mouse-cursor on the screen single-click of left button of the mouse takes effect. This force will persist for about 0.5 seconds. The average force during this time is computed. If the force is bigger than the threshold, the function of single-click of left button of the mouse takes effect

26. 26 DESIGN OF SHOE-MOUSE BASED ON THE PLATFORM OF INTELLIGENT SHOES Mapping motion of foot to motion of a mouse-cursor on the screen 3) Use two force sensors at the heel to produce single click of the right button: here we use two force sensors to judge whether the user has pressed the right button

27. 27 EXPERIMENTAL RESULTS The toe of the shoe contacts the ground with a force. By calculating the average value of this force, we can decide whether the user is intended to produce a left click of the mouse or not. However, not all toe motions can produce the corresponding click of the mouse. We did the following experiments to evaluate the performance of the click motion.

28. 28 EXPERIMENTAL RESULTS A circle with the diameter 30cm was drawn manually on the screen. Each of the three volunteers wearing intelligent shoes performs a simple click of toe to produce a point in this circle. If the click is successful, it will produce a new dark point inside the circle.

29. 29 EXPERIMENTAL RESULTS Fig. 11. Result of click test

30. 30 EXPERIMENTAL RESULTS We drew a circle as a test path, letting the users draw several identical circles in the same place as accurately as they could. Then we computed the deviation between drawn circles and the original circle.

31. 31 EXPERIMENTAL RESULTS If the deviation is bigger than the threshold, we consider this circle an unsuccessful circle. After that, we compared the performance between the ordinary mouse and our proposed Shoe- Mouse.

32. 32 EXPERIMENTAL RESULTS Fig. 12. Performance of drawing a circle using Shoe-Mouse

33. 33 CONCLUSIONS In this paper, we have built a shoe-based sensor integrating platform that can gather a lot of data inside the shoe. A novel input device called Shoe-Mouse is introduced by applying the shoe platform as a user wearable interface. The Shoe-Mouse is especially designed for the people who have difficulties in operating computers using hands. As proved in the experiments, the Shoe-Mouse performs satisfactorily in motion control and can partly replace the ordinary mouse.

34. 34 CONCLUSIONS People who do a lot of work using computers can also benefit from this invention. In the future, more experiments will be conducted on potential user groups to find out more about the requirements for the systems

35. 35 REFERENCES [1] K. N. Tarchanidis and J. N. Lygouras, “Data glove with a force sensor”,IEEE Trans. on Instrumentation and Measurement, vol. 52, issue.3,pp.984-989, June 2003. [2] T. E. Starner, “Wearable computers: no longer science fiction”, IEEE Pervasive Computing, vol.1, no.1, 2002. [3] R. E. Morley, E. J. Richter, J. W. Klaesner, K. S. Maluf, and M. J.Mueller, “In-shoe multisensory data acquisition system”, IEEE Trans. on Biomedical engineering, vol.48, no.7, July 2001. [4] J. Paradiso, E. Hu, and K. Y. Hsiao, “The cybershoe: wireless multisensor interface for a dancer’s feet”, in Proceesings of Internaional Dance and Technology, Tempe, Atizona,1999. [5] I. P. Pappas, T. Keller, and M. R. Popovic, “A novel gait phase detection system”, in Proceesings of Workshop Automatisierungstechnische Verfahren fr die Medizin, Darmstadt, 1999. [6] Stacy J. Morris and J. A. Paradiso, “A compact wearable sensor package for clinical gait monitoring”, Offspring vol.1, no.1, pp.7-15, January 31, 2003. [7] A. Watts, “On exponential smoothing of discrete time series”, IEEE Trans. on Information Theory, vol.16, issue.5, pp.630-630, Sep, 1970.