1. Scrabble Tile Counter CSC 375/475 Introduction to Robotics DePaul University Caden Howell June 10, 2009

2. ©2009 Caden Howell Third party projects are the property of their respective owners. What is it? Attempt to read the letters of a set of Scrabble tiles Report whether the set matches the standard Scrabble tile distribution.

3. ©2009 Caden Howell Third party projects are the property of their respective owners. How does it work? Use NXT hardware to physically move tiles over the webcam Break the letter down into a grid of superpixels Train a neural network on the superpixel values –Classified 2200 tiles by hand for training data. –Input is 100 superpixel nodes set to 1 or 0 –Output of neural network is 6 bits representing a letter Use the network to classify tiles as letters

4. ©2009 Caden Howell Third party projects are the property of their respective owners. Collecting training data

5. ©2009 Caden Howell Third party projects are the property of their respective owners. Qwiknet Illustration of neural net back end structure, running in QwikNet. Neural net used by the Scrabble tile project was a different implementation written in Python.

6. ©2009 Caden Howell Third party projects are the property of their respective owners. ROBOTC program Used ROBOTC sample Bluetooth code BtBasicMsg.C (See C:\Program Files\Robotics Academy\ROBOTC for Mindstorms\Sample Programs\NXT\Bluetooth) void readMessages() { static int nLastMessage = 0; while (true) { nMessage = message; if (nMessage != 0) { //Keep a running count of the number of messages successfully read ++nNearEndRead; if (nMessage != (nLastMessage + 1)) ++nReadOutOfSequence; nLastMessage = nMessage; nFarEndSent = messageParm[1]; nFarEndRead = messageParm[2]; ClearMessage(); nElapsedTime = nPgmTime; dispenseTile(15); } else ++nRcxNoMsg; wait1Msec(30); }

7. ©2009 Caden Howell Third party projects are the property of their respective owners. Python libraries wxPython for windowing OpenCV for computer vision PIL Python Image Library speech for Microsoft Speech SDK (TTS) jaraco.nxt for sending Bluetooth commands to NXT (thanks Krista) bpnn.py is an open source implementation of a simple single-layer neural network

8. ©2009 Caden Howell Third party projects are the property of their respective owners. External programs used Weka* to compare different classification methods Qwiknet* to find good parameters for the neural net *both of these programs are introduced in DePauls Machine Learning class Nconvert to preprocess images (thanks Ben) –nconvert -o nconvertout.bmp -brightness 0 -gauss 5 - gamma 4 -contrast 40 -noise reduce histogram.bmp

9. ©2009 Caden Howell Third party projects are the property of their respective owners. Accuracy Training data (tested in Qwiknet) 99.6% Test data (401 points) 90.8%

10. ©2009 Caden Howell Third party projects are the property of their respective owners. Issues/Improvements Built-in webcam badly positioned –Used external webcam Direct control Bluetooth commands not successful –Moved motor commands to ROBOTC program Bad classification accuracy from neural network –Expanded classes to include inverted tiles (52 instead of 26) –Threw out outlier training data Poor/inconsistent lighting conditions –Added bike light Increased grid size –10 x 10 instead of 8 x 10 –Had to throw out 1000 data points collected by hand Poor documentation for OpenCV –Used nconvert instead

11. ©2009 Caden Howell Third party projects are the property of their respective owners. Future Improvements Most tile sets are 100 tiles of a known distribution; distribution of remaining tiles should be factored in to tile classification Noise due to rotation and shifting could have been reduced by reducing the width and breadth of the hopper