1. CC467 Biologically inspired robotics Introduction to Lego Mindstorms Owen Holland Computer Science University of Essex

2. In most robotics work, you are given a task, and a robot that must perform it, and you program the robot. Easy. Why Lego is different

3. In the old days, there were hardly any robots. If you wanted a task to be done by a robot, you first had to build your robot, and then you would be able to program it. Not easy… - mechanical design and manufacture - electronics design and manufacture - system integration - and then you could program the robot, often in assembly language Why Lego is different

4. Now, with Lego, most of the hard bits disappear: you just stick Lego bits together to make the robot you want - NO mechanical design and manufacture - NO electronics design and manufacture - NO system integration - and you can program the robot in almost any language you want: C, Java, Lisp, Ada, Prolog, Logo, and even assembler. - easy. Why Lego is different

5. Let’s look at the history of Lego robotics. It goes back a long way… At MIT in the 1980s, the Mechanical Engineering Department started a course called 2.70: “Introduction to Design”. Students used scrap parts to build joystick- controlled machines that competed against each other at the end of the course. Isn’t Lego just a toy?

6. In 1987, an Electrical Engineering and Computer Science (Course 6) student wanted a similar course in his department, and invented the 6.270 competition: Students programmed simulated autonomous robots that competed against one another at the end of the course. The birth of 6.270

7. At about the same time, a group of people at MIT Media Lab were sponsored by Lego to develop a system to enable children to build robots using Lego, and to control them using Logo, a simple programming language fashionable at the time. One of these people, Fred Martin, began helping with 6.270, and developed some Lego-based kit to help. LEGO/Logo

8. One of the outputs of Fred’s Lego sponsored project was the MIT Programmable Brick Back to Fred Martin Note the 6 sensor inputs and 4 motor outputs

9. An in-house Lego design team, obviously inspired by the MIT work, developed the Lego RCX brick. And finally… Note the 3 sensor inputs and the 3 motor outputs

10. Released in 1998 in various kits. Little changed since. Aimed at children. All kits contain: - an RCX brick - 2-way IR communication between RCX and PC - Lego and Lego Technic parts - sensors and motors Lego Mindstorms

11. Lego light sensor Lego touch sensor Lego temperature sensor Lego rotation sensor Lego sensors

12. This measures the intensity of the light coming in through the lens in the front of the sensor. It’s quite directional. The light coming in through the lens can come from two sources: (1)The environment – we call it ‘ambient light’. (2)Light coming from the red LED on the front of the sensor that has been reflected off objects in the environment. The amount reflected depends on the colour of the object. The Lego light sensor

13. - Interfaces with all Lego and Technics parts - Practically bomb-proof - 4 types of sensors available (touch, light, rotation, temperature) - Good battery life - Not very expensive for whole kits - Good support for schools Lego Mindstorms: the Good

14. - Motor control is awful - Limited sensor inputs and motor outputs - Individual parts are very expensive Lego Mindstorms: the Bad

15. - Dumbed down programming environments Lego Mindstorms: the Ugly Visual programming (RCX code)

16. The RCX brick contains a very capable and fully featured microcontroller, the Hitachi H8/3292, running at the maximum clock speed of 16MHz. It has 16kB of on-chip ROM (Read Only Memory) containing routines for communication with the IR channel, sensor and motor interfacing, LCD drivers, some test programs, and so on. It imposes some limitations, but nothing too serious, which is just as well because nothing can be done about it. Is there any hope?

17. The RCX brick also contains 32kB of external RAM (Random Access Memory). This holds information as long as the battery is connected. It is used for two things: (1)The ‘firmware’ that forms a kind of operating system. It is uploaded to the RAM via the IR link. The firmware supplied by Lego has some limitations (no floating point, only 32 variables). (2)The user programs, compiled on the PC and uploaded via the IR link. Only 6kB is available for these! Is there any hope?

18. Here is what happens: - you produce a program on your PC - the program is transformed into low-level instructions (bytecodes) - the bytecodes are transmitted via the IR link to the RCX RAM - the firmware interprets the bytecode and converts it into machine code using the ROM - the processor executes the machine code Yes, there is hope.

19. If we only knew what the bytecodes actually caused the microcontroller to do, we might be able write a program to generate more interesting sequences of bytecodes than we can using the Lego software. In 1998, Kekoa Proudfoot was a student at Stanford University… Yes, there is hope.

20. He reverse engineered the RCX, discovering and cataloguing all the bytecodes for the interpreter, and much else besides. Respect to Kekoa Proudfoot

21. The RCX board – top view

22. Bottom view

23. Following on from Kekoa Proudfoot’s work, Dave Baum was able to write NQC – Not Quite C. This allows you to write a program for the RCX in a language very similar to C, and to upload it to the RCX, where you can run it as normal. NQC is text-based. Now other people have written pretty front ends so it’s just like normal programming in a typical IDE – you’ll be using John Binder’s Bricx Command Center. Enter Dave Baum

24. NQC uses the original Lego firmware. If we used Kekoa Proudfoot’s discoveries, could we devise new firmware that would enable us to do even more? YES. In 1999 Marcus Noga began the LegOS project, designed to replace the Lego firmware with a new version allowing more or less direct control of the Hitachi processor. Smaller firmware, bigger programs (in C), and much faster execution – but for experts only. More hope…

25. And now… The newest firmware replacement is part of the LeJos system. It permits a Java Virtual machine (JVM) to run on the RCX. You can now program your robots in Java (in fact a subset of Java). Why is this nice? Because it gives you access to lots of the neat stuff associated with Java. The LeJos package comes with classes for handling all the Lego hardware – you don’t have to build everything from scratch, which is just as well because Java is really too high level for robotics.

26. From the early days, enthusiasts began designing and building their own sensors. Some of these are excellent. Some are on sale. Some third-party sensors are even supported by Lego. Hardware developments

27. Here’s a list of home-brewed sensors from just one site…

28. From the early days, enthusiasts began designing and building their own sensors. Some of these are excellent. Some are on sale. Some third-party sensors are even supported by Lego. If you wanted to build a complex Lego robot now, you could probably find designs for the sensors you need on the Web. Some will be accompanied by the Java classes for dealing with them. The range of hardware and software now available for Lego has transformed it from a toy into a serious educational and investigative tool. Hardware developments

29. Dozens of universities in many countries, including top engineering colleges in the USA, Switzerland, Denmark, Germany, and Sweden. What do they use it for? Teaching programming Teaching control Teaching robotics Individual projects Student competitions etc. etc. So who uses it?

30. Books: Core Lego Mindstorms Programming by Brian Bagnold Good intro to Mindstorms and to LeJos Web: Too many sites to list. Just Google any mixture of mindstorms, lego, robot, sensors, java, nqc, rcx. Lots of good stuff, and most sites have links to lots of others. How can I find out more?

31. Some of the simplest robots are called Braitenberg machines. In two weeks you’re going to build and program the one known as Vehicle 2. A simple robot

32. In the space of 2 hours, working in pairs, and starting from scratch, you have to build, program, and run such a vehicle. It’s not realistic. So we’ll give you some strong guidance, in the form of the instructions to build a suitable vehicle… …and in the form of a suitable program tailored to the vehicle We have a problem

33. A suitable vehicle needs the following: - a left motor (LM) - a right motor (RM) - a left sensor (LS) - a right sensor (LR) Back to basics

34. A suitable program needs the following: - a way of driving each motor as a function of the reading on the sensor on the opposite side Back to basics

35. And the environment needs to ensure that the change in sensory inputs produced by the movement of the vehicle leads to some interesting and obviously useful behaviour… …with a simple performance measure so that we can have a competition Back to basics

36. All vehicles will have to be run in exactly the same environment The only suitable sensors are light sensors Constraints

37. Consider the termite

38. Termites (and many other trail-following insects) follow trails by sensing concentrations of special chemicals (pheromones) using their antennae. How termites work

39. They use a technique called osmotropotaxis. Roughly speaking, their left legs go at a speed controlled by the strength of the sensory input to their right antenna, and their right legs go at a speed controlled by the strength of the input to their left antenna… In other words, they operate just like little Braitenberg vehicles, and this enables them to follow trails. How termites work

40. Yes…there are some trails on the floor of the arena… Could we make visible trails on the ground, and sense them with light sensors?

41. …if the trail is either black or white, won’t the sensors only ever give maximum or minimum readings? No, because the sensor takes in reflected light from an area, not a point. If the area is wholly inside the black region, you’ll get the minimum reading (with some noise) And if it’s wholly inside the white region, you’ll get the maximum reading (with some noise) But surely…

42. …if the trail is either black or white, won’t the sensors only ever give maximum or minimum readings? No, because the sensor takes in reflected light from an area, not a point. If the area contains both black and white regions – if the sensor is somewhere on the borderline – then the reading will be somewhere in between the minimum and maximum (again, with some noise) But surely…

43. If we space the sensors so that they are separated by just less than the width of the trail, then when both sensors read the minimum, we know we must be on the trail… …and so moving forwards is a good idea A basic strategy

44. If the sensor on the left reads the maximum, and the sensor on the right reads the minimum, we know we’re moving off the trail to the left… …so turning right is a good idea A basic strategy

45. If the sensor on the right reads the maximum, and the sensor on the left reads the minimum, we know we’re moving off the trail to the right… …so turning left is a good idea A basic strategy

46. And if both sensors read the maximum, you know you’ve completely lost it Of course, you could devise some cunning way to get back on the trail (termites do)… A basic strategy

47. What if one or both sensors give a reading somewhere in between? The quick and dirty solution: keep on doing whatever you were doing – it won’t last The elegant solution: you can produce one if you want, but it won’t make any difference… But…

48. We can break this into two stages: (1) How far down the trail you get (2) If you make it to the end, how long it took The performance measure

49. You can make whatever you like, but since time is so short I recommend that you use a version of the Trusty robot described in Jonathan Knudsen’s “The Unofficial Guide to Lego Mindstorms Robots” Instruction sheets and pictures will be in the lab. First, catch your robot

50. You’ll be programming your robot in NQC (Not Quite C). You’ll use the BricxCC (Bricx Control Centre) as your development environment. When you first fire up BricxCC (by double clicking on the BricxCC icon) it looks for the RCX brick. It will find it, because you will have switched it on (using the red On-Off button) and placed it with its infra-red windows facing the IR tower and a few centimetres away from it. And finally…the program

51. Click on RCX2, then OK

52. You’ll get this. You should also get another window

53. If you don’t, press F9 once or twice until you do

54. File, New gets you a programming window

55. Copy and paste the text file ‘trusty’ from your email

56. And save it locally as trusty (a.nqc file)

57. Click on Compile to compile, and on Download to put it into the RCX.

58. You will get helpful error messages below the program If you can’t sort out the errors by yourself, call Renzo, the GTA If your program won’t compile…

59. It’s straight from the Knudsen book – a photocopy of the relevant pages will be in the lab. Since I can’t teach you NQC in 2.4 minutes, we’ll skip through some basics, and then go through the program. The trusty.nqc program

60. NQC/RCX basics: Sensors Sensor connections 1, 2, and 3 are known as SENSOR_1, SENSOR_2, and SENSOR_3 You have to tell each sensor connection what sort of sensor is plugged into it. For the light sensor, it’s just a matter of: SetSensor (SENSOR_2, SENSOR_LIGHT)

61. NQC/RCX basics: Motors Motor connections A, B, and C are known as OUT_A, OUT_B, and OUT_C You set the direction of a motor by OnFwd (OUT_A) or OnRev(OUT_B) You set the power of a motor by SetPower (OUT_A, x) where x runs from 0 to 7 You turn a motor off by Off(OUT_B)

62. NQC/RCX basics: Tasks NQC can run several tasks at the same time. (Unfortunately you have to be careful to ensure that tasks do not conflict, especially over resources.) The format of a task is: task taskname( ) {task instructions etc go here} Every program must contain a task called main: task ( ) main {stuff}

63. NQC/RCX basics: Subroutines There are several ways of avoiding having to write out long strings of repetitive chunks of code. One is to use subroutines. The format of a subroutine is: sub subroutinename ( ) {code goes here} And you call it in the main body of your code by subroutinename ( ); (Semicolons are used as separators for expressions etc.)

64. NQC/RCX basics: Control structures while (true) {stuff} will execute stuff forever. if (condition) do_something_simple will do_something_simple if the condition is true if (condition) do_something_simple else do_something_else will do_something_else if the condition is not true

65. All variables are integers. You have to declare them, otherwise the compiler will moan. int thing; at the beginning of your program will let you use the integer variable thing anywhere. If you declare it within some curly brackets (as part of an expression or subroutine etc.) it won’t work if you try to use it outside the brackets (and the compiler will complain). NQC/RCX basics: Variables

66. You can give constants meaningful names instead of numbers by defining them at the start of your program: #define MONKEY_SCORE 25 means you can write MONKEY_SCORE in your code instead of just 25, and so you will be able to distinguish it from all the other constants that might be 25 – for example, SLAMMER_RECORD NQC/RCX basics: Constants

67. int state; #define BOTH_ON 3 #define LEFT_ON 1 #define RIGHT_ON 2 #define BOTH_OFF 0 #define INDETERMINATE 255 #define DARK2 35 #define LIGHT2 40 #define DARK3 40 #define LIGHT3 45 #define POWER 4 trusty.nqc

68. int state; THIS IS THE DECLARATION OF THE VARIABLE WE WILL USE TO REPRESENT ALL THE DIFFERENT MEANINGFUL COMBINATIONS OF SENSOR STATES trusty.nqc

69. #define BOTH_ON 3 #define LEFT_ON 1 #define RIGHT_ON 2 #define BOTH_OFF 0 #define INDETERMINATE 255 THESE ARE THE VALUES THE VARIABLE state CAN TAKE. BOTH_ON MEANS BOTH SENSORS ON THE TRACK, ETC. INDETERMINATE MEANS AT LEAST ONE SENSOR IS ON THE BORDERLINE trusty.nqc

70. #define DARK2 35 #define LIGHT2 40 #define DARK3 40 #define LIGHT3 45 THESE ARE THE SENSOR INPUT VALUES FOR DECIDING WHETHER A SENSOR IS ON OR OFF THE LINE, OR ON THE BORDERLINE trusty.nqc

71. Sensor tip If the sensor port is correctly configured, you can read the value of a sensor directly from the LCD display on the RCX. Press VIEW. You will see a little ^ pointing to input 1. The number on the LCD will be the current input of the sensor connected to input 1. Press VIEW again. The ^ will move round one place to input 2. Keep on pressing and it will go 3, A, B, C, and then out of VIEW mode.

72. #define POWER 4 THIS SETS THE POWER LEVEL THAT WILL BE USED FOR MOTORS THAT ARE ON. trusty.nqc

73. task main() { initialize(); while (true) { if (state == BOTH_ON) OnFwd(OUT_A + OUT_C); else if (state == LEFT_ON) { Off(OUT_A); OnFwd(OUT_C); } else if (state == RIGHT_ON) { Off(OUT_C); OnFwd(OUT_A); } trusty.nqc

74. task main() { initialize(); while (true) { CODE HERE } THIS STARTS THE MAIN TASK, INITIALIZES EVERYTHING USING THE SUBROUTINE INITIALIZE, AND EXECUTES THE CODE IN THE MIDDLE trusty.nqc

75. if (state = = BOTH_ON) OnFwd(OUT_A + OUT_C); else if (state = = LEFT_ON) { Off(OUT_A); OnFwd(OUT_C); } else if (state = = RIGHT_ON) { Off(OUT_C); OnFwd(OUT_A); THIS GOES FORWARD FOR BOTH_ON, TURNS LEFT FOR LEFT_ON, AND TURNS RIGHT FOR RIGHT_ON trusty.nqc

76. sub initialize() { SetSensor (SENSOR_2, SENSOR_LIGHT); SetSensor (SENSOR_3, SENSOR_LIGHT); SetPower (OUT_A + OUT_C, POWER); OnFwd(OUT_A + OUT_C); start watcher; } THIS IS THE SUBROUTINE initialize. IT SETS THE SENSOR INPUTS CORRECTLY, STARTS THE ROBOT GOING FORWARD, AND STARTS THE TASK watcher. trusty.nqc

77. task watcher() { while (true) { if (SENSOR_2 < DARK2) { if (SENSOR_3 < DARK3) state = BOTH_ON; else if (SENSOR_3 > LIGHT3) state = LEFT_ON; else state = INDETERMINATE; } else if (SENSOR_2 > LIGHT2) { if (SENSOR_3 < DARK3) state = RIGHT_ON; else if (SENSOR_3 > LIGHT3) state = BOTH_OFF; else state = INDETERMINATE; } else state = INDETERMINATE; } trusty.nqc

78. task watcher() { while (true) { ONCE STARTED, WATCHER KEEPS ON RUNNING FOR EVER trusty.nqc

79. if (SENSOR_2 < DARK2) { if (SENSOR_3 < DARK3) state = BOTH_ON; else if (SENSOR_3 > LIGHT3) state = LEFT_ON; else state = INDETERMINATE; IF BOTH SENSORS ARE BELOW THEIR INDIVIDUAL DARK THRESHOLD, state IS SET TO BOTH_ON. IF THE SENSOR ON THE LEFT IS DARK, AND THE SENSOR ON THE RIGHT IS ABOVE ITS LIGHT THRESHOLD, state IS SET TO LEFT_ON. IF THE SENSOR ON THE LEFT IS DARK, AND THE SENSOR ON THE RIGHT IS BETWEEN ITS LIGHT AND DARK THRESHOLD, state IS SET TO INDETERMINATE trusty.nqc

80. else if (SENSOR_2 > LIGHT2) { if (SENSOR_3 < DARK3) state = RIGHT_ON; else if (SENSOR_3 > LIGHT3) state = BOTH_OFF; else state = INDETERMINATE; THIS SORTS OUT RIGHT_ON AND BOTH_OFF, AND ANOTHER KIND OF INDETERMINATE trusty.nqc

81. else state = INDETERMINATE; AND THIS CATCHES THE CASE WHERE BOTH SENSORS ARE BETWEEN THRESHOLDS THAT’S ALL YOU NEED – OFF YOU GO… trusty.nqc