University of Bridgeport Computer Science and Engineering Department Ph.D. Dissertation Prospectus UBSwarm: Design and Evaluation of a Software Environment to Deploy Multiple Decentralized Robots By: Tamer Abukhalil Under the Supervision of: Dr. Tarek Sobh Committee Members Dr. Khaled Elleithy, Chairman Dr. Ausif Mahmood, Member Dr. Zhengping Wu, Member Dr. Gonhsin Liu, Member Dr. Lou Manzione, External Examiner

Table of content Introduction Research problem Research scope Motivation behind the research Potential contributions of proposed research

Introduction When the swarm robotic system adopts a decentralized approach, the desired collective behaviors emerge from local decisions made by robots themselves according to their environment. Decentralized modular robotics is an emerging area that has attracted many researchers over the past few years. It has been proven that a single robot with multiple capabilities cannot necessarily accomplish an intended job whereas different robots, each one with its own configuration, are more flexible, robust and cost-effective. The fact that a typical swarm of robots consists of relatively simple and homogeneous robots allows the group to self- organize or dynamically reorganize the way individual robots are deployed. Therefore, the swarm approach is considered to be highly robust to the failure of individual robots.

Motivation/Objectives Our approach is borrowed from bees and ants, which have very limited intelligence and behaviors but can survive, reproduce, carry out tasks, attack and protect each other if they work in groups or with the entire colony. The overall high intelligence of the group is actually created by the simple acts and moderate local intelligence of each individual. We conducted experiments in which each simple robot performs 3 major tasks. Then we will add more robots when performing each task and see how the swarm system acts. Our research of swarm robotics is to study the design of robots, their physical design and their controlling behaviors. Our research aim to develop very simple and inexpensive robots that individually perform very simple behaviors such as catching images on a camera, moving things around, communicating with nearby peers and doing very simple laser scanning. In this work we propose a software application that aims to facilitate the rapid deployment of a multiple robotic agents and configuring with dynamic task allocation that helps the operator to select between several available robot agents and assign the group of robots a particular task from a set of available tasks.

Robot Deployment Environment Research Scope We provide a detailed summary of systems that have been classified under four main categories of the general multi-robot system platforms, namely: self-reconfigurable, modular, and self- replicating. and then we summarize the behavior and the configuration of most recent swarm system in tables. We also would like to bring up an analysis of existing swarm control environments and then propose a software application that aims to facilitate the rapid deployment of a multiple robotic agents and configuring these robots using a friendly GUI interface that helps the operator to select between several available robot agents and assign the group of robots a particular task from a set of available tasks. Operator Robot Agents

Literature Survey (Scope 1) Obtaining decentralized control that provides interesting collective behaviors is a central problem [16, 33-41]. Several algorithms have been developed to run on swarms of robots. The complexity varies between these algorithms. Some provided basic functionality while others exhibited complex interactions between the team of robots such as bidding on tasks according to some rules. Table 1 in our survey paper ( page in the dissertation) summarizes the most recent swarm robot systems with their corresponding algorithms. These are systems introduced in literature that only involve multiple agent teams with decentralized control A respectable number of self-reconfigurable robot systems have been proposed in the last decade. Table 2 ( page in the dissertation) shows comparisons between the most recent ten robotic reconfigurable systems.

Some part of table 1

Literature survey (Scope 2) Before talking about the multi-agent robotic deployment environments and interfaces, I will provide a brief summary of the popular educational and research platforms available in the market. Several off-the-shelf mobile robots with various sensors and diverse capabilities are illustrated in Fig. 2. Fig. 2. Educational and research mobile robotic platforms: from left to right, iRobot Create, Turtlebot, Mindstorm NXT, e-puck, MarXbot, SRV-1 Blackfin and Pioneer 3-DX, respectively.

Literature survey (Scope 2) Cont. The Roomba Create [10] from iRobot was designed for students and researchers, being very popular in the robotics community due to its small size and low cost. It is a circular platform, with extra space for larger sensors (e.g., 2D laser sensor or Kinect). The TurtleBot3 is built suing Roomba. It is a modular development platform incorporating an Xbox Kinect and an ASUS eeePC 1215N netbook.ensor or Kinect). The Mindstorms NXT [11] from Lego is an educational, academic robot kit, ideal for beginners. The robot is equipped with drive motors with encoders and a good variety of sensors like light, sound, ultrasound and touch sensors. The e-puck [12] has tiny dimensions with only 80mm of diameter, equipped with a vast set of sensors like microphones, infrared sensors, 3D accelerometer and a VGA camera. Similarly, the MarXbot [13] platform has 170 mm of diameter, being fully equipped with infrared range sensors, 3D accelerometer, gyroscope, and an omnidirectional camera. It has a good processing power with an ARM 11 processor at 533MHz. Both the e-puck and the MarXbot are programmed in a user-friendly scripting language, which uses ASEBA the SRV-1 Blackfin [14] from Surveyor is a small-sized robot equipped with tracks with differential configuration. This robot has a good processing power with a 1000MIPS at 500MHz CPU, capable of running Linux Kernel 2.6. It is equipped with two IR rangers or optional ultrasonic ranging and a 1.3MP camera. It also supports Wireless b/g communication and various I2C sensors. the Pioneer 3 DX from ActivMedia [15]. This is a robust differential drive platform with 8 sonars in a ring disposition, a high performance onboard microcontroller based on a 32-bit Renesas SH RISC microprocessor, offering great reliability and easiness of use.

Cost of our robots vs. Commercial ones SRV-1 on the right is available to purchase at the cost of $580. too expensive despite the fact it only operates using commands sent from a computer using wireless network. One of our reconfigurable robots cost $315 only and it comes with the ability to be programmed for various tasks.

Robotics Deployment Env. Our proposal is based on the assumption that there is a lack of software packages which provide control for the different platforms of robots individually and allow concurrent control of heterogeneous robotic teams. Our objective is to develop algorithms that can provide connectivity between multiple agents, besides building central software to deploy these agents. In robotic control environments, some graphical application software like MobileEyes [7] and C++ based software like URBI [8] are available as an open source systems. URBI provide GUI packages that aim to make compatible code to different robots, and simplify the process of writing programs and behaviors for these robots. URBI works by incorporating sensor data to initiate commands to the robot. Moreover the URBI’s uniform programming language is limited to few kinds of microcontrollers available on the market.78

Robotics Deployment Env. The Player/Stage proposed by Gerkey et. al. [9] also produces tools for simulating the behavior of robots without an actual access to the robots hardware and environment. Its two main products are the Player robot server, a networked interface to a collection of hardware device drivers, and Stage, a graphical, two- dimensional device simulator. The player/Stage is open source software that is available to be downloaded online on UNIX-like platforms. However, running this software requires variety of prerequisite libraries and each library requires another set of libraries. It was never been easy to understand how the system communicates with the actual robots.9 Another script-based robot programming is Pyro[10]. Pyro, which stands for Python Robotics, is a robotics programming environment written in the python programming language. Programming robot behaviors in Pyro is accomplished by programming high-level general-purpose programs in that Pyro provides abstractions for low-level robot specific features much like the abstractions provided in high-level languages. The abstractions provided by Pyro allow robot control programs written for small robots to be used to control much larger robots without any modifications to the controller. This represents an advance over previous robot programming methodologies in which robot programs were written for specific motor controllers, sensors, communications protocols and other low- level features.10

Robotics Deployment Environments, Kulis et al., [55] have proposed a software framework for controlling multiple robot agents by creating what they have named the Distributed Control Framework (DCF). DCF is an agent-based software architecture that is entirely written in Java.55 The default DCF Context is an XML file which, with the help of other externally-referenced XML files, provides all of the information necessary to launch and configure the DCF environment and the participating robot agents. Separate XML files may be used to define the capabilities of the various robot agents and to configure the sensors, actuators, and controllers used by the robot agents in the system. For those familiar with Java programming, the DCF Context is essentially a human-readable XML-serialized version of a JavaBean class (DcfBean) which stores all configurable parameters of the DCF environment.DcfBean The DCF is specifically designed to control interacting heterogeneous agents. DCF uses a high-level platform- independent programming language for hybrid control called MDLE. The DCF architecture consists of two distinct agents: a Robot Agent and a Remote Control Agent (RCA). The RCA lies within the human interface shown in Fig 3. They say the DCF is available in multiple version for commercial or academic use on the following link however, the link has never works. Figure 3: The DCF human interface (© 2008 IEEE) 1

Robotics Deployment Environments Gregory P. Ball G. et al. [8], have proposed application software built in JAVA to operate heterogeneous multi-agent robots for the sake of educational purposes named MAJIC. The system provides basic components for user interaction that enables the user to add/remove robots change the robotic swarm configuration, load java scripts into robots and so on as shown in Fig 4.8 Authors did not explain how their formal language is translated to program specific to different robots. FIGURE 4: The MAJIC Control Platform (© 2008 IEEE) 2

Our solution: UBSWARM We are developing an environment to utilize robots that have different modular design and configuration of sensory modules, and actuators. We built 5 different robots and we ran 3 experiments. The system will be implemented as a GUI interface to reduce efforts in controlling swarm robotic agents. The proposed application offers customization for robotic platforms by simply defining the available sensing devices, actuation devices, and the required tasks. The main purpose for designing this framework is to reduce the time and complexity of the development of robotic software and maintenance costs, and to improve code and component reusability. Usage of the proposed framework prevents the need to redesign or rewrite algorithms or applications due to changes in the robot’s platform, operating systems, or the introduction of new functionalities.

UBSwarm Cont. UBSwarm environment is a high end interface used for distributing algorithms to heterogeneous robotic agents. One of the key features of UBSwarm is configuring special programs which act as a middleware that gains control over agent’s parameters and devices which consequently allows auto-detection of the attached standardized components according to current system This real time reconfiguration of devices attached to different robots and driver software makes it easier and more efficient for end users to add and use new devices and software applications. The basic hierarchy of the UBSwarm deployment platform is shown in Fig. 5. Robot Deployment Environment Robot Control Middleware User interface Face Detect ion Obstacle Avoidanc e Navigatio n Coordination Agent HAL Knowled ge base Device Agent Polling routine Hardware Components XbeeSonarGPS Robot N Robot 1 Fig. 2: System Architecture Device Library

System Architecture UBSwarm is an interactive Java-based application designed for extensibility and platform independence. The system establishes communications with embedded robot modules via various mediums. At the time of startup the system will expect to operator to:  Configure the system by picking the available agents, their onboard features (sensors, motors, etc.) and the services needed to accomplish each task  Or simply run the system using the last executed configurations The system is divided into two main subsystems, a robot deployment system and a robot control and translation system. ◦ The robot control system includes a robot control agent in which the user should provide all the parameters required for all sensors incorporated on robots. The user should also describe actuation methods used. ◦ The robot deployment system encapsulates a variety of high-level applications module which contains the tasks that the platforms will perform such as navigation, area scanning, and obstacle avoidance. A hardware abstraction layer is used to hide the heterogeneity of lower hardware devices and provide a component interface for the upper layers call.

Robot Deployment System deployment system takes the responsibility of running actions according to the definition parameters and integrates the heterogeneous robots. Each application is implemented as a software module to perform a number of specific tasks used for sensing, decision-making, and autonomous action. Actions are platform independent (ex: Arduino and Digilent Max32 microcontrollers) algorithms ; for example, it can be an obstacle avoidance function written in C++, room scanning, or object grapping. Etc. The deployment system contains a developer interface, coordination agent, dynamic interpreter, and knowledge base. 1. Operator Interface: The user can interact with the computer through interaction tools which provides a list of actions/tasks and the available robotic agents. In some other parts of the interface, the user will be prompted to input the required system parameters for all sensors incorporated on robots such as the PIN location that each sensor/actuator is connected to on each robot. When connecting the robot to the USB, UBSwarm will detect the COM port automatically. The interface provides a number of tasks to be assigned the group of robots as can be seen in fig. 6 (a).

Operator Interface The user interface also provides a mechanism to allow users to update, remove, or add a robot application. After clicking on a particular task, the user will be prompted to decide the number of robots displayed in a list of the available robot types by manipulating the arrow buttons as shown in fig. 6 (b). (a)(b) Fig. 6

Operator Interface The user will be then be asked to enter each agent’s pin locations (once for each type of robot) associated with various hardware components such as ultrasonic sensors, scan servo motors, and the n pin locations for the n-Dof arm if any is attached on the robot. A value of -1 will be assigned to pin locations of components that does not exist on the particular robot. The C# programs which will be uploaded on each robot type will differ according the different pin locations associated with each type that were set by the user. Different sensors have different piece of code (functions) to operate. The particular function will be activated only of the pin number associated with the intended sensor/hardware component. Example: In main function: Void loop() { If (trigpin>-1 && echopin>-1) Ping(); If (URPWM>-1) PWM_Mode();. }

URM V3.2 Ultrasonic Sensor int PWM_Mode() { // a low pull on pin COMP/TRIG triggering a sensor reading digitalWrite(URTRIG, LOW); digitalWrite(URTRIG, HIGH); // reading Pin PWM will output pulses unsigned long DistanceMeasured=pulseIn(URPWM,LOW); if(DistanceMeasured==50000) { // the reading is invalid. Serial.print("Invalid"); } else{ distance=DistanceMeasured/50; // every 50us low level stands for 1cm } return (distance); } void PWM_Mode_Setup() { pinMode(URTRIG,OUTPUT); // A low pull on pin COMP/TRIG digitalWrite(URTRIG,HIGH); // Set to HIGH pinMode(URPWM, INPUT); // Sending Enable PWM mode command for(int i=0;i<4;i++){ Serial.write(EnPwmCmd[i]); }}

HC-SR04 and PING Ultrasonic Range Finder int ping(){ long duration, inches, cm; //Send Pulse pinMode(trigPin, OUTPUT); digitalWrite(trigPin, LOW); delayMicroseconds(2); digitalWrite(trigPin, HIGH); delayMicroseconds(10); digitalWrite(trigPin, LOW); //Read Echo pinMode(echoPin, INPUT); duration = pulseIn(echoPin, HIGH); // convert the time into a distance inches = microsecondsToInches(duration); cm = microsecondsToCentimeters(duration); Serial.print("Ping: "); Serial.println(inches); return round(cm); }

Operator Interface The computer will ask the user to connect each robot for the purpose of uploading the program on each robot as shown in fig. 7 Fig.7

2) Runtime Interpreter When new devices are plugged in, system developers can upload new platform software packages specific for the execution of the newly added devices. In other words, system developers can extend the system’s functionality by adding new service modules to the list of available modules. When new service is added to the system, the dynamic interpreter manages flow of information between these services by monitoring the creation and removal of all services and the associated static registries. The Dynamic interpreter maintains state information regarding possible & running local services. The Dynamic interpreter will be the first service created which in turn will wrap the real JVM Runtime objects. Applications Application objects Get new instance Service Service Manager Runtime agent Send service directory update (SDU) SDU = getLocalServices Program command flow Coordination Agent IfIf if service is not registered on name Get new instance Get list of registered services Get new service instant Update registry New task is added. Send msg (data) New service is added Add service(name) Fig. 8: Adding services in runtime

3) Robotic Control Middleware Each robot agent incorporates two software programs to perform its job: - Controlling Program - Polling routine (middleware) Fig. 9: (a) controlling program, (b) Interrupt execution

Polling routine When an external interrupt is activated, the processor takes immediate notice, saves its execution state, runs the polling routine, and then returns back to whatever it was doing before. Fig 5 (b) shows the sequence of actions when internal or external interrupt is triggered. The type of interrupt we used is an external button connected to an interrupt pin and the ground (GND). The polling algorithm can be summarized as follows: Initially some signal pins are connected to components Main program begin { Attach the Interrupt pin to the interrupt function If (interrupt is activated) then goto polling routine …. … } Polling routine { For each unassigned pin set its internal pull-up resistor to high For each unassigned pin wait for change in signal If signal change occur { Send a special signal and wait for response to learn type Add type of device and its pin number to vector array } Update pin assignments }

Polling routine we divided the set of pins into two categories: -Digital PWM pins can only be connected to Ultrasonic sensors or servo motors -Analog pin can only be connected to Infra-red or sonar sensors Fig. 10 Arduino board

Mapping task Our platforms as shown in fig. 11 is built using Arduino UNO, Arduino Due, and Digilent PIC baords, Fig. 11: The robots with showing different configurations

Mapping (SLAM) the first experiment deploys two robots with two sensing components whereas four robots were deployed each equipped with three sensing components in the second experiment. Fig. 12 Experiment one (a) The map generated (b) Position error (meters) in 10 Minutes running time.

SLAM Second Experiment Fig. 6 (a) and (b) shows the results of the second experiment that is when one more sensing component is added to each robot. Fig. 6. Experiment two (a) the map generated (b) Position error (meters) in 10 Minutes running time.

Conlusion We learned, as experimental results depict, the sensing and the overall task-specific capabilities of the platforms can be easily upgraded by adding another sensors, e.g., laser range finders and that is how the name reconfigurable robots come from. UBSwarm makes it easier to students to program robotic systems that use the actually available microcontrollers in the market. UBSwarm environment generates programs that cope with changes of the robots configurations. Running experiments has been easier using UBSwarm. Conducting multiple tests will eventually lead to the optimal configuration of the swarm system that the students are looking for.