1. Adrian BuckleyEvan DefibaughJohn DuhamelAlex KaiserDarnell ParkerCraig Riggins T Q ~Q

2. Project Objectives - Relay real time Nintendo 64 session data between multiple hosts for network play - Provide an interface for a Microsoft Kinect frontend in place of tradition controller - Utilize non-obtrusive electronics - Minimize cost and maximize functionality - Power consumption is a secondary concern

3. System Architecture Client 1Client 2Server Nintendo 64 ServerClient 1Client 2 Player 1 Television Player 2 Television Player 1 Controller Signal Player 2 Controller Signal Player 2 Controller Signal Player 1 Controller Signal N64 RCA Signal N64 RCA Signal N64 RCA Signal N64 RCA Signal N64 RCA Signal

4. Client Architecture Regular N64 Controllers Television ARM Cortex MCU N64 Controller Signal Kinect USB Signal Digital RCA Signal Digital Controller Signal Ethernet IO Microsoft Kinect Analog RCA Signal  MCU takes input from 4 N64 controllers  Microsoft Kinect can track 2 people

5. Server Architecture ARM Cortex MCU Digital Controller Signal Digital RCA Signal Ethernet IO Nintendo 64 N64 Controller Signal Analog RCA Signal  Can communicate with 1-4 clients  Moves controller signal from client to N64 and RCA signal from N64 to client

6. Unified Hardware N64 Controllers / Microsoft Kinect Television ARM Cortex MCU Controller Signal RCA Signal Controller Signal Ethernet IO RCA Signal Nintendo 64 Controller Signal RCA Signal  1 hardware architecture will be able to operate as either the client or the server

7. Advanced System Architecture Client 2 Client 1 / Server Nintendo 64 Client 1 / Server Client 2 Player 1 Television Player 2 Television Player 2 Controller Signal Player 2 Controller Signal Player 1 Controller Signal N64 RCA Signal N64 RCA Signal N64 RCA Signal N64 RCA Signal

8. Kinect Integrated Peripheral Kinect Processor Linux Microcontroller XBOX KinectProcessor Linux Microcontroller

9. Kinect Sensor  I/O 2 Cameras (RGB and IR Depth) IR Output 4 Array Microphone USB Port

10. Possible Implementations ROS: BeagleBoneVS C# 2010: NetduinoVS C# 2010: PC

11. Implementing Kinect via ROS on Beaglebone ROS (Robot Operating System) is an open source project focusing on interfacing microprocessors with robots (in our case the Kinect and a peripheral Linux microcontroller) Kinect Processor

12. Control Node

13. The Netduino operates on Microsoft Visual Studios C# 2010 via Net. Micro workforce Implementing Kinect via VS C# 2010 on Netduino

14. Minor Hiccups in Progression Using the Robot OS  ROS tutorial has incorrect/missing info  Open source, thus compatibility issues from node to node  Unintuitive to navigate Using the Netduino Netduino requires Net. Micro framework, where Kinect requires Net. Framework Trouble converting from standard to micro There is no problems implementing Kinect via PC

15. Last Resort and Additional info As a last resort the Kinect can be routed through a Windows PC for processing then sent to microcontroller. More ROS info http://www.ros.org/wiki/kinect (Install ROS) http://www.ros.org/wiki/ROS/Tutorials (ROS Tutorial) http://beagleboard.org/bone (Beaglebone Description) http://www.ros.org/browse/list.php (ROS software) More Net. Micro workforce http://docs.com/BTN4http://docs.com/BTN4 (Installing OpenNI) http://www.netduino.com/netduino/schematic.pdf http://www.netduino.com/netduino/schematic.pdf (Netduino Schematics) http://informatix.miloush.net/Microframework/Arti cles/WpfEmulators.aspxhttp://informatix.miloush.net/Microframework/Arti cles/WpfEmulators.aspx (.net micro install) http://informatix.miloush.net/microframework/Arti cles/Reflector.aspxhttp://informatix.miloush.net/microframework/Arti cles/Reflector.aspx (.net micro reflector) http://netduino.com/netduino/specs.htm (Netduino Specs)

16. Gesture and Speech Algorithms  Joint Data is returned in X, Y, Z, and W (confidence) data  Evaluate position of joints relative to another and trigger button pressed event  For audio grammar files must be written  Compare text output of audio and set trigger for the controller or the sensor as being used  Output control trigger and button triggers

17. Kinect Processor Interface  The Processor will output triggers in the form of buttons and sensor/controller triggers for both players one and two  The triggers will be relayed to the Linux microcontroller that will convert button trigger to N64 controller output format and use the control trigger to decide if forwarding the N64 controller data or sensor converted data

18. N64 Controller Tutorial

19. How the N64 Controller Sends Its Data The N64 controller only has three pins: Ground, data, and VCC. It takes a +3.6V supply. The N64 uses a protocol in which bits are sent serially to and from the controller as 4µs-wide pulses. ‘Low’ is 3 µs low and 1 µs high, and ‘High' is 1 µs low and 3 µs high.

20. N64 Data Polling The N64 continually polls the controller using an 8-bit command (0x01), and the controller reports the state of its buttons and analog joystick with a 32-bit response: 16 bits for the buttons, signed 8 bits for horizontal joystick position, and signed 8 for vertical. Each bit packet ends with a stop bit (1).

21. Button Configuration The buttons are mapped to their respective bits as follows: Sample Joystick Positions and respective bits: Bit01234567 ButtonABZStartD-UpD-DownD-LeftD-Right Bit89101112131415 ButtonNull LRC-UpC-DownC-LeftC-Right Left Extreme10000001 Left Off Center11111111 Right Off Center00000001 Right Extreme01111111 Up Off Center00000001 Down Off Center11111111

22. Joystick Off-Center Left & Down

23. The Translator Box The Kinect will take player movement/sound data and send over button and trigger data to the “translator box,” which then converts that data into the 64-bit struct format shown below. There will be a physical master switch on the translator box, to switch between Kinect control and N64 controller control. When this switch is set to “Kinect,” the translator will await trigger and button data from the Kinect to process. If the switch is set to “N64,” the translator will need to keep track of one or two controllers, simultaneously converting their button presses into the shown format, but with the “More Buttons” bit and time delay consistently set to zero. Button DataJoystick DataWhich Player?More Buttons?Time Delay 16 bits 1 bit (0 or 1)1 bitInt (30-bit)

24. Server to Translator to N64 Another Translator Box is needed to translate the Server’s data into N64 controller data for the N64. This translator will read in the controller data, and send the correct button sequence via serial ports to the N64, correctly timed. If the N64 polls the “controller,” but the translator has not received button data from the server, the translator will automatically respond with a 32-bit ‘0’ analog stream and continue doing so until new button data appears.

25. Audio/Visual Signal The video output of the Nintendo 64 carries visual information via an analog signal, plus two synchronizing signals for aligning the video signal on a television. The audio output, also analog, is split into two signals for stereo speakers in a television.

26. Audio/Visual Waveforms Video Audio

27. Audio/Visual Sampling The audio and visual output of the Nintendo 64 must be digitized and formatted for input into the server for transport over the internet. Once the audio/visual information is sent over the internet it must be converted back into an analog signal to be displayed on a player's television.

28. A Series of Tubes – Hardware  Initial design and testing will be done on an Arduino with an Ethernet shield. Ardiuno is easy to program to send and receive data packets.  After getting a fully functional system with the Premade hardware we will move on to a custom PCB with a Linux OS.

29. Our Internet Objectives  We need the data received from the server to be real time If there is missing data we just move to the next data packet. We know that the receiver is ready to receive packets.

30. TCP/UDP  TCP/IP uses an established connection between sender and receiver. This requires an acknowledge signal from the receiver which takes time.  UDP instead sends packets in a continuous stream and doesn’t care if packets are missing on the receiver side. Instead of waiting for acknowledgement UDP just tags the packet with the correct receiver IP address and sends it on its way.

31. UDP in depth  UDP sends data with a header, and within the header a 2byte portion telling the receiver how long the data section will be. This 2byte section allows for a maximum of 64Kb section of data which is more than enough for sending the n64 signal and frames of an RCA signal. The other parts of the header are receiver IP and port number.

32. Putting it all Together  Using UDP because we need a real time connection and don’t mind dropped packets.  Using a static IP for our server so that we have something to aim at for our UDP packets.  Using Arduino and Ethernet shield to prototype and onto custome PCB once all kinks are worked out.

33. Project Schedule Longest Design Period 11 Days for Testing Technical Reference Manual 37 Days for Integration

34. Division of Labor TaskAdrianEvanJohnAlexDarnellCraig Program Mgmt & Documentation PSS Kinect Programming SPS N64 Translation SPS Ethernet Interface PP  Controller Programming SSP Audio/Video Signal Conversion SPS N64 Gameplay Action PPPPPP P = PrimaryS = Secondary

35. Advanced Parts List TOTAL ESTIMATED PROJECT COST = $1,074 *NOTE* – ALL EXPENSES PAID TRIP TO LAS VEGAS FOR “ELECTRONICS SEMINAR” NOT INCLUDED

36. Goals Primary: Control Nintendo 64 game exclusively via an XBOX Kinect Secondary: Have one N64 controller communicate with one N64 console over the internet Control the N64 game using either the Kinect or the N64 controller remotely Tertiary: Play a complete N64 game with 2 players at one location using the Kinect, and the other 2 players at a different location using N64 controllers Allow functionality with a variety of different controllers

37. Milestone 1 Objectives Kinect Translation Kinect translator sends simple gesture data to the Linux µController, in correct format Nintendo 64 Send properly timed Nintendo 64 analog signal from the µController to the N64 console Network Configuration Send data packets across the network to and from each user using the µController Audio/Video Convert analog input into a digital code

38. Milestone 2 Objectives Kinect  Determine ideal Kinect gestures for satisfactory user interface  Transmit verbal commands through Kinect Network Configuration  Send and receive N64 and audio/video signals across server Audio/Video  Output signal from server to television

39. Risks and Contingency Plan Risks  Syncing the N64 timing sequence with the Netduino microcontroller’s internal timer  Busy internet path/lost data packets  Delay between Kinect / N64 input and television output  Data corruption or potential injury resulting from falling or player collisions while playing with a motion sensitive controller Solutions  Use only N64 controllers (no Kinect) for game control  Leave out server / direct connection from microprocessor to N64  User plays video game at own risk!!

40. Questions???