1. Touch Seen Visual Board Game Platform Computer Vision Workshop, Fall 2010 Eldad Zinger, Oren Kishon, Ehud Fisher, Allon Freedman

2. Agenda Overview Architecture – High Level – Server – Client – Camera Module Network Protocol Game Plugin Platform Client-Camera Interaction Game move detection algorithm Demonstration 2Monday, October 19, 2015Touch Seen

3. Overview 3Monday, October 19, 2015Touch Seen

4. Application framework for playing board games over a computer network using a real physical board Moves are detected by a camera and transmitted over the network Provides a simple API for development of 3 rd party games which can be uploaded to the platform in the form of plugins 4Monday, October 19, 2015Touch Seen

5. Features Input using camera or any other standard input (keyboard, mouse etc.) Multiple clients connected simultaneously Multiple games played simultaneously All currently connected users displayed in client Option to challenge another user to a game Textual chat between players 5Monday, October 19, 2015Touch Seen

6. Features Open API for game development Pluggable games Simple text based protocol Network based on ActiveMQ which supports multi language clients 6Monday, October 19, 2015Touch Seen

7. Architecture 7Monday, October 19, 2015Touch Seen

8. High Level 8 Server Client Monday, October 19, 2015Touch Seen

9. Server Two logical functions: Messaging – distributing Move and Chat messages between users User & Game Management: – User DB – User presence – Game state Monday, October 19, 20159Touch Seen

10. Server Messaging server based on Apache ActiveMQ ActiveMQ – powerful open source Messaging Server supporting JMS 1.1, J2EE 1.4, multiple cross language clients and protocols and more http://activemq.apache.org/ An ActiveMQ server is run as an embedded broker within the Touch Seen Server Monday, October 19, 201510Touch Seen

11. Server JMS A specification that describes a common way for Java programs to create, send, receive and read distributed enterprise messages loosely coupled communication Synchronous & Asynchronous messaging Reliable delivery – A message is guaranteed to be delivered once and only once Outside the specification – Security services – Management services Monday, October 19, 201511Touch Seen

12. Server JMS Messaging Domains Point-to-Point (PTP) Message Queues - each message has only one consumer Publish-Subscribe systems Message Topics - each message has multiple consumers Monday, October 19, 201512Touch Seen

13. Server JMS Message Consumption Synchronous – A subscriber or a receiver explicitly fetches the message from the destination by calling the receive method – The receive method can block until a message arrives or can time out if a message does not arrive within a specified time limit Asynchronous – A client can register a message listener with a consumer – Whenever a message arrives at the destination, the JMS provider delivers the message by calling the listener's onMessage() method All messages in Touch Seen use the asynchronous method Monday, October 19, 201513Touch Seen

14. Server JMS Messaging Domains – Queues Queue sends Msg acknowledges consumes Msg Client1 Client2 Monday, October 19, 201514Touch Seen

15. Server JMS Messaging Domains – Topics publishes subscribes Msg delivers Client1 Client2 Client3 Topic Monday, October 19, 201515Touch Seen

16. Server JMS Model Connection creates Msg receives from sends to Connection Factory Connection Factory Message Consumer Message Consumer Session Message Producer Message Producer creates Destination Monday, October 19, 201516Touch Seen

17. Server – Class Diagram Games Users Monday, October 19, 201517Touch Seen

18. Server Threads ServerController & NetworkController each operate in separate threads This ensures that server operations never block the network thread so it can always be available for sending/receiving messages Management The server has no UI Management is done using JMX (e.g. with jconsole) utilizing both APIs provided by ActiveMQ and custom API’s designed specifically for this server Monday, October 19, 201518Touch Seen

19. Client Strict modular design based on MVC Comprised of several independent modules Each module has a controller class ClientController acts as central controller for entire application All communication to and from a module is done through its controller The central controller connects to each of the module controller (module controllers don’t interact directly) Monday, October 19, 201519Touch Seen

20. Client The advantages of this design: – Modules are entirely decoupled from each other – Each module can be developed and tested independently – Communication between modules is done in one place – easy to maintain and debug – Modules can easily be replaced without affecting the rest of the application – only the central controller “knows” all the modules Monday, October 19, 201520Touch Seen

21. Client – Class Diagram Monday, October 19, 201521Touch Seen

22. Threads The ClientController, NetworkController and CameraController all operate in separate threads This ensures that the Network and Camera threads are never blocked by other modules within the client All Objects in the View operate within Swing’s EventDispatcher thread The ClientController and Model also operate within the EventDispatcher thread to avoid synchronization problems with the View Monday, October 19, 201522Touch Seen

23. 23 Network Protocols Monday, October 19, 2015Touch Seen

24. Login Queue Login username password success notify online request players players Players Request Queue Players Topic Monday, October 19, 201524Touch Seen

25. Challenge Queue Game challenge accept notify challenge accept notify Players Topic New Game Monday, October 19, 201525Touch Seen

26. Game Topic Game move Game Playing Game Chat Topic message Monday, October 19, 201526Touch Seen

27. Game Topic Game quit Game End Players Topic notify online Monday, October 19, 201527Touch Seen

28. Logout Queue Logout username notify offline Monday, October 19, 201528Touch Seen

29. 29 Game Plugin Platform Monday, October 19, 2015Touch Seen

30. Game SDK Games are developed using a provided SDK Developers must implement only two abstract classes: – GameInfo Flyweight which acts as a factory for creating instances of the game Provides common properties: name, icon etc. – Game Represents the game itself Monday, October 19, 201530Touch Seen

31. Game SDK The server and client controllers interact with the game using the Game class It’s recommended (although optional) that this class include the game’s logic Game must provide a JPanel which acts as the game’s UI Monday, October 19, 201531Touch Seen

32. Game Installation A game should be packaged as a.jar file and placed within the lib/games folder of the client and server A configuration file must be edited with the fully qualified name of the GameInfo class At startup, the client and server use a custom ClassLoader to load all the games dynamically A future release could include automatic deployment of new games to clients and installation/uninstallation without restart A simple (yet incomplete) example of Chess is included Monday, October 19, 201532Touch Seen

33. 33 Client-Camera Interaction Monday, October 19, 2015Touch Seen

34. Client-Camera Interaction The Touch Seen desktop client is written in Java The camera module is based on OpenCV and written in C Interaction between the two is done via a local TCP socket The protocol is a simple text based protocol which simplifies debugging When the user starts a game, the client starts the camera process and waits for a socket connection for a specified timeout (externally configurable) If the initialization sequence completes successfully the camera can be used to play Alternative input devices (e.g. a mouse) can be used if errors occur during the initialization sequence or if moves aren’t detected successfully Monday, October 19, 201534Touch Seen

35. 35 Game Move Detection Algorithm Monday, October 19, 2015Touch Seen

36. Detection algorithm Image subtraction: Monday, October 19, 201536Touch Seen

37. Detection algorithm (1) Initialization: – Detect chessboard – Find prospective transformation (3 x 3): rectify – Rotate coords to user’s point of view For each new image captured, do: – Apply prospective transformation (rectify) – Subtract image from previous image – to detect who moved / captured by opponent player – Subtract image from empty board image – to detect who actually cleared his location Monday, October 19, 201537Touch Seen

38. Detection algorithm (2) Dealing with noise Noise: intensity only gets higher values Filter1: image is a MIN of 20 frame captures Filter2: Time-adaptive noise threshold: – Capture two images each time, 1 sec apart – Subtract to get noise-only image – Calculate average value for square – Find max among squares (empirically ~5) – Thresh = max + const (=2, heuristics) Monday, October 19, 201538Touch Seen

39. Detection algorithm (3) Square change detection Less sensitive change detection: – Among squares who passed threshold, take max. – More sensitive change detection: – For each square, sum value of pixels, regardless of threshold. Take Max. – Monday, October 19, 201539Touch Seen

40. Detection algorithm (4) Evolution of development At first – simple subtraction algorithm (gray): hard-coded thresh, one background image Use RGB space – track each piece by its color. Each piece physically had a unique color Use HSV space – not good when dealing with B&W backgrounds Back to gray – smarter algorithm: min filter, multiple backgrounds, adaptive threshold Monday, October 19, 201540Touch Seen

41. Demonstration In addition to the live demonstration we have provided the following documentation: System Requirements Specification describing planned features and architecture This presentation which provides the final design implemented User Manual Complete Javadoc for the Client, Server and Game Development SDK including UML diagrams (embedded within the Javadoc) Fully documented code Source code for Client, Server, SDK, Camera module and example game (Chess) Installation instructions Monday, October 19, 201541Touch Seen

42. Thank you… 42Monday, October 19, 2015Touch Seen