1. AN Surface Detector Communication Team 27 May, 2010 1 RT2010 - Lisboa Hard Real-Time Wireless Communication in the Northern Pierre Auger Observatory Roger Kieckhafer (rmkieckh@mtu.edu) for the Pierre Auger Collaboration

2. 27 May, 2010RT2010 - Lisboa 2 Auger North and South  From the beginning: two sites were planned  Northern and Southern Hemispheres  Auger North:  Planning Stages  Lamar, Colorado USA  Auger South:  Fully Operational  Malargue, Argentina

3. 27 May, 2010RT2010 - Lisboa 3 Surface Detector (SD) Array Timestamps + other data Shower of secondary particles H +H + Flourescence Detector (FD) Telescopes Observatory Functional Overview

4. 27 May, 2010RT2010 - Lisboa 4 SD Wireless – Real-Time Sequence SD Station (in the field) CDAS (Observatory Campus) Record and filter timestamps T2 Trigger Message (Timestamps of promising events) Correlate Timestamps Across Stations T3 Trigger Message (Req. data on correlated timestamps) Access Data on requested timestamps T3 Response Message Stream (Return data on req. timestamps) Conduct further analysis

5. 27 May, 2010RT2010 - Lisboa 5 Auger North vs South.  Auger North puts greater demands on the wireless net  Auger North will be much larger  Auger North will have higher data rates  Auger North will have roughly the same real-time deadlines

6. 27 May, 2010RT2010 - Lisboa 6 Auger North Site Auger North Footprint Auger South Footprint

7. 27 May, 2010RT2010 - Lisboa 7 Auger South SD Communication  Wide area wireless net  Microwave towers form a communication backbone  4 Collector Towers on periphery of array  1 Destination Tower at Malargue campus  1-hop station-to-tower link for each station  Remarkably flat topography in the interior  Sizable hills on the periphery of the array  Easy to give each station a clear line-of-sight to a tower

8. 27 May, 2010RT2010 - Lisboa 8 Auger North Communication Problem  Array is 7 x Bigger than South  Need > 30 towers @ 45 m to stay within radio range  Terrain is Rougher than South  No convenient peripheral hills  Many small internal hills & ridges  Can not get reliable line-of-sight over that terrain  Bottom line:  1-hop Station-to-Tower comm. does not work  >10% of stations incommunicado Springfield Lamar

9. 27 May, 2010RT2010 - Lisboa 9 Only Viable Option: Peer-to-Peer WSN  Auger North uses a new paradigm called “WAHREN”  WA  Wireless Architecture  Infrastructure Nodes (INs) = the dominant feature  Non-Infrastructure Nodes (NINs) = optional “guest” nodes  HR  Hard Real-time  Time-Bounded  subject to hard real-time deadlines  Deterministic  not subject to stochastic-timing  EN  Embedded Networks  Fault-Tolerant  Fully Redundant Comp. & Comm.  Fully Distributed  No leaders, coordinators, masters

10. 27 May, 2010RT2010 - Lisboa 10 WAHREN Interconnection Topology  Uses 2 nd Order Power chains (“Braided Chain”)  Graph Topology:  Start with a basic chain – nearest neighbor comm only  Extend range to reach second-nearest neighbors  Useful Physical Realization = 2D Triangular Mesh 12345678910 1 2 3 4 5 6 7 8 9 

11. 27 May, 2010RT2010 - Lisboa 11  Our Target Infrastructures are not always straight lines  Gentle curves are no problem  But what about sharp corners?  Existence of Mobius fold is transparent to the topology 7 8 9 10 7’ 8’ 9’ 10’ Turning Corners 1 2 3 4 5 6 Mobius Fold Must be Prevented by Protocol

12. 27 May, 2010RT2010 - Lisboa 12 Hybrid TDMA/CSMA MAC Protocol  Hybrid Window Combines TDMA & CSMA slots  Enough TDMA slots for all INs within interference range  Enough CSMA slots for expected/desired number of NINs  Auger North-Specific Comm Window  8 TDMA slots – for neighboring Infrastructure Nodes  Based on predicted interference range, and  Alternating use of 2 RF channels  1 CSMA slot – expecting few non-infrastructure nodes NIN Trans CSMA Window IN Transmissions TDMA

13. 27 May, 2010RT2010 - Lisboa 13 Systolic Broadcast Scheduling Protocol  Unidirectional Single-Source Broadcast:  Window 0: Node 0 originates a message  Window w: Node w forwards node 0’s message  Redundancy: Node Red = Path Red = Time Red = 2  Unidirectional Multi-Source Broadcast  Window 0: All nodes originate a message  Window w: Node k forwards node (k-w)’s message 0 Win-2 Win-1 Win-0 Win-4 Win-5 Win-3 1 264 35

14. 27 May, 2010RT2010 - Lisboa 14 Adapting WAHREN to Auger Topology  Array is partitioned into “service areas”  Each served by one Concentrator Station  Each Service Area is partitioned into “sectors”  Sectors can be triangular, rectangular, or quite amorphous Service Area Sector

15. 27 May, 2010RT2010 - Lisboa 15 Comm Sequence in One Sector  Auger North SD is a  2 mi square array tilted 45 degrees  Any adjacent pair of E-W rows or N-S columns naturally forms a 2 nd order power chain  Can easily be organized into Backbones & Side Chains  Linked at Mobius Folds Side Chains Backbone Concentrator Station Fiber to Campus PC Host

16. 27 May, 2010RT2010 - Lisboa 16 System Throughput  Can Meet all required deadlines:  With up-to 128 stations / sector  4,400 stations  35 sectors  8 sectors/concent.  5 concents.  IF they are all centrally located.  Current Proposal is to put concentrators at the 5 FDs  Power, fiber & bldgs already exist  But, some FDs are on Periphery  They serve fewer sectors  May need 6-7 Concentrators  5 at FDs + 1-2 standalone  That is still much cheaper than 30 tall towers

17. 27 May, 2010RT2010 - Lisboa 17  Being built Near Lamar, CO  Goal: to test the riskier new technologies  Station Configuration  10 fully functional stations  10 comms-only stations  1 concentrator  Array Layout  1 backbone (A-J)  1 side chain (K-O)  6 off-grid “infill” stations (L-T) Research & Development Array (RDA) Lamar

18. 27 May, 2010RT2010 - Lisboa 18 Summary  WAHREN paradigm is well suited to Auger North:  Power Chain Architecture  path & node redundancy  Hybrid TDMA/CSMA MAC  real-time stability  Systolic Broadcast Sched  time redundancy & efficiency  Deadlines and throughput can be satisfied:  With a handful of concentrators  Much more cheaply than tall towers  Some items we had to skip today:  Formal validation & verification of the protocol – done  Detailed Markov Reliability modeling of a sector – done  Multimodal Performability modeling of a sector – in progress  Station radio hardware/software development – in progress

19. AN Surface Detector Communication Team 27 May, 2010 19 RT2010 - Lisboa Questions?