1. Authentication services for big Wi-Fi Eriks Rugelis, Krzysztof Adamski Network Development team York University, UIT

2. YorkU context Population – 47K undergrad students, 6K grad students, 7K faculty and staff Physical infrastructure – 2 primary campuses, 6 satellite locations – Total of 70 academic + admin buildings, approx. 600K sq.m. – Plus… total of 16 residence buildings, 3400 beds

3. WLAN infrastructure at YorkU Academic and administration buildings – Cisco AP1121, AP3602i, AP3702i, AP3702e, AP3702p Residence buildings – Cisco AP1252, AP1142, AP702W, AP3602i Total of 14 Cisco CT5508 WLCs – One running AireOS 7.0.252.0 for legacy APs – Balance running AireOS 8.0.120.13

4. WLAN Services at YorkU AirYorkPLUS – The primary SSID for WLAN service to YorkU users – 802.1x authentication, PEAP/MSCHAPv2 – 27K peak concurrent client devices in 2015-16 eduroam – Available to both guests and local users – 802.1x authentication, PEAP/MSCHAPv2 AirYork – Captive portal service – Hidden SSID, used for guest professional development users (difficult onboarding SSID with 802.1x credentials) Wi-Fi Info – Open SSID used to provide online help information to Wi-Fi users

5. Identity management infrastructure Passport York – Home grown identity management system – Is the administrative source-of-truth for identity for all centrally-supported IT services at YorkU and many Faculty/Department supported services as well – Accessed via own API or via LDAP – Stores unique identifier code, userid, password, access rights attributes (PYflags, PYgroups)

6. Authentication/Authorization back- end infrastructure Active Directory – YORKU domain – Root of enterprise identity tree – Used by WLAN services as backend authentication and authorization DB – Synchronized identity and passwords with PY – Receives group ID feed from PY – 5 servers (SunFire 4170), 16 cores, 32 GB RAM

7. RADIUS server YorkU selected PacketFence What is it? – Packaged FreeRADIUS + accounting database + NAC – FOSS project developed and maintained by Inverse.ca – Active community forum – Paid telephone support and custom SW development available from Inverse

8. Packetfence Components PF front-end – Responds to RADIUS transactions from network authorization / authentication / accounting – Uses winbind to call Active Directory for authentication – Uses LDAP to call Active Directory LDAP for authorization – May be used as captive portal server – May be used to probe and evaluate security posture of Wi- Fi client and apply NAC policy PF database – Stores user credentials (user name, person name, MAC address of device, session login time, data volume accounting, history of policy violations)

10. The challenge Poor performance of 802.1x authentication service for WLAN service has high impact on end- user perception of overall WLAN service – bad for student satisfaction: – Plan for performance – Plan for scale appropriate to your environment – Monitor performance and capacity metrics 802.1x authentication transaction rates in higher education have high peak-to-average ratio

11. Total RADIUS requests/sec (x 0.5)

12. YorkU scale-out configuration Two RADIUS load-balancer clusters (pfclust10, pfclust20), one cluster per VM host 6x PF guests per cluster – 8 vCores, 16 GB RAM 1 st node acts as RADIUS LB only, nodes 2 thru 6 process the transactions Odd numbered WLC’s prefer pfclust10 first Even numbered WLC’s prefer pfclust20 first

14. YorkU scale-out config - continued If pfclust11 fails then RADIUS LB function passes to next sequential pfclust1x guest; LB function returns to lowest numbered pfclust guest whenever it is ready to resume service Any single pfclust* guest has capacity to handle up to 300 to 500 transactions per second Pfdb11 is a single VM guest; setup for auto- restart by VMware HA We have not yet observed a transaction rate limit on pfdb11 – but pay attention to DB connected thread count!!

15. pfdb11: thread stats

16. Other PF deployments tried Active-Standby clusters (PF/RADIUS, DB) Pair of Active-Active PF/RADIUS servers + Active-Standby DB cluster Long list of Active PF/RADIUS servers + single pfdb11 with VMware HA

18. Active-Standby clusters Pro: Resilient/fault-tolerant authentication service Con: Under-utilized idle capacity in Standby servers DB cluster synchronization uses a unique and unfamiliar (for YorkU) clustering technology Difficult to upgrade SW without impacting entire authentication service Limited to <500 RADIUS transactions/second

20. Active pair of PF/RADIUS Pro: Steady-state capacity of 2x <500 RADIUS transactions/sec. Con: Fail-mode capacity is still only <500 transactions/sec (same as previous model) DB synch technology is still unfamiliar to YorkU

22. Active x6 PF/RADIUS servers Pro: Steady-state capacity 6x <500 RADIUS transactions/sec DB resiliency converted to familiar VMware HA mechanism Con: Cisco WLC will send workload of a failed RADIUS server to the next sequential server – if total workload >500 transactions/sec then that server will also fail – congestive collapse is likely during peak loads!!

23. Be mindful of… Peak projected RADIUS transaction rate per WLC (or equivalent) Does WLC itself have sufficient CPU capacity to handle max. projected workload of RADIUS transactions? How does WLC (e.g. Cisco, Aruba, HP, Meru, etc.) distribute load when configured to use more than one RADIUS server? How does WLC respond when preferred RADIUS server goes offline? Returns online? Transaction rate capacity of back-end authentication DB (e.g. Active Directory) Measuring/reporting transaction rates and latencies of all authentication system components

24. Future refinements Move RADIUS LB function to dedicated (Layer 3 – Layer 7) LB appliance Introduce LB appliance between pfclust* and ADLDAP to mitigate service outages of individual AD-DC’s (e.g. during monthly patching)

25. References PacketFence http://packetfence.org/ FreeRADIUS http://freeradius.org/

26. Thank you!

27. Questions? Eriks Rugelis eriks@yorku.ca +1 416 736 5756 Krzysztof Adamski kadamski@yorku.ca +1 416 736 2100 x.22675