WiFi Troubleshooting & Performance Monitoring Case Study : Troubleshooting wireless issues via Mojo Aware - Asvin

Case studies In this session, we will cover how Mojo Aware helped system administrators to identify following issues in corporate environment: Scenerio1: Wireless connection is lost when the wireless session times out in Windows 7 or Windows Server 2008 R2 Scenerio2: Issues with clients staying connected to an Access Point that has bad signal (Sticky Clients)

Case Study: Window 7 Clients loses WiFi connectivity Vertical: SME branch office Problem Wireless connection is lost when the wireless session times out in Windows 7 or Windows Server 2008 R2 Flag Mojo Aware flagged Invalid MIC failure on dashboard Cause Win 7 clients randomly introduce invalid MIC causing Mojo AP’s to reject M2 messages

Learning Objectives User impact of random client disconnection Problem Statement Detailed description of EAPOL 4-Way handshake functionality Symptoms Cause How Mojo Aware quickly identifies Invalid MIC failures and promptly alerts administrator

User Impact of the Problem Few minutes disruption of WiFi during office hours Issue occurs randomly on Windows 7 laptop Takes nearly 1 to 2 mins for the laptop to reconnect to wireless network

Debugging “Invalid MIC failure” Problem Statement Few minutes random disruption of WiFi Windows 7 clients connecting to wireless networks configured with WPA2 and session timeout may get disconnected during the key exchange after re- authentication Over the air, M1 and M2 packet retried several times causing delay in reconnection AP logs during problem:

EAPOL 4-way Message in detail (M1) Apply display filter EAPOL-Key messages using “eapol.keydes.type == 2” wireshark display filter. Message 1 (M1) Authenticator sends EAPOL-Key frame containing an ANonce(Authenticator nonce) to supplicant. With this information, supplicant have all  necessary input to generate PTK using pseudo-random function(PRF)

EAPOL 4-way Message in detail (M2) Message 2 (M2) Supplicant sends an EAPOL- Key frame containing SNonce to the Authenticator. Now authenticator has all the inputs to create PTK. Supplicant also sent RSN IE capabilities to Authenticator & MIC Authenticator derive PTK & validate the MIC as well.

EAPOL 4-way Message in detail (M3) Message 3 (M3) If necessary, Authenticator will derive GTK from GMK. Authenticator sends EAPOL- Key frame containing ANonce, RSN-IE & a MIC. GTK will be delivered (encrypted with PTK) to supplicant. Message to supplicant to install temporal keys.

EAPOL 4-way Message in detail (M4) Message 4 (M4) Supplicant sends final EAPOL-Key frame to authenticator to confirm temporal keys have been installed. From this point onwards data frame will be encrypted using PTK or GTK (depending upon unicast or multicast/broadcast frame)

Symptoms This issue occurs when a Windows 7-based computer is connected to a wireless network by using the WiFi- WPA2 protocol and the wireless access point (AP) starts a new exchange of WPA2 keys. In the four-way handshake, the Windows 7-based computer sends a Message 2 (M2) with an invalid message integrity check (MIC) Note This issue may occur every 12 hours or more frequently, and it takes one minute to regain the network connectivity. 

Cause This issue occurs because the WPA2 key context is not set correctly before the four-way handshake rekeys. Certain variables are not reset after the previous four-way handshake. This causes the secure bit to be set incorrectly and the stale Pairwise Transient Key (PTK) to be used to calculate the MIC in the M2 key messages. APs reject the M2 messages because of these errors.

Root cause analysis Microsoft confirmed this bug and fixed in Hotfix Win7 release. https://support.microsoft.com/en-in/kb/3094412 Win7 laptop required above hotfix upgrade.

How Mojo-aware quickly identifies Invalid MIC failures and promptly alerts administrator Mojo Aware pinpoint exact cause of failure and saves administrator time and effort for debugging Wireless capture is saved here!

Aware: Capture for corresponding failure Mojo Aware display exact packet capture during problem statement without applying any display filters!

Mojo Aware advantages Promotes actual cause of issue, not just client connectivity failure Administrator identifies complex EAPOL issues in single glance Save time and debugging effort without using real time wireless sniffer Wireless client failure logs preserved in cloud and can be looked and accessed any time.

Case Study2: Bad signal(Sticky clients) causing network slowness Vertical: SME branch office Problem Issues with clients staying connected to an Access Point that has bad signal (Sticky Clients) Flag Mojo Aware flagged sticky clients on dashboard Cause Clients still have good signal strength to far away AP so do not disassociate

Learning Objectives User impact of bad signal and sticky client Problem statement What is sticky client Symptoms Cause Resolution How Mojo-Aware quickly identifies sticky clients and promptly alerts administrators

User Impact of the Problem Far away clients contend with rising error rates due to the lower signals Overall wireless efficiency of the cell is reduced as clients wait longer than they should for a slower speed client to send its data Key to high performance WiFi network is airtime efficiency Even a small number of sticky clients, using sub- optimal speeds, can very quickly drag down the performance of Wi-Fi network

Debugging “Bad signal and network slowness” Problem Statement Customer reported “Overall network slowness ” Basic file copy operation within the network takes long time to transfer.

Sticky clients and roaming decision Wireless clients tend to hang on to the original access point they associated with, rather than moving to a nearby AP that would generally be a better choice for them. Roaming Decision is a client decision, not a network decision AP don’t tell client when to roam – the network has to respect the wishes and behavior of client devices.

Symptoms Identifying Sticky clients - behavior Do not probe on other channels Probe infrequently Remain associated to an AP even through better Aps are available Transmit on low PHY rate consuming more air

Sticky client end user impact

Sniff capture with sticky client in network Low Data rate Low RSSI

Mojo Aware Mojo Aware quickly identifies sticky clients and promptly alerts administrators to take appropriate action.

Resolution Enable Smart Steering Enable Min Association RSSI Disassociate “Sticky clients” Prevent them from re-associating to the AP Encourage/Force roam to better AP Configured per SSID (Enable/Disable) Enable Min Association RSSI RSSI Threshold Reduce the number of probe/assoc response Prevent clients with RSSI below the threshold from associating

Sniff Capture-After Enabling smart steering Deauth after enabling smart steering

Mojo Aware - After Enabling Smart steering

Mojo Aware Advantages Warns the administrator about network slowness (eg. Sticky client) Ability to quickly identify the WLAN issues Ability to quickly detect total no of sticky or bad clients in the network Ability to isolate problematic clients with complete details include packet captures and system logs for further analysis.

Thank You

Backup slides

EAPOL 4-Way Handshake functionality Process: The AP sends a nonce-value to the STA (ANonce). The client now has all the attributes to construct the PTK. The STA sends its own nonce-value (SNonce) to the AP together with a MIC, including authentication, which is really a Message Authentication and Integrity Code: (MAIC). The AP sends the GTK and a sequence number together with another MIC. This sequence number will be used in the next multicast or broadcast frame, so that the receiving STA can perform basic replay detection. The STA sends a confirmation to the AP.

What is session timeout? Session timeout means that authenticated user session expires in 1800 seconds based on implementation, it is not an activity or idle timeout.  So depending on authentication method, this could cause client to disconnect.

Root cause analysis Microsoft confirmed this bug and fixed in Hotfix Win7 release. https://support.microsoft.com/en-in/kb/3094412 Win7 laptop required above hotfix upgrade. Per Microsoft, this issue can also be mitigated by reducing the EAPOL key retransmission timeout. The issue was first seen with timeout value of 3ms. When reducing this value to 1msec the issue was fixed. Note: Do be aware that reducing this value might negatively impact key negotiations with some very old and slow clients.

How Mojo-aware quickly identifies Invalid MIC failures and promptly alerts administrator Mojo-aware pinpoint exact cause of failure and saves administrator time and effort for debugging