OS: Raspberry Pi OS Bookworm Lite (64 Bit)
Platform: RPi 3 B+
Issue: after several hours of operation, all of the Pi3 on the network lose wifi connectivity (see log below). Pi4 on 5GHz are not impacted. The disconnect always occurs at 38 minutes past the hour. The problem does not manifest every hour, but reliably occurs at 38 minute past. Typically the pi disconnects with the following error:. NetworkManager attempts to reconnect several times, but eventually gives up.
Reason 16 appears to be "Group key handshake timeout".
Restarting the NetworkManager service will bring the connection back up. The connection will inevitably fail again after a few hours at 38 minutes past the hour.
All of the Pi on the network have fixed IPs assigned by the DHCP server. The DHCP renewal interval is set to 1 day. I can reproduce this on any Pi3 running bookworm 64. I've tried multiple Pi3 with a variety of power supplies to rule out brown-out problems.
I suspect this has something to do with Group Rekey Interval which is set at 3600 seconds (1 Hour), but I don't have a good way to monitor that on the AP to see when that's happening. I've recently set the interval to 1200 seconds to see if I can get the failures to occur at a time other than 38 minutes past the hour, but do not yet have any data on that.
Network Manager Configuration:Relevant Log Entries
Platform: RPi 3 B+
Issue: after several hours of operation, all of the Pi3 on the network lose wifi connectivity (see log below). Pi4 on 5GHz are not impacted. The disconnect always occurs at 38 minutes past the hour. The problem does not manifest every hour, but reliably occurs at 38 minute past. Typically the pi disconnects with the following error:
Code:
wpa_supplicant.service wpa_supplicant[529] STATS wlan0: CTRL-EVENT-DISCONNECTED bssid=e0:63:da:18:43:db reason=16Reason 16 appears to be "Group key handshake timeout".
Restarting the NetworkManager service will bring the connection back up. The connection will inevitably fail again after a few hours at 38 minutes past the hour.
Code:
sudo systemctl restart NetworkManagerAll of the Pi on the network have fixed IPs assigned by the DHCP server. The DHCP renewal interval is set to 1 day. I can reproduce this on any Pi3 running bookworm 64. I've tried multiple Pi3 with a variety of power supplies to rule out brown-out problems.
I suspect this has something to do with Group Rekey Interval which is set at 3600 seconds (1 Hour), but I don't have a good way to monitor that on the AP to see when that's happening. I've recently set the interval to 1200 seconds to see if I can get the failures to occur at a time other than 38 minutes past the hour, but do not yet have any data on that.
Network Manager Configuration:
Code:
[connection]id=preconfigureduuid=0b1cae9a-73f1-4cee-9531-0dc6a665b3eetype=wifi[wifi]mode=infrastructuressid=Shackleton Nhidden=false[ipv4]method=auto[ipv6]addr-gen-mode=defaultmethod=auto[proxy][wifi-security]key-mgmt=wpa-pskpsk=REDACTEDCode:
│Aug 05 15:38:02 office-paperpi wpa_supplicant[571]: wlan0: CTRL-EVENT-DISCONNECTED bssid=e0:63:da:18:43:db reason=16 ││Aug 05 15:38:02 office-paperpi wpa_supplicant[571]: wlan0: CTRL-EVENT-REGDOM-CHANGE init=CORE type=WORLD ││Aug 05 15:38:02 office-paperpi wpa_supplicant[571]: wlan0: CTRL-EVENT-REGDOM-CHANGE init=USER type=COUNTRY alpha2=NL ││Aug 05 15:38:02 office-paperpi NetworkManager[568]: <info> [1722865082.3997] device (wlan0): supplicant interface state: completed -> disconnected ││Aug 05 15:38:02 office-paperpi NetworkManager[568]: <info> [1722865082.3999] device (p2p-dev-wlan0): supplicant management interface state: completed -> disconnected ││Aug 05 15:38:02 office-paperpi NetworkManager[568]: <info> [1722865082.4916] device (wlan0): supplicant interface state: disconnected -> scanning ┤│Aug 05 15:38:02 office-paperpi NetworkManager[568]: <info> [1722865082.4917] device (p2p-dev-wlan0): supplicant management interface state: disconnected -> scanning ┤│Aug 05 15:38:03 office-paperpi wpa_supplicant[571]: wlan0: Trying to associate with SSID 'Shackleton N' ││Aug 05 15:38:03 office-paperpi NetworkManager[568]: <info> [1722865083.0925] device (wlan0): supplicant interface state: scanning -> associating ││Aug 05 15:38:03 office-paperpi NetworkManager[568]: <info> [1722865083.0927] device (p2p-dev-wlan0): supplicant management interface state: scanning -> associating ┤│Aug 05 15:38:07 office-paperpi wpa_supplicant[571]: wlan0: CTRL-EVENT-ASSOC-REJECT bssid=e0:63:da:18:43:db status_code=16 ││Aug 05 15:38:07 office-paperpi NetworkManager[568]: <info> [1722865087.1670] device (wlan0): supplicant interface state: associating -> disconnected ││Aug 05 15:38:07 office-paperpi NetworkManager[568]: <info> [1722865087.1672] device (p2p-dev-wlan0): supplicant management interface state: associating -> disconnected ││Aug 05 15:38:07 office-paperpi NetworkManager[568]: <info> [1722865087.6670] device (wlan0): supplicant interface state: disconnected -> scanning ││Aug 05 15:38:07 office-paperpi NetworkManager[568]: <info> [1722865087.6671] device (p2p-dev-wlan0): supplicant management interface state: disconnected -> scanning ││Aug 05 15:38:08 office-paperpi wpa_supplicant[571]: wlan0: Trying to associate with SSID 'Shackleton N' ││Aug 05 15:38:08 office-paperpi NetworkManager[568]: <info> [1722865088.5535] device (wlan0): supplicant interface state: scanning -> associating ││Aug 05 15:38:08 office-paperpi NetworkManager[568]: <info> [1722865088.5536] device (p2p-dev-wlan0): supplicant management interface state: scanning -> associating ││Aug 05 15:38:08 office-paperpi wpa_supplicant[571]: wlan0: CTRL-EVENT-ASSOC-REJECT bssid=e0:63:da:18:43:db status_code=16 ││Aug 05 15:38:08 office-paperpi NetworkManager[568]: <info> [1722865088.6004] device (wlan0): supplicant interface state: associating -> disconnected ││Aug 05 15:38:08 office-paperpi NetworkManager[568]: <info> [1722865088.6006] device (p2p-dev-wlan0): supplicant management interface state: associating -> disconnected ││Aug 05 15:38:09 office-paperpi NetworkManager[568]: <info> [1722865089.5981] device (wlan0): supplicant interface state: disconnected -> scanning ││Aug 05 15:38:09 office-paperpi NetworkManager[568]: <info> [1722865089.5983] device (p2p-dev-wlan0): supplicant management interface state: disconnected -> scanning ││Aug 05 15:38:10 office-paperpi wpa_supplicant[571]: wlan0: Trying to associate with SSID 'Shackleton N' ││Aug 05 15:38:10 office-paperpi NetworkManager[568]: <info> [1722865090.4863] device (wlan0): supplicant interface state: scanning -> associating ┤│Aug 05 15:38:10 office-paperpi NetworkManager[568]: <info> [1722865090.4866] device (p2p-dev-wlan0): supplicant management interface state: scanning -> associating ││Aug 05 15:38:10 office-paperpi wpa_supplicant[571]: wlan0: CTRL-EVENT-ASSOC-REJECT bssid=e0:63:da:18:43:db status_code=16 ││Aug 05 15:38:10 office-paperpi wpa_supplicant[571]: wlan0: CTRL-EVENT-SSID-TEMP-DISABLED id=0 ssid="Shackleton N" auth_failures=1 duration=10 reason=CONN_FAILED ││Aug 05 15:38:10 office-paperpi NetworkManager[568]: <info> [1722865090.5111] device (wlan0): supplicant interface state: associating -> disconnected ││Aug 05 15:38:10 office-paperpi NetworkManager[568]: <info> [1722865090.5113] device (p2p-dev-wlan0): supplicant management interface state: associating -> disconnected ┤│Aug 05 15:38:15 office-paperpi NetworkManager[568]: <info> [1722865095.5142] device (wlan0): supplicant interface state: disconnected -> scanning ┤│Aug 05 15:38:15 office-paperpi NetworkManager[568]: <info> [1722865095.5143] device (p2p-dev-wlan0): supplicant management interface state: disconnected -> scanning ┤│Aug 05 15:38:17 office-paperpi NetworkManager[568]: <warn> [1722865097.4914] device (wlan0): link timed out. ┤│Aug 05 15:38:17 office-paperpi NetworkManager[568]: <info> [1722865097.4915] device (wlan0): state change: activated -> failed (reason 'supplicant-timeout', sys-iface-state: 'managed') ┤│Aug 05 15:38:17 office-paperpi NetworkManager[568]: <info> [1722865097.4931] manager: NetworkManager state is now DISCONNECTED ││Aug 05 15:38:17 office-paperpi NetworkManager[568]: <warn> [1722865097.4948] device (wlan0): Activation: failed for connection 'preconfigured' ││Aug 05 15:38:17 office-paperpi dbus-daemon[499]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.8' (uid=0 pid=568 c││Aug 05 15:38:17 office-paperpi NetworkManager[568]: <info> [1722865097.4987] device (wlan0): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed') ││Aug 05 15:38:17 office-paperpi avahi-daemon[496]: Withdrawing address record for fe80::df06:c0ef:3153:c67d on wlan0. ││Aug 05 15:38:17 office-paperpi avahi-daemon[496]: Leaving mDNS multicast group on interface wlan0.IPv6 with address fe80::df06:c0ef:3153:c67d. ││Aug 05 15:38:17 office-paperpi avahi-daemon[496]: Interface wlan0.IPv6 no longer relevant for mDNS. ││Aug 05 15:38:17 office-paperpi NetworkManager[568]: <info> [1722865097.5220] dhcp4 (wlan0): canceled DHCP transaction ││Aug 05 15:38:17 office-paperpi NetworkManager[568]: <info> [1722865097.5221] dhcp4 (wlan0): activation: beginning transaction (timeout in 45 seconds) ┤│Aug 05 15:38:17 office-paperpi NetworkManager[568]: <info> [1722865097.5223] dhcp4 (wlan0): state changed no lease ┤│Aug 05 15:38:17 office-paperpi systemd[1]: Starting NetworkManager-dispatcher.service - Network Manager Script Dispatcher Service... ││Aug 05 15:38:17 office-paperpi avahi-daemon[496]: Withdrawing address record for 192.168.1.127 on wlan0. ┤│Aug 05 15:38:17 office-paperpi avahi-daemon[496]: Leaving mDNS multicast group on interface wlan0.IPv4 with address 192.168.1.127. ││Aug 05 15:38:17 office-paperpi avahi-daemon[496]: Interface wlan0.IPv4 no longer relevant for mDNS. ││Aug 05 15:38:17 office-paperpi dbus-daemon[499]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher' ││Aug 05 15:38:17 office-paperpi systemd[1]: Started NetworkManager-dispatcher.service - Network Manager Script Dispatcher Service. ││Aug 05 15:38:17 office-paperpi kernel: warning: `iwconfig' uses wireless extensions which will stop working for Wi-Fi 7 hardware; use nl80211 ││Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3642] device (wlan0): supplicant interface state: scanning -> disconnected ││Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3643] device (p2p-dev-wlan0): supplicant management interface state: scanning -> disconnected ┤│Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3780] policy: auto-activating connection 'preconfigured' (a0aa67dd-3307-4e1c-8cb5-280e8d9469f4) ┤│Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3798] device (wlan0): Activation: starting connection 'preconfigured' (a0aa67dd-3307-4e1c-8cb5-280e8d9469f4) ││Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3802] device (wlan0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed') ││Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3814] manager: NetworkManager state is now CONNECTING ││Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3825] device (wlan0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed') ││Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3837] device (wlan0): Activation: (wifi) access point 'preconfigured' has security, but secrets are required. ┤│Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3838] device (wlan0): state change: config -> need-auth (reason 'none', sys-iface-state: 'managed') ││Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3882] device (wlan0): state change: need-auth -> prepare (reason 'none', sys-iface-state: 'managed') ││Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3895] device (wlan0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed') ││Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3915] device (wlan0): Activation: (wifi) connection 'preconfigured' has security, and secrets exist. No new secrets needed. ││Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3918] Config: added 'ssid' value 'Shackleton N' ││Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3919] Config: added 'scan_ssid' value '1' ││Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3919] Config: added 'bgscan' value 'simple:30:-70:86400' ││Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3920] Config: added 'key_mgmt' value 'WPA-PSK WPA-PSK-SHA256 FT-PSK' ┤│Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.3920] Config: added 'psk' value '<hidden>' ┤│Aug 05 15:38:18 office-paperpi wpa_supplicant[571]: wlan0: Trying to associate with SSID 'Shackleton N' ││Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.4100] device (wlan0): supplicant interface state: disconnected -> associating ││Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.4102] device (p2p-dev-wlan0): supplicant management interface state: disconnected -> associating ││Aug 05 15:38:18 office-paperpi wpa_supplicant[571]: wlan0: CTRL-EVENT-ASSOC-REJECT bssid=e0:63:da:18:43:db status_code=16 ┤│Aug 05 15:38:18 office-paperpi wpa_supplicant[571]: wlan0: CTRL-EVENT-SSID-TEMP-DISABLED id=0 ssid="Shackleton N" auth_failures=1 duration=10 reason=CONN_FAILED ┤│Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.4467] device (wlan0): supplicant interface state: associating -> disconnected ┤│Aug 05 15:38:18 office-paperpi NetworkManager[568]: <info> [1722865098.4468] device (p2p-dev-wlan0): supplicant management interface state: associating -> disconnected ┤│Aug 05 15:38:27 office-paperpi systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully. ┤│Aug 05 15:38:28 office-paperpi NetworkManager[568]: <info> [1722865108.4546] device (wlan0): supplicant interface state: disconnected -> scanning ┤│Aug 05 15:38:28 office-paperpi NetworkManager[568]: <info> [1722865108.4547] device (p2p-dev-wlan0): supplicant management interface state: disconnected -> scanning ┤│Aug 05 15:38:29 office-paperpi wpa_supplicant[571]: wlan0: CTRL-EVENT-SSID-REENABLED id=0 ssid="Shackleton N" ││Aug 05 15:38:29 office-paperpi wpa_supplicant[571]: wlan0: Trying to associate with SSID 'Shackleton N' ││Aug 05 15:38:29 office-paperpi NetworkManager[568]: <info> [1722865109.3145] device (wlan0): supplicant interface state: scanning -> associating ┤│Aug 05 15:38:29 office-paperpi NetworkManager[568]: <info> [1722865109.3146] device (p2p-dev-wlan0): supplicant management interface state: scanning -> associating ┤│Aug 05 15:38:29 office-paperpi wpa_supplicant[571]: wlan0: CTRL-EVENT-ASSOC-REJECT bssid=e0:63:da:18:43:db status_code=16 ││Aug 05 15:38:29 office-paperpi wpa_supplicant[571]: BSSID e0:63:da:18:43:db ignore list count incremented to 2, ignoring for 10 secondsStatistics: Posted by txoof — Wed Aug 07, 2024 3:26 pm — Replies 0 — Views 32