Updating Cisco WLC from 7.x to 8.3.x

Recently I needed to upgrade a Cisco WLC 4404 to a new hardware. That means upgrading from 7.0.x to 8.3.x. I thought that it’s going a piece of cake, but I’ve been so wrong…

I first configured the new WLC, although a lot has changed in the new release, it worked quite well. Using most of the configuration settings from the old WLC-4404 could be set directly on the new one. I used the command line and copied the command from a backup more or less one by one to have more control.

After checking the settings in the GUI I noticed that some tweaks were still needed. The FlexConnect (formerly HREAP) is not sending VLAN considering VLAN tags by default and has to be activated manually. After these optimisations, I thought the moment has come to finally switch. Hence I shutdown the WLC-4404 and setup the new WLC interfaces with the right IP addresses.

I then thought that I simply need to reboot the APs and they’ll fetch the new firmware from the controller and I’m done. That was the moment when the pain began. The AP wasn’t able to join the WLC.

I noticed the following message in the log

*spamApTask4: Jan 18 11:13:12.248: %DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:948 Failed to complete DTLS handshake with peer 1.2.3.4

Damn it. OK, searching the error message gave me some new insights. The manufacturer certificates are likely expired and therefore a join of the AP is not possible. Ignoring the expiration shall sort that out. So I set that on the WLC.

config ap cert-expiry-ignore ssc enable
config ap cert-expiry-ignore mic enable

Unfortunately that didn’t solve the problem. Trying to sort that out and further investigation took me quite a while. One suggestion was to step up from 7.x to 7.6 and then to 8.x. OK, that’ll be an option, but not my preferred one. So I thought about resetting the APs to a newer recovery image. Since I just have about 10 APs that could work out fine.

Unfortunately most tutorials use a serial cable, but I don’t have a machine with a serial port at hand. So I looked for other options to upgrade the firmware and found the solution in a blog entry about Convert Lightweight AP into Autonomous AP.

Noteworthy that newer SSH versions use stronger encryption and ciphers. To connect to the AP via SSH you have to set some legacy options.

ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 -oCiphers=aes128-cbc user@1.2.3.4

The rest is quite simple. Enable the full console

debug capwap console cli

and then download the archive from your TFTP-server

archive download-sw /overwrite tftp://server-ip/ap3g1-rcvk9w8-tar.153-3.JF4.tar

After a reboot the SSL errors are gone, but wait, WTF is that? The AP can’t join now…

*spamApTask0: Jan 18 11:16:32.731: %LWAPP-6-CAPWAP_SUPP_VER: spam_lrad.c:2000 Discarding discovery request in LWAPP from AP 11:11:11:11:11:11 supporting CAPWAP
*spamApTask0: Jan 18 11:16:32.726: %LWAPP-6-RADIUS_AUTHORIZATION: spam_radius.c:152 The system could not send join reply, AP authorization failed; AP:11:11:11:11:11:11
*aaaQueueReader: Jan 18 11:16:32.726: %AAA-4-RADSERVER_NOT_FOUND: radius_db.c:2641 Could not find appropriate RADIUS server for WLAN 0 - no authentication servers configured
*spamApTask2: Jan 18 11:16:32.724: %DTLS-5-ESTABLISHED_TO_PEER: openssl_dtls.c:873 DTLS connection established to 1.2.3.4
*spamApTask2: Jan 18 11:16:32.724: %LOG-6-Q_IND: spam_lrad.c:2000 Discarding discovery request in LWAPP from AP 11:11:11:11:11:11 supporting CAPWAP
*spamApTask0: Jan 18 11:16:21.540: %LWAPP-6-CAPWAP_SUPP_VER: spam_lrad.c:2000 Discarding discovery request in LWAPP from AP 11:11:11:11:11:11 supporting CAPWAP

MAC addresses are changed of course. Since I don’t have a RADIUS server in place there is nothing the AP can be authenticated against. The solution is to add the APs MAC addresses to the security settings.

Done, finally, APs join and everybody is happy again.