Catalyst Express(?!) 500 switches

On the weekend I helped a friend set up some Cisco kit involving Catalyst Express 500 switches, Aironet 1310 wireless bridges, and various Cisco IP phones. What should have been a relatively simple job ended up taking about 10 hours.

My friend had already configured the 2800 series router, with Cisco Call Manager Express. Phones attached to a local Cat 500 worked fine – upon booting up, they configured their VLAN, got a DHCP lease, and registered to CCME. Next came the Aironets. Having never configured VLAN trunking on a wireless link, this was new ground for me. At first we had each VLAN (native VLAN 1, and voice VLAN 200) using its own SSID. It wasn’t until later that we realised we could run both over the same SSID.

Where the problems arose, was on the remote Cat 500 (ie, on the other side of the wireless bridge). The phones connected to that switch appeared to time out, trying to get a DHCP lease. Since we had previously plugged them in on the local Cat 500, they eventually fell back to their previous DHCP lease, and were actually able to make calls. This wouldn’t help us for plugging new phones in though. Since my friend had done the initial configuration of the Cat 500’s, and wasn’t sure at the time whether the Aironets were supposed to run as bridges, access points or both, he had set the Smart Port type on Aironets’ Cat 500 switchports to “Access Point”.

When I looked at the interface stats on each of the Aironets, neither of them reported any broadcasts for the radio interface. I thought this was odd, since a DHCP request is a broadcast. I even tried debugging IP packets on the Aironet, while a phone booted up. Nope, don’t see any DHCP requests. I started to get the feeling the switches were blocking broadcasts somehow, but at that point still assumed my friend had set up the switch correctly. Also, without a CLI on the Catalyst 500’s we couldn’t debug packets or VLAN encapsulation on the suspect port. In between a few drinks of vodka and Coke, some pizza, and a few rounds of the game “Burnout”, we chased our tails for several hours, eventually giving up and sleeping on it.

The next morning, I revisited my theory of the Cat 500 being at fault. We upgraded the IOS of all the switches, and the Aironet bridges. Still no luck. I went back into the Cat 500 web config (did I mention these things have no CLI?), and examined the switchport config. Neither my friend or I were exactly sure what the various Smart Port roles did, in terms of spanning tree, VLAN access/trunking, and QoS. We could only guess, since the documentation doesn’t really make it very clear either, and without a CLI, we couldn’t even verify the config against our own knowledge of Cisco IOS. Then I said, “Hey, we have a VLAN trunk going across the radio, these Aironet ports are gonna have to be in trunk mode too. What does the ‘Switch’ Smart Port role do?”

And then it worked.

Yep, a “Switch” Smart Port allows VLAN trunking. Apparently an “Access Point” Smart Port does too, otherwise we wouldn’t have been able to make calls earlier (remember the Skinny & RTP traffic would have been in VLAN 200). For some reason I’m still scratching my head about though, it was not passing broadcasts (even from the native VLAN, when we tried to get a notebook attached to the remote Cat 500 to pick up a DHCP lease). The only hint given by the documentation, is that an “Access Point” Smart Port allows up to 30 connected users on the access point. Ok, so it’s enforcing a MAC address limit of 31 on that switch port. Why not just say that… it still didn’t explain not forwarding broadcasts though. How is a client associated to an access point supposed to obtain an IP address?

So that was all good, and we proceded to set up the encryption on the wireless link. That was Brainfuck #2. Over the years, Cisco have added so many extensions to IEEE and IETF standards, that setting up Aironets is like eating alphabet soup. It was only due to my previous training and experience with Proxim wireless gear that I was able to correctly choose the ciphers that are more commonly known as WPA2. After another treasure hunt through the config to find where to set the pre-shared key for WPA, we had encryption working.

In the meantime however, the Catalyst 500’s had disabled the ports that the Aironets were connected to. According to the Cat 500 event log, it was because it had detected one way communication on the port. Err… what? Ok, sure, the wireless link was down while we were setting up the encryption, but disabling a port because of that could get real annoying, especially if the link got knocked out by weather. We re-enabled the ports (a tedious process of going into the Cat 500 web GUI, disabling the port, applying that, re-enabling the port, applying that again).

Finally, it seemed everything was working. Looking back on it after some thought, I suspect the Catalyst 500’s are getting upset with the “Switch” Smart Ports when they don’t hear any spanning tree BPDU’s on them. STP was enabled for both VLANs on the Catalyst 500’s, we told it we had a switch connected to those ports, and it recognised the active ethernet link status. With the wireless link down however, the switches wouldn’t have seen any BPDU’s arrive on that port for several minutes (we are not running STP on the Aironets). That would be long enough for spanning tree to get upset – while on the other hand, the ethernet link status was still ok. The switch might have thought there was a faulty TX at the other end of the cable. Since these Catalyst 500 switches are seriously dumbed down, your guess is as good as mine.

Either that, or it was secretly using UDLD (unidirectional link detection), a feature often used by “real” Catalyst switches to detect faulty transmit circuitry on GBIC interfaces. Except that UDLD should only be applied on fibre interfaces (ie, where one fibre of the pair could be damaged). Copper interfaces shouldn’t need UDLD.

So, since we are not running STP on the Aironet bridges, my only guess is that enabling it might keep the Catalyst 500’s happy in the event of a wireless link failure. The Cat 500 will still see BPDU’s from its directly-attached Aironet… Enabling spanning tree on the Aironets could be Brainfuck #3 though, since there is no way (that I could find) to set the bridge priority on a Catalyst 500. In an environment such as the one I was helping set up, there were no other “real” Catalysts that we could have set as the spanning tree root bridge. So the Catalyst 500 (or Aironet) with the lowest MAC address would be elected as the root bridge. It would be seriously uncool if that switch just happened to be the one at the far end of the wireless link.

So what did I learn from that experience? I won’t be using a Catalyst 500 anytime soon, and even if I do, only as an access layer switch in conjunction with a more fully featured Catalyst (at least a 2950).