Upgrading IOS remotely

I recently had to upgrade a bunch of Cisco routers to an up to date IOS. These routers were scattered up and down the country, and I don’t have much to do with the servers sitting behind them, so I needed to do a remote upgrade over the Internet.

Now, TFTP is pretty hit and miss at upgrading remotely – and not particularly fast either. Given that TFTP runs over an reliable transport protocol (UDP), I tend to only use it on LANs, or for truly “trivial” things like backing up configs (and SCP is more secure for that). Since the routers were running an older IOS that didn’t support HTTP, I decided to have a crack at using FTP. What a drama…

Firstly, you need to realise that by default, the FTP client in IOS tries to use passive mode. The server I was hosting the new IOS images from was behind a firewall that was only configured for active FTP (i.e., only port 20 and 21 open). So when the router tried a passive FTP download of the new image, the firewall denied the randomly-chosen port that the router had chosen to connect on.

Cisco “ip inspect” to the rescue. I added a stateful FTP inspection rule on the firewall (Cisco also) like so:

ip inspect fw-in ftp
!
interface Dialer0
ip inspect fw-in in
!

Now the firewall would do a stateful inspection of the FTP connection, and allow the subsequent randomly-chosen port passive FTP transfer.
That got a little further, but now the connection was stalling, even though vsftpd was showing a successful login and transfer begin. After searching for a bit, I came across some references to Cisco routers and FTP ABOR(t) commands causing problems with ProFTPD. I read through the vsftpd config on the FTP server and discovered an option for asynchronous aborts.

async_abor_enable=YES

I suspect the need for this arises from the fact that, when upgrading IOS, the router always checks to make sure it can actually read the file you specify, before it offers to wipe the flash. So, in this case, the router was starting an FTP transfer, then aborting it, then wiping the flash, then trying to start the transfer again. Once I had enabled that option, the transfer seemed to work. I say “seemed to work”, because I actually only got this to work on one router, and by this time it was about 2:30am. I was rapidly coming to the conclusion that the FTP client is a bit borked in older IOS releases.

So in the end I had to resort to upgrading a few routers via TFTP. Hopefully they are now running recent enough IOS that the FTP is a bit more reliable, or even better, supports HTTP (which is much more likely to succeed, since it carries control and data in a single connection).

It seems that the “ip inspect” feature of IOS is one of the most misunderstood commands of all, since I only ever see it being used in the outbound direction. Apart from using it to inspect outbound TCP sessions, and do away with the need for a rather insecure “permit tcp any any established” in an access-list, I don’t see a lot of point in inspecting outbound traffic. A few tricky protocols need a bit of assistance here and there, such as instant messaging and P2P protocols, to allow return traffic to establish an unrelated connection inbound. But the most use I see for it, is handling those tricky inbound connections such as when you’re hosting FTP, so that you don’t have to leave gaping holes in your inbound access-list.

I also found that HTTP works a metric shitload faster if you don’t inspect it in the outbound direction. Even Cisco don’t recommend enabling it, unless you want to do Java blocking.

:-/