Retrieving IOS running-config via scp

Ok, now I’ve only tested this with routers running IOS – it may be a little different with Catalyst switches, since they store their config on flash, rather than nvram. On the other hand, it may be exactly the same, since we’re retrieving running-config, not startup-config.

First, you need to ensure that ssh and scp have been enabled. I strongly recommend that you run ssh version 2.

ip ssh version 2
ip scp server enable

Then, on your PC:

scp user@router-hostname:system:running-config .

You should then have a file called “running-config” in that directory. Pretty simple…

If you want to grab the startup-config instead of the running-config, try:

scp user@router-hostname:nvram:startup-config .

By using RSA keys to eliminate the password prompt at login, this method could be expanded to form the basis of an automated config backup. I know that various apps already exist, but a lot of them retrieve the config via “expect” scripts, basically executing a “show run” and capturing the output.

Another method of retrieving the config is via SNMP, however unless you’re using SNMP v3 with encryption, this method is potentially insecure.

Upgrading IOS remotely

I recently had to upgrade a bunch of Cisco routers to an up to date IOS. These routers were scattered up and down the country, and I don’t have much to do with the servers sitting behind them, so I needed to do a remote upgrade over the Internet.

Now, TFTP is pretty hit and miss at upgrading remotely – and not particularly fast either. Given that TFTP runs over an reliable transport protocol (UDP), I tend to only use it on LANs, or for truly “trivial” things like backing up configs (and SCP is more secure for that). Since the routers were running an older IOS that didn’t support HTTP, I decided to have a crack at using FTP. What a drama…

Firstly, you need to realise that by default, the FTP client in IOS tries to use passive mode. The server I was hosting the new IOS images from was behind a firewall that was only configured for active FTP (ie, only port 20 and 21 open). So when the router tried a passive FTP download of the new image, the firewall denied the randomly-chosen port that the router had chosen to connect on.

Cisco “ip inspect” to the rescue. I added a stateful FTP inspection rule on the firewall (Cisco also) like so:

ip inspect fw-in ftp
!
interface Dialer0
ip inspect fw-in in
!

Now the firewall would do a stateful inspection of the FTP connection, and allow the subsequent randomly-chosen port passive FTP transfer.
That got a little further, but now the connection was stalling, even though vsftpd was showing a successful login and transfer begin. After searching for a bit, I came across some references to Cisco routers and FTP ABOR(t) commands causing problems with ProFTPD. I read through the vsftpd config on the FTP server and discovered an option for asynchronous aborts.

async_abor_enable=YES

I suspect the need for this arises from the fact that, when upgrading IOS, the router always checks to make sure it can actually read the file you specify, before it offers to wipe the flash. So, in this case, the router was starting an FTP transfer, then aborting it, then wiping the flash, then trying to start the transfer again. Once I had enabled that option, the transfer seemed to work. I say “seemed to work”, because I actually only got this to work on one router, and by this time it was about 2:30am. I was rapidly coming to the conclusion that the FTP client is a bit borked in older IOS releases.

So in the end I had to resort to upgrading a few routers via TFTP. Hopefully they are now running recent enough IOS that the FTP is a bit more reliable, or even better, supports HTTP (which is much more likely to succeed, since it carries control and data in a single connection).

It seems that the “ip inspect” feature of IOS is one of the most misunderstood commands of all, since I only ever see it being used in the outbound direction. Apart from using it to inspect outbound TCP sessions, and do away with the need for a rather insecure “permit tcp any any established” in an access-list, I don’t see a lot of point in inspecting outbound traffic. A few tricky protocols need a bit of assistance here and there, such as instant messaging and P2P protocols, to allow return traffic to establish an unrelated connection inbound. But the most use I see for it, is handling those tricky inbound connections such as when you’re hosting FTP, so that you don’t have to leave gaping holes in your inbound access-list.

I also found that http works a metric shitload faster if you don’t inspect it in the outbound direction. Even Cisco don’t recommend enabling it, unless you want to do Java blocking.

:-/

CCNA exam

I renewed my CCNA certification today, which should now mean that my CCNP is complete. What a wonderful little racket Cisco runs with these exams. Funny how there are no official prerequisites whatsoever for CCIE.

Maybe it’s time to take a look at some Juniper certifications…

CIT exam, part 2

Second time around, passed with flying colours. It’s amazing the difference it makes when you have up to date study material. So that’s all four exams done, but it appears I might not yet be a CCNP. Since my CCNA expired a couple of years ago, I think I will have to re-sit the CCNA exam just to renew it so that it counts towards my CCNP.

It’s a pretty stupid rule, IMHO, since, if you can pass all the CCNP exams, you are obviously well beyond CCNA level. It seems like it’s just another way for Cisco to make money. I’ll be in big trouble if I ever let my CCNP expire, because that would mean sitting all four (or maybe five, including CCNA) exams again!

Of course, CCIE does not have any prerequisites – not even a current CCNP or CCNA. You’d be pretty brave to attempt CCIE without at least several years experience and/or having at least gained CCNP once. But since I plan to tackle CCIE next, I doubt I’ll worry too much about keeping my CCNP up to date. The way I see it, CCIE trumps all previous qualifications anyway.

Cisco 857 router

I’ve finally replaced my trusty old D-Link DSL500, which I’ve had for about four years, with a Cisco 857. What can I say about these routers… well…

My 857 router arrived with SDM Express, but not SDM, installed on the flash drive. While SDM Express is an improvement over the old Cisco Router Web Setup (CRWS), one of the reasons I bought the 857 was to see whether SDM is as good for routers as ASDM is for PIX. So I set up the router using SDM Express, and had a look at the lovely mess of a config it generated. It would probably have sufficed for a non-technical user, but being three-quarters of the way to a CCNP, I don’t think I qualify as that anymore.

First up is the ATM0.1 sub-interface that SDM creates. Ok, this probably is a good way to do it, since, even when configuring a single DLCI with frame-relay, I’ve got into the habit of using a sub-interface. But in NZ, I think we’re far less likely to have the option of multiple ATM PVC’s on ADSL than we are of having multiple DLCI’s on frame-relay.

The 857 (in fact, all the 850 and 870 series routers) have a built-in four port fast ethernet switch. While this shows up as four individual interfaces in the config, and you manually set some options per interface (layer 2 options only, I suspect), it does not function as a VLAN-capable switch, such as in the 870 series routers.

So, now for some of the gotchas. If you plan to run a server behind a router like this (and this probably would affect any Cisco ADSL router), and you only have the one public IP assigned to the Dialer interface, there are two ways you can go about it. If you run a large number of public services on that server, you may be tempted to do something like:

ip nat inside source static 10.0.0.5 interface Dialer0
ip nat inside source list 1 interface Dialer0 overload
access-list 1 permit ip 10.0.0.0 0.0.0.255 any

Of course, you should apply an access-list inbound on the Dialer0 interface, so you don’t completely expose that server. Cisco IOS is smart enough that you can have other hosts on your internal network NAT outbound. You can even specify individual inbound port-NAT entries, such as:

ip nat inside source static tcp 10.0.0.31 4662 interface Dialer0 4662

for a P2P eMule client, and the port NAT will take precedence over the whole IP NAT for the server.

Where this comes unstuck however, is if you want to terminate an IPSEC tunnel on your router. Remember, we’ve only got one public IP on our Dialer0 interface. Unfortunately, IOS is not smart enough to figure out that it should locally process incoming ESP and ISAKMP traffic – and instead forwards it to the server that you specified. So, faced with this situation myself, I have had to create individual port NAT entries for all the services I host on my server. Fortunately, IOS no longer seems to suffer a bug I enountered years ago, where UDP DNS packets didn’t NAT properly. Since DNS quite often uses UDP (like, if the query is less than 512 bytes), this bug used to make it impossible to host a DNS server behind a router like this.

The next gotcha I came across is the “ip inspect” command having a fit when confronted with out-of-sequence packets. When running an IPSEC tunnel to a NetScreen 25, I found that certain protocols that were in my “ip inspect” list were stalling. Debug revealed that large numbers of packets were being dropped, due to being out-of-sequence. After some research, I learned that Cisco’s IOS-based IPS (ip inspect) really doesn’t like having to deal with fragments. I suspect this is the reason for the relatively new IOS command “ip virtual-reassembly”, which attempts to reassemble packets prior to “ip inspect” checking them. I suspect my problem was that I was getting a lot of fragments over the VPN, due to incorrect TCP-MSS settings, and the smaller fragments were arriving before the larger fragments – hence “ip inspect” considered them out-of-sequence. Debugging “ip virtual-reassembly” revealed “invalid parameters” – which I could find no further information on. It seemed the best course of action would be to eliminate the fragmentation to begin with. After spending several hours unsuccessfully experimenting with MTU and TCP-MSS settings, the solution finally came down to setting one parameter on the far-end NetScreen – “set flow path-mtu”. Once this was enabled, everything worked fine. Obviously, PMTU discovery figured out it needed to decrease the TCP-MSS to account for the ESP encapsulation overhead. This turned to be a preferable solution to manually clamping the TCP-MSS for all traffic.

Getting back to SDM, I installed the full SDM on my router via TFTP (since the actual SDM installer just hung repeatedly, despite following Cisco’s instructions for retro-fitting existing routers with SDM). SDM is certainly more feature rich than SDM Express, but I don’t rate it quite as highly as ASDM for PIX. I ended up doing the bulk of my config by hand, from CLI, and using SDM just as a monitoring front end. It does have an audit tool however, which can be a nice security check of your config. It mostly suggests turning off services like pad and finger. Hopefully someday soon, these will be off by default anyway.

A few complaints about SDM – setting the timezone for your router is kind of weird. It called my timezone “Napier”, which, although is in NZ, and the same timezone as Auckland, I’ve never seen it referred to like that before. Officially, our timezone should be NZST/NZDT or Pacific/Auckland. SDM also configured absolute dates for daylight saving start/end. This is not correct – DST start/end is determined by week number in October and March respectively.

Configuring the IPSEC tunnel initially in SDM was a lesson in Cisco etiquette. It had some default IPSEC proposals that it wouldn’t let me delete, so I had to add my preferred proposals as secondary options. Afterwards, I tweaked the crypto map by hand in the CLI.

Don’t rely on SDM to get the ordering right of access-list entries. For ease of editing, I no longer used numeric access-lists, except for simple one or two-liners. Instead I use the “ip access-list extended ” format, makes it easy to remove individual entries. You can also easily insert entries by specifying the entry line-number, a bit like a BASIC program listing. Lastly, be careful when closing the SDM window, because it closes all your browser windows!

A couple of things to beware of with the 857 (as opposed to the 877). The 857 is the successor to the SOHO 97, not the 827 or 837 as one might think. As such, it is not particularly grunty, and if you run a lot of sessions or IPSEC tunnels (maximum of 5), you might find the CPU getting quite bogged down. The 857 does not support IPv6, which is surprising, since an 827/837 can, with the right IOS image. It also does not support class-based queuing, which can be a problem if you wanted to reserve bandwidth for, and prioritise VoIP traffic. I haven’t yet found a way to run the router’s SSH on a non-standard port, since the vty complains if you try to assign it to a different rotary group.

So, while the 857 is successfully doing firewalling, NAT and IPSEC for me now, I’m sorta wishing I’d spent the extra money and bought an 877.

CIT exam

Uh-oh… didn’t pass this one. Thanks to my Cisco Press CIT Exam Preparation Guide being well and truly out of date (despite being the latest edition), I went into this exam completely unprepared. The CIT exam seems to cover the widest range of topics out of all the CCNP exams. It basically boils down to troubleshooting any and all of the technologies covered by BCMSN, BSCI and BCRAN. I’m planning to re-sit the exam, and hopefully will do better next time around.

BCRAN exam

Another one bites the dust. BCRAN exam done and dusted. Once again, lots of US-centric technology that I’m unlikely to run into in NZ, or even Europe. And of course, they never test you on the topics that you spend the most time studying :\

Anyway… three down, one to go.

BSCI exam

Well, after about a month of studying dynamic routing protocols, some of which I doubt I’ll ever use or come into contact with, I passed my Cisco BSCI exam. Two down, two to go.