Case Study: Firewalls: a cautionary tale

We had been using the access lists on our Cisco router for some years. They were useful as a stop gap but unwieldy to manage and prone to letting stray packets through when they should not have. Caught up in the security frenzy which mushroomed over recent years we eventually decided that we must deploy a border firewall of our own.

Looking around for something suitable we found that numerous software and hardware firewalls were available. We read the widely varying reviews and bench-test results. We found firewalls that were too expensive, firewalls that didn’t do this, firewalls that didn’t do that, firewalls that had awkward interfaces, Finally we settled on a software firewall, the brand of which shall remain nameless for reasons which will become obvious.

We had specified our requirements carefully; we run a 100Mbps connection to the Internet and have thousands of users. We needed something that could handle a lot of connections.

Digging deeper

We were told this firewall could do everything we would ever want. Decision made, we purchased it from a reseller. We prepared a server to run the software, taking care to specify the machine as highly as possible. The firewall software arrived in its gleaming packaging, the colour of which was somewhat reminiscent of the little birds that used to go down the coalmines. This bird, however, was already dead in its cage.

We had discussed our intended security policy with the reseller’s consultant beforehand and provided him with diagrams and details of what we intended to do. Now we scheduled the necessary downtime, the consultant arrived on site and with his assistance the product was installed. We turned to configuration.

The consultant immediately took umbrage because he didn’t like our security policy. We are an academic institution and have to accommodate the peculiarities common to such an environment. We have to open ports for a variety of unusual services that need to be run from outside the network in the name of collaborative working. The consultant was having none of it; forget about the customer always being right. Though we accepted his advice in theory, there we had no alternative but to allow certain services through the firewall. He vehemently disagreed with this. After much griping he reluctantly completed the job and the firewall was in place.

It was 8am on a weekday. “It’s looking good,” we said. We weren’t to say that again for a long time.

Network usage started to increase as 9am approached and the firewall suddenly experienced a massive panic. As soon as a few packets came along it seized up. Puzzled, we went through the configuration once more and checked the machine. There was evidence of a hardware problem. We changed the hardware. We put the firewall back into the connection. Again, as soon as a few packets came along the firewall had an attack of the vapours.

User connections were hanging, performance was dire. “You need to have fewer rules,” we were told. We had around 20; yes only 20. We reluctantly modified the rule set. “You need to switch off some of the functionality,” said support. We did so. We cut it down to the most basic functionality possible and it still couldn’t handle the network traffic. We twiddled and patched and massaged, all with the help of the consultant, but we could not make it work.

By this point the Internet connection had been down for nearly 6 hours. Over the next few weeks the whole saga was repeated several times as different hardware and software configurations were tried. Eventually we decided to abandon ship. It was incredibly depressing and we felt that our credibility had been wrecked. Extraordinary as it seems our users were really quite sympathetic.

We declined to pay the company who had supplied the software on the grounds that the product did not do what it said on the box. We had been extremely patient with them and had suffered many hours of downtime. We had experienced horrible stress and ended up with a product that didn’t work. So they sued us.

The reseller claimed that it was our fault because we had too many rules. This in spite of the fact that we had clearly stated our requirements and they had recommended the product as being able to meet them. The case is still open a year later, even though the reseller has gone bankrupt in the meantime.

I’d like to say that there is a moral to this tale but I’m not sure it’s that simple. We did everything we were advised; we researched extensively, we specified carefully, we took up references. Subsequently we have moved on and bought another firewall - a hardware firewall which does do what it is supposed to do.

Thank goodness. I can only handle one lawsuit a year.

*Vanessa Watkins is the network manager at the Royal Holloway College, London.