I think there are a lot of lessons to be learned from this experience, not the least of which is “trust nobody”
Last Friday, as I was tying up loose ends at the office, preparing to wind down in anticipation of the weekend, I made a terrifying discovery.
It started with an IDS alert, and ended up when I discovered a problem on one of my company’s firewalls — a serious problem.
One of the things I’ve been working on recently is building a new network intrusion-detection system (IDS). It’s been on my security road map for a while now, but our new budget constraints have made it hard to keep the project moving forward. To get my IDS sensors deployed has required a combination of scare tactics (which I hate to use, but desperate times call for desperate measures), outsourcing and old-fashioned, do-it-yourself resolve. For the past several weeks, I’ve been gradually making progress. The IDS sensors are in place, reporting to a security incident and event management (SIEM) system that correlates all the alerts along with log entries from network devices and servers, antivirus systems and other technologies. It’s a pretty cool system, and having been done on the cheap, it provides great bang for the buck.
So, on this Friday afternoon, I was sorting through the alerts being generated by the new system when I ran across something odd: a large number of remote desktop connections from the Internet into some computers on our internal network. In his Aug. 22 Security Manager’s Journal, Mathias Thurman referred to remote desktop as “probably the top method for unauthorized access.” It’s certainly not something I expected to see coming from the Internet. At first, I thought it must be a false positive, or maybe it was just a poorly designed access method for some remote vendor. In short, I didn’t believe this could actually be happening.
But on closer inspection, it turned out that those remote desktop connections were all originating from another country — one that is, shall we say, less than friendly with the U.S. A country that recently made headlines with state-sponsored network attacks against some famous U.S. companies. A country that now appeared to be connecting to my company’s computers!
Well, that was the last thing I wanted to see on a Friday afternoon. But there was no way I could leave without knowing exactly what was going on. I cornered the network admin. (In my company, firewalls are considered part of the network plumbing and as such, they are managed by the network team.) I told him I needed to see the policy configuration on the firewall in front of those systems. Within seconds, he produced the requested information, and even though I thought I was prepared for the worst, I was physically shocked at the result.
The firewall was configured with several ip-any-any rules. That means, for several computers on our internal network, any computer on the Internet could connect using any protocol – in other words, the firewall was wide open for about 16 computers on my company’s network. With an ip-any-any rule, you essentially have no firewall at all, because it’s allowing all the same traffic you would get from directly connecting a network cable.
If you’re familiar with firewalls, you probably know the sensation of horror I felt. If not, I’m not sure I can really describe it — but it’s basically my worst nightmare. My network had a huge hole that hostile attackers were exploiting. It was like emptying out a cupboard in your kitchen and finding a hole in the wall that nasty critters were using to get at your food.
I sent the network admin off to close the firewall holes and initiated an audit of configurations on all our firewalls. Naturally, I had been auditing our firewall configurations on a regular basis, but with my lack of staffing and resources, I hadn’t been able to do it very often. And these changes appear to have been fairly recent.
I think there are a lot of lessons to be learned from this experience, not the least of which is “trust nobody.”
This week’s journal is written by a real security manager, “J.F. Rice,” whose name and employer have been disguised for obvious reasons. Contact him at firstname.lastname@example.org.