By Joel Snyder
Network World, July 31, 2006
Original Article on Network World Web Site
What's on your network? Sourcefire's Realtime Network Awareness and Tenable's Passive Vulnerability Scanner attempt to answer that question without leaving muddy footprints all over the network. Both use a technique called passive network analysis to listen to traffic as it flows by, thereby discovering systems, topologies and vulnerabilities.
We tested RNA and PVS on a production network for more than a month. Overall, while both tools are fairly good at what they do, the tangible value for either product would be realized only in a big network. Security managers who need to monitor a large, dynamic network can probably gain significant value from these products, because they trim the number of intrusion-detection system (IDS) alerts that need to be investigated, and help detect system vulnerabilities. For smaller networks, the value proposition is not as strong, because other techniques, such as active scanning (see Active vs. passive scanning), give more accurate results in those networks.
Passive network-analysis tools are designed to pull information out of the network as the traffic flows by. Although the two tools we tested are similar in that they focus on network application inventory and vulnerability analysis, they have different design strategies.
With Tenable's PVS, the goal is to detect and report on system applications and vulnerabilities. Tenable is home to the popular Nessus active vulnerability-scanning freeware. PVS (originally called NeVO) is the passive complement to Nessus. The latter product works by performing active scans of systems using a wide variety of techniques ranging from pinging to logging into a system and looking at the file system and registry, but PVS does its detection without sending a single packet.
We tested PVS linked to Tenable's Security Center V3, a security-management tool used to integrate multiple vulnerability scanners - active, passive or a combination of both - and correlated vulnerability information with IDS and syslog data sent to Security Center by sensors and servers.
The goal of Sourcefire's RNA is to build host profiles for all the systems on the network and assist in prioritizing and analyzing IDS events. As home to the open source Snort IDS engine, Sourcefire has made a name for itself selling a commercial version of Snort along with Defense Center, which acts as a centralized management system and data analysis console for multiple IDS and RNA sensors. We tested it as part of a larger Sourcefire installation with an IDS sensor and Defense Center V4.5.1
These products will be of greatest use in larger networks with multiple subnets and 1,000 stations or more. For example, Tenable's PVS provides less information than an active vulnerability scanner. However, PVS carries none of the risks of system crashes or the political problems of active scanning - problems that are magnified in large networks. PVS is also arguably more effective than active scanning for large networks, because it detects changes in configuration and topology as they happen. RNA brings the same advantage to the ever-changing face of an enterprise network by providing a real-time network inventory function that directly supports the process of managing IDS alert information.
Both products had good performance as far as keeping up with network traffic and providing a lot of information about the systems talking on the network. Both also made errors in their analysis.
We pulled out 50 high vulnerabilities, as reported by Tenable's PVS, and researched each one. With an out-of-the-box, untuned configuration, the error rate is pretty high. We found PVS was wrong almost half the time in identifying vulnerabilities. Some of the problems were systemic, such as in misidentifying mail servers. For example, PVS identified one of our antispam appliances as running outdated versions of Lotus Notes, Mutt and Outlook Express e-mail clients. In reality, it wasn't running any e-mail client. PVS had seen the banner fly by in a mail message and marked the system as a client.
The nice thing about PVS is that you suppress these kinds of errors quickly by creating a dynamic host group of mail servers, an easy operation, and then telling PVS to ignore this vulnerability for that kind of server.
Other errors were not so easily explained or worked around. For example, PVS misidentified mail servers as running a vulnerable IMAP server from an entirely different vendor than was the case. PVS incorrectly picked out another of our antispam appliances as running an unpatched version of Mac OS X - but did identify correctly the Macs that had not been patched either. And even though PVS properly identified OS X servers, it assigned high-priority Windows vulnerabilities to them.
As we tuned PVS over time, we were able to fix many identification errors and got the real error rate down to just less than 20% for high-priority vulnerabilities. In a more static network, the error rate could be taken even lower, although at the risk of not catching new systems and changes in the network.
Trying to determine the error rate for Sourcefire's RNA gave us greater difficulties, because each part of the results RNA came up with for each host had a different class and frequency of errors.
RNA breaks out services, vulnerabilities and other host attributes separately, and tries to map the network by identifying bridges and routers. In our testing, RNA's topology-mapping strategy showed flaws, finding only five of 18 bridges in our network and misidentifying an additional 12 systems (including an American Power Conversion UPS) as bridges.
In terms of host identification, we audited RNA's results for all 120 systems on one network segments and found that RNA was right 42% of the time; found a running system but marked it as unknown 49% of the time; and was outright wrong 9% of the time. It called one of our HP switches and two of our Juniper firewalls HP Laserjet printers, and identified one of our antispam appliances and a Nokia firewall as IBM AIX systems. RNA did a better job in listing server applications, finding Web, mail and Secure Shell servers running on nonstandard ports on a regularly accurate basis.
Although RNA does include some vulnerability-analysis tools, they're far from useful at this stage, and the listings RNA provided were extraordinarily inaccurate. For example, once RNA had identified one of our servers as running particular versions of Linux and Apache, it immediately attached 229 vulnerabilities to the server - even though only a few were applicable. If Tenable uses an "innocent until proven guilty" approach to marking vulnerabilities, Sourcefire considers every system "guilty until proven innocent." For the list to be accurate, you have to manually go in and mark each of the incorrect vulnerabilities - an inconceivably onerous task in a large network.
RNA and PVS are outstanding tools to help sort through immense mounds of data and bring order to a large, chaotic network. To extract that value, however, you need to look closely at both when integrated with their respective management consoles.
We put an unpatched system inside our firewall and let RNA and PVS learn about it over a period of one week. When we opened a hole in the firewall to let the world start attacking this system, RNA's Defense Center immediately correlated an IDS alert of a Windows Remote Procedure Call attack with the vulnerable operating system, and then elevated the alert to the top of the list of attacks to worry about. We did the same test using a Unix system, and Sourcefire demoted the attack, pushing it down in the list.
A purist might say this result is deceptive, because we have no guarantee the most significant attacks always will be identified and promoted. That's very true. RNA is good at identifying certain operating systems and is especially good at identifying the operating systems most likely to be attacked and compromised - it misidentified only one of our Windows servers over the entire 45-day test. So the benefit of combining RNA with IDS alerts is that at least some of the alerts that really matter will be highlighted. The reality of IDS alerts, however, is that there are usually too many to be examined anyway, which means any help the network manager can get from an IDS console is going to be a big improvement.
RNA offers another benefit. Although you can't correct misinformation that RNA has learned, you can assign device-importance information, such as giving a high priority to your servers and a low priority to systems in a test lab. These priorities also can be used by Defense Center in assigning weights to different IDS events.
Defense Center is far advanced from our last look at it two years ago, but security managers hoping to integrate RNA with their IDS should be prepared for a hefty learning curve. Defense Center brings a great deal of power to a security analyst through its Web-based GUI, but in a counter-intuitive and difficult-to-use way.
Tenable's Security Center makes it possible to combine the passive information of PVS and any active scan information you might be willing to gather from Nessus. Together, these can be managed, reported on, analyzed and even trended over time. Although Tenable can use vulnerability information to help in IDS event management, the real sweet spot for this product is in networks where vulnerability information is an important part of security policy.
As a superconsole for multiple IDS sensors and other security devices, Security Center uses vulnerability information along with other correlation data, such as firewall connection logs and IDS events, to filter alerts from other devices. For example, if we picked one day, our IDS and firewalls sent 8,263 alerts to the Security Center. By punching the correlated button, we narrowed that to a list of two alerts correlated among multiple logs.
The IDS management features of Security Center are fairly immature at this stage, however. For example, it lets you look only at IDS events across a 24-hour period or less. If you want to search a whole week, you have to do seven separate searches. Also, alerting and reporting tools are barely present in Security Center, and the Web-based management tool is missing the important data summary and drill-down facilities needed when the number of elements viewed is large.
Sourcefire's RNA and Tenable's PVS don't directly compete, because they meet separate needs. If you have a large network and are using Sourcefire's IDS and Defense Center, then adding RNA is a no-brainer. If you've got another vendor's IDS in the mix and are drowning in alerts, because your network moves too quickly for you to tune the IDS properly, a Sourcefire-based solution with RNA might solve your problems. Buying RNA for its vulnerability-analysis features at this stage in the product's life cycle would be a mistake.
If you use Tenable's Nessus for active scanning of your network and cannot get the quality of information you need, or if you are realizing that your scanning efforts are not effective, then PVS (along with Security Center) is the ideal complement and an easy upgrade to justify. If you're using a different active-scanning approach, jumping to Tenable might or might not be a good idea. Because the Tenable solution doesn't necessarily include all the alerting, reporting and database-management tools of competing products, your choice is harder. Tenable certainly can improve the quality and effectiveness of your vulnerability-analysis tools, but if you're giving up other critical features, the trade-off should be considered before jumping ship.