The Vision gigabit-class packet capture howto

by Andrew Watters

July 2, 2016 02:42 PDT

This page is a chronicle of my search for an affordable high performance packet capture appliance. Ultimately, I decided to build my own device, and this became The Vision. Here, I share with you some of the hard work behind that product and I give you the opportunity to build your own solution as well. If you don't want to read all of the preliminary information, skip to the howto below.

The search before I knew anything

I posted the following on LinkedIn in October 2015. Wow, I was clueless at the time:

You really get a sense of a company's priorities when you cold-call the pre-sales department and ask detailed questions about their products. My experience in a recent search for some specialized networking hardware revealed more than I expected about how several leading companies actually react to prospective customers.

I started my search for a high performance packet capture appliance by emailing General Dynamics Mission Systems-- an existing vendor I've bought stuff from-- about any product lines in this area. Within twenty minutes, I received an informative message back indicating that they had no such product lines anymore, but inviting me to contact a different company, AASKI Technology, whose offerings I might find interesting. Now that is great service.

As suggested, I contacted AASKI Technology about any products they might have. Unfortunately, I didn't hear back and was unable to reach either their headquarters or their California office by phone. The AASKI website lists no products in this area, but if GD referred me there then there could at least be some products in development. Dead end.

I next contacted a company that supports the open source software application Wireshark, which is used for packet capture and network monitoring. This company is Riverbed, which makes the NetShark appliance. I asked their pre-sales department a couple of technical questions and also how to buy a NetShark, and they promised to have someone contact me. I waited about ten days for a contact, but didn't receive a call or email.

I then tried to contact Fidelis Cyber Security, which makes the XPS appliance formerly sold by General Dynamics. That product line was sold to a private equity firm, apparently. I emailed and called at least three times, but no one returned any of my inquiries. Granted, the Fidelis XPS is the most expensive of these systems, but when they decline to share price information on their website they should expect inquiries from smaller companies.

After striking out with AASKI, Riverbed, and Fidelis, I randomly found a company called Klos Technologies on the internet by searching for "packet capture appliance." I emailed them and promptly received a detailed reply. I set up a call and spoke at length with one of their reps, asking several technical questions and receiving good answers. Everything looked great, and I liked all the product features. I put the Klos PacketVault at the top of the list.

In an effort to do due diligence, I followed up with Riverbed and was able to set up a call to get some answers about their NetShark, but I don't know how that will turn out. I also noticed that Cisco has an offering in this area called the FirePower. So I contacted Cisco via their online chat. Sadly, the chat gave me zero information about their product, and instead seemed like a cheap way for Cisco to gather as much information about me as possible before deciding whether to devote the resources to a callback.

After my mild frustration with AASKI, Riverbed, Fidelis, and Cisco, I stumbled upon a small company in Italy called Ntop. Ntop makes a range of packet capture appliances using their own open-source software and off-the-shelf hardware. The Ntop appliances are way cheaper than any other vendors' offerings and seem to offer a lot of features. In addition, their CEO and chief engineer responded to my initial email.

So basically, it looks like the smaller the company is and the more specialized the product is, the better their pre-sales assistance is. Klos and Ntop were both great, and I now have a difficult decision to make on which company's product will be the better value for me.

I learned a few things in this search effort, starting with the fact that I have to lower my expectations when speaking with large companies. I understand that a small customer ordering a single unit is not a priority for large companies, but the questions asked should indicate that I'm not kicking tires here. If I don't buy the product, at least I can talk about their product with competitors like I've been doing with the Klos PacketVault. In any event, this was an interesting effort. I'd like to thank Klos Technologies and Ntop for being on top of their game in pre-sales.

I know so much more now about the industry and the various offerings. I'm almost embarrassed, but that was my perception back then.

Commercially available capture solutions

The Riverbed NetShark

I participated in a conference call/screen share with Riverbed about the NetShark product. It was pretty good, but the product is made more for network diagnostics than traffic recording. Also, it really is a ton of money for what you get.

The Klos Technologies PacketVault

Klos Technologies is a small company on the East Coast. Their product is great, no joke. They also had amazing pre-sales assistance. They even offered to let me try a demo unit at my own office. But I had to turn them down because I knew I couldn't afford their product. (I signed a NDA preventing me from sharing their price list, FYI, but I can say that their prices are totally reasonable for what you get).

The nTop nBox

Continuing my search, I found a really promising solution from the makers of nTop. They call it the nBox. I have no doubt this product is also amazing, but as with the Klos PacketVault, I knew I couldn't afford it. (I also signed a NDA with nTop preventing me from disclosing their prices. Their prices are reasonable).

Fidelis

After I posted my rant on LinkedIn, the guys at Fidelis called me and talked about their products. It was a great conversation, but I realized that their target market is Fortune 500 companies and I don't have a million dollars to spend. Anyway, the conversation concluded with them referring me to Savvius, formerly known as WildPackets. Savvius makes the industry standard traffic analysis tool, OmniPeek Enterprise. Had I not kept after Fidelis, I probably wouldn't have found Savvius. So I would like to thank Noel at Fidelis for pointing me in the right direction.

Savvius

These guys put out OmniPeek Enterprise, which is like Wireshark times ten. I had the honor of attending a workshop put on by Savvius in which they let us install full copies of OmniPeek Enterprise and play around with it. We went through several exercises to teach us about its capabilities. Seriously, that was a great workshop, and I highly recommend OmniPeek Enterprise. The only downside of OmniPeek is that it requires Windows to run. But you could probably run it via CrossOver on a Mac or Linux machine, so that's on my list of things to try.

The Vision

Finally, it occurred to me that I could just build my own capture appliance. There are several reasons for this, starting with the fact that no other solution out there offers exactly what I want to do: leverage the capabilities of each capture utility to generate a coherent picture of what is going on in a network. With a powerful enough server, I could run tcpdump, ntop, and Snort at the same time and use each one for what it does best. Building it myself is the only solution that is affordable, too.

My experience with GDMS PitBull

In 2014, I was pursuing a different project called Rællic. It was going to be a secure cyber communications platform to prevent the government from reading the contents of messages exchanged using the service. Unfortunately, there were many other solutions cropping up during this post-Snowden time period, including the app that became Signal, and I decided I could not compete. But that is another story. What is relevant here is that I had the opportunity to test and evaluate PitBull from General Dynamics C4S (now GD Mission Systems). I had a great experience with this product, due in large part to the excellent service provided by PitBull reseller/installer CodeWeavers. In 2016 it really hit me that I could offer something no one else does: a packet capture appliance running a trusted operating system with multi-level security. In other words, a system that would let companies or government agencies capture on multiple interfaces at different security levels. So if they have a classified connection they want to tap, and an unclassified connection they want to tap, they can do it on the same machine without any other hardware or software. Honestly, this is brilliant and as far as I can tell, not offered by any competitor. So I decided to aggressively market the multi-level security version of The Vision and have it certified by NSA's CSfC program.

The Vision enters production

I have the first unit, which is a Vision Janus (dual-boots RHEL and Windows Server), monitoring my personal network. This is also my test platform. All I have to do is copy the configuration-related files to any new system I build, and I'm in business. The first ad is going in Signal magazine in the August 2016 issue :)

How to build your own gigabit packet capture appliance

Step One: acquire appropriate hardware

At a minimum, you will need a decent server with at least two gigabit ethernet interfaces. However, for the best result you're going to want six ethernet interfaces: one for connecting to the machine, one in reserve, and four hooked up to taps (send and receive are on separate wires, so you can tap two connections with four ports). Intel makes a great 4x gigabit ethernet PCI card and Viavi makes a great gigabit tap. I suggest buying two taps so you can tap the connection before your firewall and after your firewall in order to analyze what traffic is getting through the firewall. You can either build your own server or order one from a systems integrator. I personally would recommend King Star Computer in Sunnyvale, as they are great.

Step Two: install and configure the OS

Although this is debatable, I like using RHEL because all of the updates are handled through RedHat's subscription service. It makes sense to offload the security updates to a skilled vendor. But if you want to roll your own solution, CentOS or Fedora will be adequate. Install tcpdump with libpcap, gulp, ntop (optional), and Snort. You could certainly build something good with BSD, as well. In fact, if you are running other than top-of-the-line hardware, I would recommend BSD because its network stack seems to outperform RHEL, at least in my testing.

Step Three: bridge send and receive into one connection

RHEL includes the standard Linux brctl utility. If you have tapped a connection using a typical gigabit tap, you have send on one wire and receive on another wire. Type the following as root or with sudo to bridge send and receive into one connection that you can capture with a single instance of your preferred capture utility:

ifconfig
[note the interface numbers. Here, I use eth5 and eth4.]
brctl addbr bridge0
brctl addif bridge0 eth5
brctl addif bridge0 eth4

You will need to execute these commands every time you start your server unless you make the bridge persistent, which is beyond the scope of this article.

Once the bridge is added, you can capture on it like any other interface. Here are some examples of commands:

tcpdump -i bridge0 -nn -s0 -Z apache -vvv -C 100000000 -W 100 -w /media/RedHat Data/apachecaps/foo.cap
Capture on bridge0, don't perform DNS lookups, capture complete packets, execute as user apache, maximum verbosity, 100 megabyte capture files, keep the most recent 100 capture files, and output in binary format for use with OmniPeek or Wireshark. This is the standard command I use, although I will change the buffer size if the network is expected to be saturated. If you want to capture in the background with this set of parameters, issue the same command with nohup. Be sure to redirect all of the output to /dev/null or a file so that you don't interrupt the command when logging out. It took me months to figure out how to do this, so you get the benefit of my hard work:
nohup sudo tcpdump -i bridge0 -nn -s0 -Z apache -vvv -C 100000000 -W 100 -w /media/RedHat\ Data/apachecaps/foo.cap > nohup.out 2>/dev/null < /dev/null &

Step Four: configure the sudoers file

If you want a web-based interface, you need to give the apache user permission to capture on the bridged interface. The easiest way to do this is to add a command alias to the sudoers file so that the apache user doesn't have to enter a password when using tcpdump or some other command. Yes, I am fully aware that this is a security risk. You should not be doing this on a machine that is exposed to the internet anyway, at least without some countermeasures that are essentially trade secrets. With that warning, I will show you how to do it. Your sudoers file should look like this:

Cmnd_Alias TCPDUMP_APACHE = /usr/sbin/tcpdump
apache	ALL=(ALL)	NOPASSWD:TCPDUMP_APACHE
This lets apache execute only one command, tcpdump, as root. And it does not require a password.

Step Five: set up some web scripts



This is The Vision web interface version 0.1. Lots of work left to do! This is a good start, though.

Step Six: success!

If you followed the steps here exactly right, you should be capturing at will on a bridged, tapped connection and well on your way to doing cool stuff. Thanks for reading.

Lessons learned

  1. Capturing a saturated gigabit network between two BSD-based devices results in packet loss on a Linux-based capture appliance.
  2. There is no need to assign an IP address to a tapped connection.
  3. Ethernet interfaces see packets even when they are "down" and have no IP address.

Prices

I could build you The Vision with RHEL and 24 TB of storage for around $10,000 depending on the specific configuration. The PitBull edition is going to run you about $30,000 plus yearly support charges due to the cost of PitBull, which is a specialty item. Both of these prices are competitive with all the other solutions I have looked at. If you want to build your own to save a couple thousand dollars off those prices, I wish you the best of luck. Please email me to let me know how well this guide worked for you.

10G and 40G networks

It is going to take some special effort to get The Vision working on a 10G or 40G network. I am looking for venture funding to develop "Ultra" versions of The Vision for these faster networks. It isn't just a matter of throwing processor power at the problem; the drivers for the network card have to be upgraded, among other challenges. I look forward to solving those problems and releasing something amazing in the future.

top