6 Testing Methodologies

A topic which comes up regularly, is how to use tcpreplay to test products like intrusion detection/prevention devices (IDS/IPS) and deep inspection firewalls. Generally, I hear people suggest three things:

1. Use security scanners like Nessus
2. Use ``real attacks'' like those generated by Metasploit
3. Use a replay tool like tcpreplay to generate attack traffic

First, let me say that security scanners like Nessus do a really crappy job of testing the effectiveness of IDS/IPS and firewalls. The simple reason is that security scanners don't try to exploit vulnerabilities because it creates problems on the network. IT managers don't like it when their servers start rebooting or routers crash, so scanners use other non-agressive techniques like banner grabbing to find potentially vulnerable systems. Simply put, these non-agressive techniques often look nothing like a real attack.

That leaves generating ``real attacks'' and replay tools.

Advantages of real attacks:

• It's clear when you have a valid test case because the target system is compromised
• Exploit code and attack tools are widely available for many attacks

Disadvantages of real attacks:

• After the test case is run, the target system may be unstable or corrupted, requiring a reboot or re-install
• Generally requires two systems: a target (often running VMWare) and an attacker system
• Installing, configuring and managing various operating systems and applications to attack is a lot of work
• Difficult to automate test cases since there is no standardized interface to these tools
• You have to be careful about trojaned exploit code or worms which escape your lab

Advantages of replay tools:

• Since both the victim and attacker are virtual, there is no need to reboot/re-install systems after each test
• A complete test bed requires only a single system with two NIC's
• Once you have a library of pcap files, there is virtually zero management overhead
• Replay tools provide a common interface to emulating any attack against any OS/application making automation simple
• Pcap files are not executable, so trojans and escaping worms aren't an issue

Disadvantages of replay tools;

• There are trust issues regarding pcap files. Are you 100% sure that pcap file is correct (not corrupted, doesn't have truncated packets, actually contains the valid exploit)
• There are few publicly available pcap's which contain attacks useful for testing so you must create your own