Next: Other Resources
This document is still in the process of being re-written due to the significant CLI and configuration file changes between versions 2.x and 3.x. For the definative source of configuration options, please see the tcpprep, tcprewrite, tcpreplay and tcpbridge man pages.
Tcpreplay is a suite of utilities for UNIX systems for editing and replaying network traffic which was previously captured by tools like tcpdump and ethereal. The goal of tcpreplay is to provide the means for providing reliable and repeatible means for testing a variety of network devices such as switches, router, firewalls, network intrusion detection and prevention systems (IDS and IPS).
Tcpreplay provides the ability to classify traffic as client or server, edit packets at layers 2-4 and replay the traffic at arbitrary speeds onto a network for sniffing or through a device.
Some of the advantages of using tcpreplay over using ``exploit code'' are:
The goal of this manual is to provide an idea of what tcpreplay and it's utilities can do. It is not however intended to be a complete document which covers every possible use case or situation. It is also very much a work in progress and is far from complete and has numerous errors since a lot of things have changed since tcpreplay 2.x. It is expected that most of these issues will be ironed out before the offical 3.0 release is made. You should keep in mind the following conventions when reading this document:
If you still have a question after reading the Tcpreplay manual, man pages and FAQ, please contact the Tcpreplay-Users <tcpreplay-users@lists.sourceforge.net> mailing list. Note that if you ask a question which has clearly been covered in either the manual or FAQ, you will most likely be told to RTFM. Also, please try to explain your problem in detail. It is very difficult and fustrating to get requests from people seeking help who only provide vague and incomplete information.
I've tried to keep this document up to date with the changes in tcpreplay, but occasionally I get too busy, make a mistake or just forget something. If you find anything in this document which could be improved upon, please let me know.
The source code is available as a tarball on the tcpreplay homepage: http://tcpreplay.sourceforge.net/ I also encourage users familiar with Subversion to try checking out the latest code as it often has additional features and bugfixes not yet found in the offical releases.
Two easy steps:
# make install
You may also choose to run:
To replay a given pcap as it was captured all you need to do is specify the pcap file and the interface to send the traffic out interface 'eth0':
You can also replay the traffic at different speeds then it was originally captured4.
Some examples:
Using the loop flag you can specify that a pcap file will be sent two or more times5:
To replay the sample.pcap file 10 times:
There are a number of ways you can edit packets stored in a pcap file:
Anything other then just replaying packets at different speeds requires additional work and CPU cycles. While older versions of tcpreplay allowed you to do many of these calculations while replaying traffic, it had a negative effect on the overall throughput and performance of tcpreplay. Hence, these secondary features have been placed in two utilities:
Both tcpreplay and tcprewrite process a single pcap file and generate output. Some features, such as rewriting IP or MAC addresses or sending traffic out two different interfaces, require tcpreplay and tcprewrite to have some basic knowledge about which packets were sent by ``clients'' and ``servers''. Such classification is often rather arbitrary since for example a SMTP mail server both accepts inbound email (acts as a server) and forwards mail to other mail servers (acts as a client). A webserver might accept inbound HTTP requests, but make client connections to a SQL server.
To deal with this problem, tcpreplay comes with tcpprep which provides a number of manual and automatic classification methods which cover a variety of situations.
The easiest way to split clients and servers is to let tcpprep do the classification for you. Tcpprep examines the pcap file for TCP three-way handshakes, DNS lookups and other types of traffic to figure out which IP's mostly act like clients and which mostly act like servers. There are four different automatic modes that you can choose between:
| TCPPREP AUTOMATIC ROUTER MODE PROCESS
STEP 1: Categorize Clients, Servers and Unknowns |
| STEP 3: Unknowns Now Marked as Clients and Servers
|
Classifying clients and servers in automatic mode is as easy as choosing a pcap file, an output ``tcpprep cache file'' and the mode to use:
Sometimes, you may not want to split traffic based on clients and servers. The alternative to using on of the automatic modes in this case, is to use one of the manual modes. One manual way of differentiating between clients and servers using tcpprep is by specifying a list of networks in CIDR notation which contain ``servers''. Of course the specified CIDR netblocks don't have to contain
Tcpreplay can also split traffic so that each side of a connection is sent out a different interface6. In order to do this, tcpreplay needs the name of the second interface (-j) and a way to split the traffic. Currently, there are two ways to split traffic:
Sometimes, you want to do some post-capture filtering of packets. Tcpreplay let's you have some control over which packets get sent.
Using the limit packets flag (-L) you can specify that tcpreplay will only send at most a specified number of packets.
If you want to skip the beginning of a pcap file, you can use the offset flag (-o) to skip a specified number of bytes and start sending on the next packet.
Occasionally, you might find yourself trying to replay a pcap file which contains packets which are larger then the MTU for the sending interface. This might be due to the packets being captured on the loopback interface or on a 1000Mbps ethernet interface supporting ``jumbo frames''. I've even seen packets which are 1500 bytes but contain both an ethernet header and trailer which bumps the total frame size to 1518 which is 4 bytes too large.
By default, tcpreplay will skip these packets and not send them. Alternatively, you can specify the -T flag to truncate these packets to the MTU and then send them. Of course this may invalidate your testing, but it has proven useful in certain situations. Also, when this feature is enabled, tcpreplay will automatically recalculate the IP and TCP, UDP or ICMP checksums as needed. Example:
It's not always necessary to write packets to the network. Since tcpreplay has so many features which modify and select which packets are sent, it is occasionally useful to save these changes to another pcap file for comparison. Rather then running a separate tcpdump process to capture the packets, tcpreplay now supports output directly to a file. Example:
You can also split traffic into two files by using -W <2nd output file>.
New to version 2.0 is the ability to extract the application layer data from the packets and write them to a file. In the man page, we call this ``data dump mode'' which is enabled with -D. It's important to specify -D before -w (and -W if you're splitting data into two files). Example:
You can now replay live traffic sniffed on one network interface and replay it on another interface using the -S flag to indicate sniff mode and the appropriate snaplen in bytes (0 denotes the entire packet). You can also enabling bi-directional traffic using the bridge mode flag: -b.
NOTE: It is critical for your sanity (and to prevent your murder by your network administrators) that the input interface and the output interface be on separate networks and additionally that no other network devices (such as bridges, switches, routers, etc) be connecting the two networks, else you will surely get a networkstorm the likes that have not been seen for years.
There are about as many different capture file formats as there are sniffers. In the interest of simplicity, tcpreplay only supports libpcap9. If you would like to replay a file in one of these multitude of formats, the excellent open source tool Ethereal easily allows you to convert it to libpcap. For instance, to convert a file in Sun's snoop format to libpcap, issue the command:
A common question on the tcpreplay-users list is how does one replay the client side of a connection back to a server. Unfortunately, tcpreplay doesn't support this right now. The major problem concerns syncing up TCP Seq/Ack numbers which will be different. ICMP also often contains IP header information which would need to be adjusted. About the only thing that could be easy to do is UDP, which isn't usually requested.
This is however a feature that we're looking into implementing in the flowreplay utility. If you're interested in helping work on this feature, please contact us and we'd be more then happy to work with you. At this time however, we don't have an ETA when this will be implemented, so don't bother asking.
If the tcpdump binary is installed on your system when tcpreplay is compiled, it will allow you to decode packets as they are sent without running tcpdump in a separate window or worrying about it capturing packets which weren't sent by tcpreplay.
If you ever want to send traffic to another device on a switched LAN, you may need to change the destination MAC address of the packets. Tcpreplay allows you to set the destination MAC for each interface independently using the -I and -J switches. As of version 2.1.0, you can also specify the source MAC via -k and -K. Example:
Occasionally, it is necessary to have tcpreplay rewrite the source and destination IP addresses, yet maintain the client/server relationship. Such a case might be having multiple copies of tcpreplay running at the same time using the same pcap file while trying to stress test firewall, IDS or other stateful device. If you didn't change the source and destination IP addresses, the device under test would get confused since it would see multiple copies of the same connection occurring at the same time. In order to accomplish this, tcpreplay accepts a user specified seed which is used to generate pseudo-random IP addresses. Also, when this feature is enabled, tcpreplay will automatically recalculate the IP and TCP, UDP or ICMP checksums as needed. Example:
Occasionally, it is necessary to replay traffic which has been truncated by tcpdump. This occurs when the tcpdump snaplen is smaller then the actual packet size. Since this will create problems for devices which are expecting a full-sized packet or attempting checksum calculations, tcpreplay allows you to either pad the packet with zeros or reset the packet length in the headers to the actual packet size. In either case, the IP and TCP, UDP or ICMP checksums are recalculated. Examples:
Starting in the 2.0.x branch, tcpreplay can replace the existing layer 2 header with one of your choosing. This is useful for when you want to change the layer 2 header type or add a header for pcap files without one. Each pcap file tells the type of frame. Currently tcpreplay knows how to deal with the following pcap(3) frame types:
Potential uses for this are to add a layer 2 header for DLT_RAW captures or add/remove ethernet tags or QoS features.
Tcpdump uses a special frame type to store captures created with the ``-i any'' argument. This frame type uses a custom 16 byte layer 2 header which tracks which interface captured the packet and often the source MAC address of the original ethernet frame. Unfortunately, it never stores the destination MAC address and it doesn't store a source MAC when the packet is captured on the loopback interface. Normally, tcpreplay can't replay these pcap files because there isn't enough information in the LINUX_SLL header to do so; however two options do exist:
Pseudo-NAT allows the mapping of IP addresses in IPv4 and ARP packets from one subnet to another subnet of the same or different size. This allows some or all the traffic sent to appear to come from a different IP subnet then it actually was captured on.
The mapping is done through a user specified translation table comprised of one or more source and destination network(s) in the format of <srcnet>/<masklen>:<dstnet>/<masklen> deliminated by a comma. Mapping is done by matching IP addresses to the source subnet and rewriting the most significant bits with the destination subnet. For example:
tcpreplay -intf1=eth0 -N 10.100.0.0/16:172.16.10.0/24 sample.pcap
would match any IP in the 10.100.0.0/16 subnet and rewrite it as if it came from or sent to the 172.16.10.0/24 subnet. Ie: 10.100.5.88 would become 172.16.10.88 and 10.100.99.45 would become 172.16.10.45. But 10.150.7.44 would not be rewritten.
For any given IP address, the translation table is applied in order (so if there are multiple mappings, earlier maps take precedence) and occurs only once per IP (no risk of an address getting rewritten a second time).
Pseudo-NAT also works with traffic splitting (using two interfaces or output files) but with a few important differences. First you have the option of specifying one or two pseudo-NAT tables. Using a single pseudo-NAT table means that the source and destination IP addresses of both interfaces are rewritten using the same rules. Using two pseudo-NAT tables (specifying -N <Table1> -N <Table2>) will cause the source and destination IP addresses to be rewritten differently for each interface using the following matrix:
| Out Primary Interface | Out Secondary Interface
Src IP |
While seemingly a bit confusing, this feature provides a number of interesting possibilities such as the ability to rewrite the IP headers of packets in the case where traffic is captured on the loopback interface (and the source and destination address is always 127.0.0.1) so that tcpreplay can make it look like two different systems are talking to each other (you'll probably also need to specify the source and destination MAC addresses via -I, -J, -k and -K).
While pseudo-NAT provides a great deal of flexibility, it is often more complicated then is necessary for testing of inline devices. As a simplier alternative, tcpreplay supports the concept of rewriting all traffic to so that it appears to be between two IP addresses:
Since a number of tcpreplay's packet editing functions require splitting traffic between client and servers, one problem that may arrise is needing to edit packets but still output to a single interface or file. The solution to this is to use the one output option -O which causes packets to be processed as if they will be split between the interfaces/files, but then always go out the primary interface or file. Note that even though only one interface/file will be written to, both -i and -j must be specified; although they can be the same physical interface.
Tcpreplay can send traffic out two network cards, however it requires the calculations be done in real-time. These calculations can be expensive and can significantly reduce the throughput of tcpreplay.
Tcpprep is a libpcap pre-processor for tcpreplay which enables using two network cards to send traffic without the performance hit of doing the calculations in real-time.
Tcpprep has three basic modes which require the user to specify how to split traffic.
Tcpprep supports the same CIDR mode that tcpreplay supports using the -cidr flag. Additionally, tcpprep also supports regex(7) regular expressions to match source IP addresses using the -regex flag.
Tcpprep does an initial pass over the libpcap file to build a binary tree (one node per IP). For each IP, it keeps track of how many times it was a client or server. It then does a second pass of the file using the data in the tree and the ratio to determine if an IP is a client or server. If tcpprep is unable to determine the type (client or server) for each and every packet, then auto/bridge mode will fail. In these cases, it is best to use a different auto mode.
Tcpprep does the same first pass as Auto/Bridge mode. It then trys to convert the binary tree into a list of networks containing the servers. Finally it uses the CIDR mode with the list of server networks in a second pass of the libpcap file. Unlike auto/bridge mode, auto/router mode can always successfully split IP addresses into clients and servers.
Tcpprep uses the following methods in auto/router and auto/bridge mode to determine if an IP address is a client or server:
Since a system may send traffic which would classify it as both a client and server, it's necessary to be able to weigh the traffic. This is done by specifying the client/server ratio (-R) which is by default set to 2.0. The ratio is the modifier to the number of client connections. Hence, by default, client connections are valued twice as high as server connections.
Tcpprep supports the same -include and -exclude options to selectively send or drop packets.
Just run:
In versions of tcpprep >= 2.1.0, you can specify a comment to be embeded in the tcpprep cache file. Comments are user specified and automatically include the command line arguments passed to tcpprep.
Each of the applications in the tcpreplay suite offers the choice of specifying configuration options in a config file in addition to the traditional command line. Each command line option has an equivalent config file option which is listed in the man page. To specify the configuration file you'd like to use, use the -load-opts=<filename> option.
Configuration files have one option per line, and lines beginning with the pound sign (#) are considered comments and ignored. An example config file follows:
------BEGIN CONFIG FILE-------
# send traffic out 'eth0'
intf1 eth0
# loop 5 times
loop 5
# send traffic 2x as fast
multiplier 2
-------END CONFIG FILE--------
You would then execute:
------BEGIN CONFIG FILE-------
cachefile=example.tcpprep
[TCPREPLAY]
intf1 eth0
intf2 eth1
topspeed
[TCPPREP]
auto=bridge
comment='This cache file was created with a config file'
pcap=sample.pcap
[TCPREWRITE]
infile=sample.pcap
outfile=newsample.pcap
vlan=add
vlan-tag=44
endpoints=10.0.0.1:10.0.1.1
------END CONFIG FILE-------
While tcpreplay is a great way to test NIDS and firewalls, it can't be used to test servers or HIDS since tcpreplay can't connect to a service running on a device. The solution to this problem is flowreplay which instead of sending packets at Layer 2 (ethernet header and up), it can actually connect via TCP or UDP to server and then sends and receives data based upon a pcap capture file created with a tool like Ethereal or tcpdump.
Please note that flowreplay is currently alpha quality and is missing a number of key features.
Put simply, flowreplay opens a socket connection to a service on a target system(s) and sends data over that socket based on the packet capture. Flowreplay has no understanding of the application protocol (like HTTP or FTP) so it is somewhat limited in how it can deal with complicated exchanges between client and server.
Some of these limitations are:
See the flowreplay(8) man page for details.
Regardless of the size of physical memory, UNIX kernels will only allocate a static amount for network buffers. This includes packets sent via the "raw" interface, like with tcpreplay. Most kernels will allow you to tweak the size of these buffers, drastically increasing performance and accuracy.
NOTE: The following information is provided based upon our own experiences or the reported experiences of others. Depending on your hardware and specific hardware, it may or may not work for you. It may even make your system horribly unstable, corrupt your harddrive, or worse.
NOTE: Different operating systems, network card drivers, and even hardware can have an effect on the accuracy of packet timestamps that tcpdump or other capture utilities generate. And as you know: garbage in, garbage out.
NOTE: If you have information on tuning the kernel of an operating system not listed here, please send it to me so I can include it.
The following is known to apply to the 2.4.x series of kernels. If anyone has any information regarding other kernel versions, please let us know. By default Linux's tcpreplay performance isn't all that stellar. However, with a simple tweak, relatively decent performance can be had on the right hardware. By default, Linux specifies a 64K buffer for sending packets. Increasing this buffer to about half a megabyte does a good job:
echo 524287 >/proc/sys/net/core/wmem_default
echo 524287 >/proc/sys/net/core/wmem_max
echo 524287 >/proc/sys/net/core/rmem_max
echo 524287 >/proc/sys/net/core/rmem_default
On one system, we've seen a jump from 23.02 megabits/sec (5560 packets/sec) to 220.30 megabits/sec (53212 packets/sec) which is nearly a 10x increase in performance. Depending on your system and capture file, different numbers may provide different results.
*BSD systems typically allow you to specify the size of network buffers with the NMBCLUSTERS option in the kernel config file. Experiment with different sizes to see which yields the best performance. See the options(4) man page for more details.
As of tcpreplay v1.4, you'll need to have libpcap installed on your system. As of v2.0, you'll need at least version 0.6.0 or better, but I only test our code with the latest version. Libpcap can be obtained on the tcpdump homepage11.
Tcpreplay v1.3 is the last version to support the old libnet API (everything before 1.1.x). As of v1.4 you will need to use Libnet 1.1.0 or better which can be obtained from the Libnet homepage12.
As of 2.0, tcpreplay uses tcpdump (the binary, not code) to decode packets to STDOUT in a human readable (with practice) format as it sends them. If you would like this feature, tcpdump must be installed on your system.
NOTE: The location of the tcpdump binary is hardcoded in tcpreplay at compile time. If tcpdump gets renamed or moved, the feature will become disabled.