Parsing ERF (pcap-like) format files

sblanton has asked for the wisdom of the Perl Monks concerning the following question:

Net::Pcap doesn't understand ERF format files such as those that come from an Endace capture. (I get "unknown file format" from the open_offline sub). ERF is identical to pcap, except that there is an additional timestamp that has a higher resolution than what pcap can provide.

tcpdump can parse ERF files no problem, but it seems that knowledge is in the tcpdump source and not libpcap itself (and therefore not Net::Pcap). Before I download and read the tcpdump source code to find the precise format and write my own parser, does anyone have one they'd like to share or is there a module out there I didn't find?

Update:

Actually tcpdump can not parse the native files that come from Endace resulting in the same error returned by Net::Pcap, which makes sense ('unknown file format'). We discovered that mergecap is changing the files into the libpcap format which can then be read by libpcap/tcpdump/Net::Pcap. mergcap converts from (File type: Endace ERF capture, File encapsulation: Endace Record File) -> (File type: Wiresharek/tcpdump/... - libpcap, File encapsulation: Endace Record File

Comment on Parsing ERF (pcap-like) format files

Replies are listed 'Best First'.
Re: Parsing ERF (pcap-like) format files by Khen1950fx (Canon) on Mar 17, 2011 at 07:27 UTC
As I understand it, if you want to parse the native Endace files, you have to configure libpcap using `./configure --with-dag` [download] It seems that the DAG libraries aren't opensource, and I couldn't find a download link for them. Without them, you can't do a live capture. The DAG Project recommends libtrace. It understands ERF, but, once again, it can't do a live capture with out the DAG libraries.	[reply] [d/l]
Re^2: Parsing ERF (pcap-like) format files by sblanton (Sexton) on Mar 17, 2011 at 12:20 UTC
I've found that encapsulation and file formats are different. We've mislead ourselves on what's happening with the file formats. The `capinfos` program from Wireshark reveals that Endace creates: `File type: Endace ERF capture File encapsulation: Endace Record File` [download] Whereas after being processed through `mergecap` (also from Wireshark), the file format has changed to `File type: Wireshark/tcpdump/... - libpcap File encapsulation: Endace Record File` [download] libpcap can read the latter, but not the former. So, the wireshark programs can read the native Endace file format, but libpcap can not. libpcap can however capture live from DAG's and the code seems to be there (pcap-dag.c). The code from wireshark to read the Endace files is in erf.h, erf.c, but there are so many conditionals in how the specifics of the file can look, that this looks better to be a small C project if I want to avoid manually processing the files before processing with Perl. It would be great to convert this to a perl module, but I just need the data and don't have the time or C->Perl experience to be an open source hero! Thanks for the feedback!	[reply] [d/l] [select]
Re: Parsing ERF (pcap-like) format files by sfd (Initiate) on Mar 18, 2011 at 00:06 UTC
Sounds like you figured it out already. ERF (Endace Record Format) is a proprietary binary format for captured network packets. An ERF File is simply one or more records concatenated, with no file header. This makes it conceptually simpler than pcap files, which it is not directly compatible with. The Wireshark package (including mergecap, editcap, capinfos, tshark, wireshark etc) has the ability to read ERF Files and dissect the records. This functionality is split between wiretap (wtap/erf.c) and a dissector (epan/dissectors/packet-erf.c). Because of this, editcap/mergecap/tshark/wireshark can read in an ERF File and output any format they support writing into, defaulting to pcap. The DLT you get will depend on the tool settings. There is a pcap DLT_ERF specifically for encapsulating ERF records. If you output this type you will have a pcap file of type DLT_ERF where the payload of each pcap 'packet' is a complete ERF record including the header. Alternatively you can use these tools to output a regular DLT format such as DLT_EN10MB for Ethernet. If you are capturing live you can use the native APIs or libpcap. If you are using libpcap by default the ERF records are converted into regular pcap DLTs such as DLT_EN10MB. You can optionally specify DLT_ERF to retain all the extra metadata including the high precision timestamp. When Wireshark tools read a pcap file containing DLT_ERF they discard the pcap wrapper information and use the ERF header preferentially. As mentioned libtrace can also be used to read ERF Files as well as pcap files with a common API.	[reply]
Re^2: Parsing ERF (pcap-like) format files by sblanton (Sexton) on Mar 28, 2011 at 14:32 UTC
Thanks, that clarifies things. It turns out I've gone with capturing over a socket, not a device (which is on another machine), so again libpcap doesn't help. However, I just went byte by byte over the ERF header from the Wireshark gui and have constructed a Net::Format::Layer::ERF module that does the job. I keep looking for an excuse to do more projects in C, but can't justify it because Perl does the job with acceptable performance, is faster to implement and easier to maintain!	[reply]
Re: Parsing ERF (pcap-like) format files by erikh (Initiate) on Nov 26, 2012 at 15:21 UTC
A better solution is to use editcap to convert ERF to PCAP and completely remove the ERF headers in the pcap. There is a good writeup here: http://www.netresec.com/?page=Blog&month=2012-11&post=Convert-Endace-ERF-capture-files-to-PCAP /erik	[reply]