Farsight's Network Message, Volume 2: Introduction to nmsgtool
By Mike Schiffman
This article is the second in a multi-part blog series intended to introduce
and acquaint the user with Farsight Security's NMSG suite. This article is an
nmsgtool and provides several useful recipes and examples.
Before reading this article, it is recommended that you read
Farsight's Network Message, Volume 1: Introduction to NMSG and have a local installation of
nmsgtool. To get the most from this article and be able to run all of
the examples, an account on Farsight Security's Security Information Exchange (SIE) is recommended. If you
don't have SIE, you can
order it here. This article covers
What is nmsgtool?
To paraphrase the Unix manpage,
nmsgtool is the command line interface to
libnmsg and is a thin wrapper around libnmsg's I/O engine. It controls the
transmission, storage, creation, and conversion of NMSG payloads.
NMSG Inputs and Outputs
nmsgtool program is a single tool for taking inputs from a variety of
different inputs like data streams from the network, capturing data from
network interfaces, reading data from files or even standard input and making
NMSG payloads available to one or more outputs. The outputs are files in binary
or human-readable (ASCII presentation) form, or binary payloads to network
sockets for transport. Without having to create a program for each function,
nmsgtool handles all sorts of data processing including serialization,
fragmentation, compression, striping or mirroring, rolling file outputs, and
executing data processing programs on file outputs.
nmsgtool inputs can take the following forms:
- A file containing binary NMSG data (i.e.: the output of a previously
- A socket that is plumbed to contain binary NMSG data
- Reassembled IP datagrams from a pcap file
- Reassembled IP datagrams from a network interface
- A file containing ASCII presentation data
nmsgtool outputs can take the following forms:
- Binary NMSG data to a file
- Binary NMSG data to a network socket
- ASCII presentation form data to a file (including stdout)
You can specify more than one of each.
The following are a handful of useful
nmsgtool recipes intended to showcase
its functionality and demonstrate different ways to use the tool.
Read Data from SIE
A common use case for SIE customers is to use
nmsgtool to read live SIE data
directly from the wire and write the output to the screen.
$ nmsgtool -C ch212 -c 1 -o -  [2015-02-03 13:35:29.678474903] [2:5 SIE newdomain] [a1ba02cf]   domain: s47rbh.xyz. time_seen: 2015-02-03 13:33:20 rrname: s47rbh.xyz. rrclass: IN (1) rrtype: NS (2) rdata: ns1.51dns.com. rdata: ns2.51dns.com.
The above invocation reads a single NMSG payload
[-c 1] from SIE Channel 212
(Newly Observed Domains)
[-C ch212] and emits it to stdout as ASCII
Note that if no outputs are specified, ASCII presentation to stdout
is the default behavior of
nmsgtool. In future examples, it will be omitted.
Behind the scenes
nmsgtool uses a configuration file called
nmsgtool.chalias that contains channel number to IP address/UDP port
mappings. When a channel number is specified on the command line,
looks it up in the
nmsgtool.chalias file and listens on the specified
The captured NMSG datagram header is the emitted first. Breaking this down we have the following individual fields:
: The message size in bytes
[2015-02-03 13:35:29.678474903]: A UTC timestamp with nanosecond resolution
[2:5 SIE newdomain]: Vendor and message ID, vendor and message type
[a1ba02cf]: The source identifier (optional)
: The operator code (optional)
: The group code (optional)
The message payload is a combination of key-value pairs. They follow a
schema defined by the vendor and message type (in the above example, the
SIE and the message type is
includes dynamically loadable modules that enable it to present this
data as you see above and also enable NMSG-based programs or scripts load the
key-value pairs into structures. These concepts will be more rigorously
explained in future NMSG articles.
Read and Write Binary NMSG Files
Another common use case for SIE customers is to read messages from SIE into a local binary file for later analysis.
$ nmsgtool -C ch208 -c 100000 -w ch208.nmsg $ stat -c "%n %s" ch208.nmsg ch208.nmsg 15246581 $ nmsgtool -r ch208.nmsg -c 1  [2015-02-01 00:07:53.596907788] [2:1 SIE dnsdedupe] [a1ba02cf]   type: EXPIRATION count: 2 time_first: 2015-01-31 07:29:37 time_last: 2015-01-31 07:29:37 bailiwick: <redacted> rrname: <redacted> rrclass: IN (1) rrtype: A (1) rrttl: 43200 rdata: <redacted>
The above invocation reads reads 100,000 NMSG payloads
[-c 1000000] from
SIE Channel 208 (Passive DNS, deduplicated, verified, in-bailiwick)
[-C ch208] and emits them as binary NMSGs to a file
The result is 15 megabytes.
That binary file is then read
[-r ch208.nmsg] and a single NMSG payload
[-c 1] containing a
dnsdedupe message is emitted to stdout as ASCII
nmsgtool can compress binary payload output (to a file or for emission
across the network) using
zlib compression (the same algorithm used by the
ubiquitous gzip tool). To see an example of the on-disk
storage benefit compression can offer, we compress the data captured in the
$ nmsgtool -r ch208.nmsg -w ch208z.nmsg -z $ stat -c "%n %s" ch208-z.nmsg ch208-z.nmsg 6428829
The above invocation reads the binary file from the previous example
[-r ch208] and writes a new file
[-w ch208z.nmsg], compressing each
[-z]. The resultant file is just over six megabytes for a 58%
decrease in file size. It is important to note that the compression is
performed per payload, not across the entire file.
Kicker Scripts and Output File Rolling
Another useful feature
nmsgtool offers is the ability to perform automatic
file rolling (rotation) based on timer expiry or payload count. Additionally,
the user can specify a kicker command to run on output files after rotation.
Consider the following simple shell script:
#!/bin/sh echo "$1: " `nmsgtool -r $1 | grep "[2:1 SIE dnsdedupe]" | wc -l`
The above script,
count.sh, counts the number of
dnsdedupe payloads from a
binary NMSG file. The script is invoked in the following example:
$ nmsgtool -C ch202 -w ch202 -t 2 -z -k count.sh ./ch202.20150202.0110.1422839406.364843292.nmsg: 3099450 ./ch202.20150202.0110.1422839408.013136741.nmsg: 3384114 ./ch202.20150202.0110.1422839410.024261700.nmsg: 3090827 ./ch202.20150202.0110.1422839412.024284315.nmsg: 3100505 ./ch202.20150202.0110.1422839414.033887391.nmsg: 3026627 ./ch202.20150202.0110.1422839416.014162500.nmsg: 3208181
The above invocation reads payloads from SIE Channel 202 (Raw Passive DNS)
[-C ch202] and writes compressed payloads
[-z] to a binary file
[-w ch202]. Every two seconds
[-t 2] the file is closed, rotated, and the
kicker script is run on the output file
[-k count.sh]. The output from
count.sh invocation is the filename followed by the number of NMSG
Transfer NMSGs across the network
nmsgtool can be used to transfer NMSGs across an IPv4 or IPv6 network to
either a unicast or broadcast address.
For this example, we instantiate two
nmsgtool sessions on separate hosts. On
the receiving host, we run
nmsgtool as follows:
$ nmsgtool -l 10.0.1.52/9430
The above invocation listens for NMSGS on a network socket connected to
10.0.1.52 on UDP port 9430
[-l 10.0.1.52/9430]. When NMSGs appear, they will
be emitted as ASCII presentation data to stdout.
On the sending host, we run
nmsgtool as follows:
$ nmsgtool -r ch202.20150202.0110.1422839406.364843292.nmsg -c 2 -s 10.0.1.52/9430
The above invocation reads two payloads
[-c 2] from the binary NMSG file
created in the previous example
[-r ch202...]. They are written to the
network destined for 10.0.1.52 on UDP port 9430
On the receiving host we see the following output:
 [2015-02-02 01:08:21.902736000] [1:9 base dnsqr] [e9b019b8]   type: UDP_QUERY_RESPONSE query_ip: <redacted> response_ip: <redacted> proto: UDP (17) query_port: 31211 response_port: 53 id: 7644 qname: <redacted> qclass: IN (1) qtype: AAAA (28) rcode: NOERROR (0) delay: 0.182413 udp_checksum: ABSENT [...]  [2015-02-02 01:08:22.095911000] [1:9 base dnsqr] [e9b019b8]   [...]
Two presentation format
dnsqr NMSGs (redacted and cropped for publication)
are emitted to stdout.
As a side-note, the sender has options to tune the network performance including setting the NMSG container maximum transmission unit size (note this is distinct from IP MTU), buffering, and rate limiting.
Payload Striping vs Mirroring
$ nmsgtool -C ch211 -c 100 -o - -s 10.0.1.52/9430 --mirror  [2015-02-03 08:50:19.277158975] [2:5 SIE newdomain] [a1ba02cf]   [...]
The above invocation reads 100 payloads
[-c 100] from SIE Channel 211 (Newly
Active Domains) and mirrors
[--mirror] across two outputs, one ASCII
presentation to stdout
[-o -] and one network socket destined for
10.0.1.52 on UDP port 9430
Input from a Network Interface or Pcap File with BPF filtering
Perhaps you'd like to create your own NMSG stream sourced from live network
traffic. More specifically, you want only DNS traffic. To this end, you can
nmsgtool to read IP datagrams directly from a network interface or
a pcap file. Additionally, a
BPF can be specified to
winnow packets. When receiving input from a network interface or a pcap file,
nmsgtool requires the user to set the vendor and message type so it knows how
to encode each payload.
$ nmsgtool -i eth1 -V base -T dnsqr -b "udp 53"  [2010-05-09 05:08:54.951124000] [1:9 base dnsqr]    [...]
The above invocation reads data from a network interface
[-i eth1] and
winnows those packets to just UDP port 53
[-b "udp 53"] and encodes this data
[-V base] and
[-T dnsqr] and emits as ASCII presentation
data to stdout.
Input from a pcap file is syntactically similar, just substitute
[-p example.pcap] for
The next article in the NMSG series will examine low-level NMSG implementation details such as header composition and data encoding. Future articles will introduce the programming APIs.
Mike Schiffman is a Protocol Legerdemainist for Farsight Security, Inc.
Read the next part in this series: Network Message, Volume 3: Headers and Encoding