Farsight's Network Message: Loss Tracking Explained




This article explains Farsight's Network Message (NMSG) loss tracking feature.

To get the most from this article, it is recommended that you be comfortable with the material from the following Farsight Security Blog articles:


NMSG is an extensible container wire and file format for storing and transmitting blobs of data with support for dynamic message types, compression, fragmentation, sequencing, and rate limiting. We will cover NMSG's data sequencing feature.

NMSG's data sequencing feature, referred to as "seqsrc" was added in version 0.8.0 to provide a mechanism for tracking container loss across UDP-socket based NMSG sessions. This is especially useful when a user wants to track packet loss in high-throughput NMSG sessions.

The seqsrc feature is available in nmsgtool and exposed via the C API, examples of both are given below.

How it Works

Seqsrc loss tracking is performed by nmsg on the receiver side on a per-container basis (as opposed to per-payload). Each NMSG container carries with it a sequence number and sequence_id. The sequence number is an unsigned 32 bit number that serves as – you probably guessed – the container's sequence number. The sequence_id is an unsigned 64 bit nonce used as an opaque pseudo-random cookie to prevent inadvertent sequence space reuse. From the nmsg API documentation, the sequence_id is:

a randomized ID number identifying the sequence number space that the 'sequence' parameter exists in. This ID number is used by NMSG consumers to disambiguate multiple disparate sequences of NMSG containers when consuming a multiplexed stream from multiple sources. This ID number should be generated by a cryptographically secure PRNG such as the one provided by nmsg_random_init() to minimize the possibility of collisions.

Indeed, during the development of seqsrc and prior to the addition of a sequence_id, Farsight Security did actually see birthday collisions due to the very high UDP port number reuse on high bit-rate SIE channels.

To the NMSG consumer, seqsrc is implemented as a series of separate and independent sequence number spaces. Each of these sequence number spaces is relevant only to the NMSG connection that it references (there should be one per nmsg_output_t object on the sender-side), and only for the time that it exists. In a lossless system, this value should increase monotonically without gaps.

NMSG will partition a new seqsrc space using the IP source address, IP protocol, source port, and sequence_id to create a tuple that gets its own sequence space. As such, a seqsrc key is defined as:

struct nmsg_seqsrc_key {
       uint64_t                        sequence_id;
       sa_family_t                     af;
       uint16_t                        port;
       union {
               uint8_t                 ip4[4];
               uint8_t                 ip6[16];

Internally, Farsight Security uses seqsrc to monitor container loss in real-time on its Security Information Exchange (SIE).

Loss Tracking with nmsgtool

The reference implementation of nmsg, nmsgtool, supports seqsrc when invoked with (at least) four levels of debug (-dddd).

To see seqsrc in action, we'll use a corpus of channel 202 NMSGs containing just over 2,000,000 payloads (we use the nmsgpcnt program to count containers and payloads) and send these, from one unbuffered nmsgtool instance, over the loopback interface, to another listening nmsgtool instance.

The nmsg file we'll use:

$ ls -l 202-2000000.nmsg
-rw-r--r--  1 mschiffm  staff  753800034 Mar  7  2015 202-2000000.nmsg
$ nmsgpcnt-fsi 202-2000000.nmsg
containers:    720
payloads:      2000481

First, we instantiate an nmsg listener on loopback/UDP port 8888 with the proper debug level and instruct it to write binary nmsgs to file:

$ nmsgtool -l -dddd -w foo.nmsg
...<debug messages omitted>...

Next, we fire off an nmsgtool instance with the --unbuffered option which will send each payload in its own container, as fast as it can (over the loopback interface on my several year old MacBook Pro this should result in some container loss):

$ nmsgtool -r 202-100000payloads.nmsg -s --unbuffered

We then quit the initial nmsgtool session (via ctrl-c) and inspect the debug output:

^Cnmsgtool: signalled break
nmsg_io: iothr=0x7faf8c6002c0 count_nmsg_payload_in=1989299
_input_seqsrc_destroy: input=0x7faf8c502e50 count_recv=1989299 count_drop=11182 (0.0056)
nmsg_io: io=0x7faf8c502fe0 count_nmsg_payload_out=1989299

nmsgtool reveals that of the 2,000,481 containers sent, it received 1,989,299 and 11,182 were lost.

Loss Tracking with the nmsg C API

The nmsg C library exposes a very simple API to enable, disable, and access seqsrc data. The following code snippet is taken from the working sample nmsg program nmsgjsonseqsrc:

First, we explicitly enable seqsrc verification on a previously instantiated nmsg input object (note that as of version 0.12.1, nmsg enables it by default):

        /* Instruct io engine to enable container loss tracking. */
        res = nmsg_input_set_verify_seqsrc(input, true);
        if (res != nmsg_res_success)
                fprintf(stderr, "nmsg_input_set_verify_seqsrc(): %s\n",
                return (EXIT_FAILURE);
        printf("verify seqsrc enabled\n");

After running the nmsg io engine and (presumably) processing many NMSGs, we cull and emit the received and dropped container count:

        /* Retrieve the total number of NMSG containers that have been
         * received since the nmsg_input_t object was created. */
        res = nmsg_input_get_count_container_received(input, &count);
        if (res != nmsg_res_success)
                        "nmsg_input_get_count_container_received(): %s\n",
                return (EXIT_FAILURE);
        printf("\nrecevied %llu containers\n", count);

        /* Retrieve the total number of NMSG containers that have been
         * dropped since the nmsg_input_t object was created. */
        res = nmsg_input_get_count_container_dropped(input, &count);
        if (res != nmsg_res_success)
                        "nmsg_input_get_count_container_dropped(): %s\n",
                return (EXIT_FAILURE);

Using the same NMSG file and unbuffered sending invocation as in the example above, we see the following result:

$ ./nmsg-seqsrc -l
nmsg JSON emitter / seqsrc example program
nmsg initialized
nmsg io engine initialized
nmsg socket input initialized
verify seqsrc enabled
socket input added to io engine
callback initialized
callback added to io engine
entering io loop, <ctrl-c> to quit...
...<newline delimited json output omitted>...
received 1041381 containers
dropped 959079 containers

nmsgjsonseqsrc reveals that of the 2,000,481 containers sent, it received 1,041,381 and 959,100 were lost (we lost more here because the nmsgjsonseqsrc program deserializes each incoming NMSG as JSON, a more computationally expensive operation than simply writing binary NMSGs to a file).

Mike Schiffman is a personal packet shopper for Farsight Security, Inc.