JSON Serialization Support in nmsg

← Blog Home

RSS

By

Introduction

We are pleased to announce that we have added support for JSON serialization to libnmsg in version 0.10.0. This makes it easier than ever for you to consume NMSG data from our Security Information Exchange (SIE). Data can now be directly imported into your Apache Hadoop, Splunk, HP ArcSight, and IBM Security QRadar systems.

It is also easier for you to produce data for publication on SIE without the need of integrating libnmsg into your application. You only need to emit compatible Line Delimited JSON objects that can then be imported by nmsgtool.

The Format

There are four required fields in the JSON schema: time, vname, mname, and message. time is an ISO 8601 timestamp in UTC with nanosecond precision and no time zone designator. vname is a string corresponding to the vendor, mname is the msgtype name of the msg module that should load the message object. The message object contains presentation format fields specific to the msg module. Binary data is encoded in base64 because JSON is a unicode format and does not support the full range of binary characters. Below is an example of a message from SIE channel 204: deduplicated domain names.

{
    "time": "2015-09-02 17:10:00.464105589",
    "vname": "SIE",
    "mname": "dnsdedupe",
    "source": "a1ba02cf",
    "message": {
        "type": "EXPIRATION",
        "count": 1,
        "time_first": "2015-09-02 11:29:14",
        "time_last": "2015-09-02 11:29:14",
        "bailiwick": "191.54.in-addr.arpa.",
        "rrname": "84.152.191.54.in-addr.arpa.",
        "rrclass": "IN",
        "rrtype": "PTR",
        "rrttl": 300,
        "rdata": ["server2.mometrix.com."]
    }
}

Installation

Follow the SIE Installation instructions. You will need to install wdns 0.8.0 or newer, nmsg 0.10.0 or newer, and pynmsg 0.4.0 or newer.

You will also need to install a patched version of yajl until this pull request is incorporated (there exists a memory leak in the yajl tree parser library that occurs whenever you try to parse invalid JSON strings that has nested objects or arrays. See the linked pull request for the 3-line fix. This can also, of course, be worked around by not parsing any invalid JSON strings).

Command line Usage via nmsgtool

We have added two command line options to nmsgtool: -j for input and -J for output. Use it just as you would the -r and -w options for binary format.

[-j file]         or --readjson   read json format data from file
[-J file]         or --writejson  write json format data to file
Read a message from channel 204 and emit to stdout as JSON
$ nmsgtool -C ch204 -c 1 -J -
Read a JSON-formatted file and emit to stdout as presentation format
$ nmsgtool -j input.json

API Usage in C

You can encode and decode messages using the nmsg_message_from_json() and nmsg_message_to_json() functions. Both functions set the pointer in their second argument on success (return code of nmsg_res_success) to a block of memory that the caller is responsible for deallocating after use.

nmsg_res nmsg_message_from_json(const char *json, nmsg_message_t *msg);
nmsg_res nmsg_message_to_json(nmsg_message_t msg, char **json);

You may also instantiate input and output objects that may be used via API or by attaching to a nmsg_io object. Simply pass an open file descriptor and then call using the normal nmsg_input and nmsg_output APIs.

nmsg_input_t nmsg_input_open_json(int fd);
nmsg_output_t nmsg_output_open_json(int fd);

msgmod API Changes

We have added two new functions to the msgmod field structure: format and parse. You will need to implement them in your custom nmsg msgmods if you want presentation formatting in the message object. Without them, libnmsg will fall back on the standard formatting for the field’s data type.

typedef nmsg_res (*nmsg_msgmod_field_format_fp)(nmsg_message_t m, struct nmsg_msgmod_field *field, void *ptr, struct nmsg_strbuf *sb, const char *endline);

typedef nmsg_res (*nmsg_msgmod_field_parse_fp)(nmsg_message_t m, struct nmsg_msgmod_field *field, const char *value, void **ptr, size_t *len, const char *endline);

struct nmsg_msgmod_field { ..., nmsg_msgmod_field_format_fp format; nmsg_msgmod_field_parse_fp parse; ... }

Python

Usage of these new features within the Python API is similar to that of the C API. You will need to use pynmsg 0.4.0 or later.

nmsg.message.from_json(str)
m.to_json()

The nmsg.message.from_json function takes a JSON object as a parameter and return a nmsg.message object. Correspondingly, each message object has a to_json function that returns the JSON reperesentation of that object.

You can instantiate input and output objects using the nmsg.input.open_json and nmsg.output.open_json functions. They are called with file names, integer file descriptors or file objects. The returned object can be used as-is or attached to a nmsg.io object.

nmsg.input.open_json(...)
nmsg.output.open_json(...)

Conclusion

We hope that this new addition to the NMSG suite will make it easier for you to integrate our data more tightly with your back-end analysis platform of choice as well as make it easier to share your data over SIE.

If you have a sales question, please contact the Farsight Security Sales department at sales@farsightsecurity.com or call +1-650-489-7919. If you have a technical question, please reach out to our technical team.

Henry Stern is a Senior Distributed System Engineer for Farsight Security, Inc.

← Blog Home

Protect against cybercriminal activity in real-time.

Request demo

Email: sales@farsightsecurity.com Phone: +1-650-489-7919