Security Information Exchange (SIE) User Guide

Introduction

Farsight Security, Inc's Security Information Exchange (SIE) is the world's largest real-time threat intelligence platform — it aggregates, filters and broadcasts diverse Internet-security related information so security professionals can more accurately and quickly identify, map, and protect from cybercriminal activity. Collected from Farsight's network of over 500 global sensors, SIE streams more than 200,000 new observations per second. SIE can be used to map and measure the Internet, to detect and track attacks, to build derivative works such as reputation and threat intelligence feeds, or to distribute such feeds to other members of the Internet Security community.

Several data feeds are available on SIE. Farsight maintains a Channel Guide to show what's available. To hold and carry SIE data, Farsight created an extensible container wire and file format for storing and transmitting blobs of data called Network Message (NMSG). As its core, NMSG leverages Google Protocol Buffers for binary encoding with pre-defined schemas, or in a native packetized format like PCAP. Other data structures like JSON or XML can be encapsulated in NMSG for consistent transport across Farsight's exchange infrastructure and extracted at end points. Most of this document addresses what is needed to access and process these messages.

Audience

This document is intended for system administrators and programmers who want to interact with SIE.

System Requirements

Farsight supports 64-bit Intel platforms running open-source operating systems such as Linux or FreeBSD. Data from some of the lower bandwidth data channels can be processed with an Atom server or cloud instance with a 1GHz core, a hard disk for local storage, and a modest amount of RAM (1GB). Servers that Farsight provides for trial or rental typically have an Intel quad-core Xeon single-socket processor, 16 GB DDR3 ram, and a couple 2TB RAID1 7200 RPM hard disks. This is enough to be able to read/filter/relay any channel on SIE, but may not have enough RAM to do interesting processing on the channels with the highest data volumes. For data processing for its highest-bandwidth raw Passive DNS feed, Farsight utilizes a 20-core two-socket server with 256GB RAM and tries not to utilize disks during processing. Using the SIE model where the switch is a broadcast bus, Farsight utilizes a waterfall computing model to take streams of input of data, perform analysis or filtering, and re-broadcast that data as a smaller and refined stream back out into the stream environment that's easier for the next stage of processing.

The software environment Farsight develops upon and supports is Debian 7 on 64-bit Intel-based servers. Exchange participants and customers are able to utilize Debian 7, Ubuntu LTS, RHEL 6, CentOS 6, and FreeBSD to read or create data streams. Farsight maintains installation instructions for each (see below).

While it's possible for the software to run under Solaris or MacOS, Farsight doesn't currently support or test those environments.

Delivery options

The Farsight SIE brings together hundreds of megabits per second of real time telemetry from cooperating sensors all over the world. There are two access methods available to subscribers:
  1. Directly connect to the SIE network in one of our U.S. data centers.
  2. Remotely connect to the SIE using one of our remote access software packages (discussed below).
Customers of SIE can obtain access through one or more technologies, each technology facilitates different capabilities, knowing what data feeds you want to process will help determine what technology is the best tool to use. When first starting out with SIE, SIE Remote Access (SRA) or our RESTful API are likely the easiest and best choices for remotely connecting.

SRA

Purpose of Farsight SRA
With SRA, the utility and reach of Farsight SIE is now available anywhere on the Internet, not just inside Farsight's data centers. SRA makes SIE data available to subscribers via "wide area transit" TCP/IP which allows subscriber analysis and processing equipment to be located for each subscriber's convenience and economy.

Features of Farsight SRA
Subscribers to SRA will use Farsight's Advanced Exchange Access (AXA) protocol, an bespoke, open transport supporting command, control, and delivery. AXA allows a subscriber to select a set of SIE channels to be monitored and to specify assets to "watch" such as IP address blocks and DNS names, after which SRA will search the selected channels for the watched assets, returning only relevant data. This real time streaming search capability is the key SRA feature for delivering SIE's very high value without incurring the wide area transport costs of SIE's extremely high data volume.

Farsight AXA is an unencumbered transport protocol for which an open source middleware implementation has been published for subscriber-side use. Farsight has also published buildable source code examples showing how to access SIE via the SRA service and the AXA protocol, and a simple "tunnel" application which reproduces SIE channels on local sockets, loopback interfaces or files, allowing direct reuse by an SRA subscriber of any Network Message (NMSG) or Packet Capture (PCAP) based analysis software that previously required direct SIE access.

The SRA tools are freely available.

SRA Service description
SRA is carried inside TLS tunnels, and subscriber authentication and access control is provided by TLS private key. An SRA subscriber first generates a private key and public certificate and provides Farsight with the public certificate. Farsight then provisions SRA subscriptions according to the subscriber's public key and a list of subscribed SIE channels. To access SIE data remotely, the subscriber will either use existing Farsight-maintained open source tools or write a custom AXA application using the C or Python APIs.

The SRA service allows a subscriber to express a set of SIE channels of interest, to set per-second rate limits, and to add IP and/or DNS watches. Based on the subscriber's commands, SRA will deliver SIE data to the subscriber in real-time over the AXA protocol.

SRA Resources
To learn more about SRA see the SRA User Guide

AXA RESTful Interface

Purpose of Farsight AXA REST
Farsight also makes available a RESTful middleware layer in front of its AXA servers. This service, called the AXA Middleware Daemon (AXAMD) adds a standard RESTful streaming interface to AXA in order to enable developers of web-based applications to interface with Farsight's SRA and Realtime Anomaly Detector (RAD) servers. As a convenience, Farsight publishes a command line tool / Python extension library called axamd_client.

Access is controlled via an API key that is passed as the X-API-Key HTTP header and upon purchase, Farsight will provision a subscriber's account and provide the api key and instructions on how to connect.

The AXAMD client is freely available.

AXA REST Resources
To learn more about RESTful interface made available by AXAMD, please see the User Guide.

SIE Server Rental

A server pre-configured with all of the software can be rented from Farsight. The most recent version of the hardware includes a Quad-core Intel processor, 16 GB RAM, one or two SIE cross-connects, and a 100Mbps Internet uplink. The Debian 7 operating system is pre-installed along with all SIE software components needed for accessing data, running scripts, or even performing development work.

For provisioning, the Farsight needs two items from the customer:
  1. the public part of an ssh key pair used to login to the server
  2. the IPv4 or IPv6 addresses from which remote ssh access will be allowed by our firewall
The user will be given root access to the server with the ability to modify the operating environment to suit their needs. Customers typically use this option to do their own pre-processing of data before bringing it back into their own analysis infrastructure over the Internet.

Configuring the SIE Network Interface
The sie-update utility is required to connect SIE network interfaces to the SIE switch port fabric. This python utility sets up required VLAN interfaces and updates configuration files needed by libnmsg and nmsgtool. The MAC address of a participant server's SIE network interface must be provisioned in Farsight's system in order for sie-update to work.

The latest version of the sie-update script available as a Debian/Ubuntu package after installation of Farsight's package repository. One can run in Debian/Ubuntu: apt-get install python-daemon sie-update

For other operating systems, one can download the script and install it:
$ wget -O /usr/local/bin/sie-update https://raw.github.com/farsightsec/sie-update/master/sie-update
$ chmod +x /usr/local/bin/sie-update

# For optional "daemon" support:
easy_install python-daemon # requires python setuptools
# or install from https://pypi.python.org/pypi/python-daemon
For sie-update to run properly, the name of the SIE network interface must be passed on the command line. It should be run using the --daemon flag to periodically run in the background. For example, to use sie-update with the eth1 interface as the SIE network interface, run:
$ sie-update -i eth1 -d
One can specify multiple interfaces on the command line (like -i eth1 -i eth3). This command must be run at system startup, for instance by adding the following line to the /etc/rc.local script:
$ sie-update -i eth1 -d
One might need to specify the absolute path of the script if it's installed in /usr/local/bin instead of /usr/sbin.

Note that /etc/rc.local must be executable in order to run at startup.

The sie-update program by default places the nmsg alias files into the /etc directory, but this can be overriden by specifying the -e / --etcdir parameter to sie-update. Note that, when compiling nmsg from source, --sysconfdir=/etc should be passed to ./configure so that libnmsg searches the correct directory for alias files, otherwise the configuration files default to being installed in /usr/local/etc.
$ /usr/local/bin/sie-update -v -i eth1 -e eth3 -e /usr/local/etc

SIE Port Access

FSI customers can order a cross-connect to its SIE switches hosted at Equinix (Ashburn DC3 and Palo Alto SV8). An FSI account manager can help guide cross-connect provisioning details or hosting or colocation options.

For processing raw Passive DNS data, FSI recommends provisioning either Intel 540-T2 adapters for servers colocated near its switches. Long-range single-mode optics are recommended for participants accessing SIE from outside FSI's cage.

For provisioning, an FSI sales engineer will need the MAC address from the customer's interfaces that are connecting to the SIE switch along with an uplink IP address for the customer's server. The MAC address is utilized by an auto-configuration script to make sure VLANs are installed correctly on the server. The IP address (IPv6 and/or IPv4) is needed to allow access to the configuration data from the server.

SIE Software

NMSG

NMSG is an extensible container wire and file format for storing and transmitting blobs of data with support for dynamic message types, compression, fragmentation, sequencing, and rate limiting. More information is available here.

Farsight performs development primarily on Debian 7 with installation instructions available for Linux and FreeBSD operating systems listed below. Installation from source code for software like nmsgtool has previously worked on MacOS or Solaris, but may need some modifications to keep up with updates.

Debian / Ubuntu See the SIE on Debian page.

CentOS / RHEL See the SIE on EL page.

FreeBSD See the SIE on FreeBSD page.

From source SIE participants or SRA customers will want to pre-install Google Protocol Buffers, LibPCAP, and Zlib on their server before compiling SIE-related software.

Source code tarballs are available for the packages below are all available from https://dl.farsightsecurity.com/dist/.

In particular, the wdns, nmsg, and sie-nmsg distributions will be needed. For SRA, the axa distribution will also be needed. To install, download the latest tarball, unpack it, and run in its directory:
$ ./configure
$ make install

Python

Farsight maintaines a Python module named pynmsg, a Python 2.7 extension module implemented in Cython for the nmsg C library.

Perl

Net::Nmsg Perl modules

For Debian/Ubuntu, Farsight maintains a package called libnet-nmsg-perl.
$ apt-get install libnet-nmsg-perl
For FreeBSD, Net::Nmsg is available as an official binary package.
$ pkg install p5-Net-Nmsg
For other operating systems it's possible to install Net:Nmsg using CPAN. Before installing though, it requires libpcap development header files or libpcap installed from source.
$ perl -MCPAN -e shell
cpan> install Bundle::CPAN
cpan> install Net::Nmsg
Some of the dependent packages might ask installation questions like hitting [enter] when asked for a mathematic expression, or entering some minimal host information when configuring IO::Socket.

Protect against cybercriminal activity in real-time.

Request demo

Email: sales@farsightsecurity.com Phone: +1-650-489-7919