The Farsight SIE brings together hundreds of megabits per second of real time telemetry from cooperating sensors all over the world. There are two access methods available to subscribers:
- Directly connect to the SIE network in one of our U.S. data centers.
- Remotely connect to the SIE using one of our remote access software packages (discussed below).
Customers of SIE can obtain access through one or more technologies, each technology facilitates different capabilities, knowing what data feeds you want to process will help determine what technology is the best tool to use. When first starting out with SIE, SIE Remote Access (SRA) or our RESTful API are likely the easiest and best choices for remotely connecting.
Purpose of Farsight SRA
With SRA, the utility and reach of Farsight SIE is now available anywhere on the Internet, not just inside Farsight's data centers. SRA makes SIE data available to subscribers via "wide area transit" TCP/IP which allows subscriber analysis and processing equipment to be located for each subscriber's convenience and economy.
Features of Farsight SRA
Subscribers to SRA will use Farsight's Advanced Exchange Access (AXA) protocol, an bespoke, open transport supporting command, control, and delivery. AXA allows a subscriber to select a set of SIE channels to be monitored and to specify assets to "watch" such as IP address blocks and DNS names, after which SRA will search the selected channels for the watched assets, returning only relevant data. This real time streaming search capability is the key SRA feature for delivering SIE's very high value without incurring the wide area transport costs of SIE's extremely high data volume.
Farsight AXA is an unencumbered transport protocol for which an open source middleware implementation has been published for subscriber-side use. Farsight has also published buildable source code examples showing how to access SIE via the SRA service and the AXA protocol, and a simple "tunnel" application which reproduces SIE channels on local sockets, loopback interfaces or files, allowing direct reuse by an SRA subscriber of any Network Message (NMSG) or Packet Capture (PCAP) based analysis software that previously required direct SIE access.
The SRA tools are freely available
SRA Service description
SRA is carried inside TLS tunnels, and subscriber authentication and access control is provided by TLS private key. An SRA subscriber first generates a private key and public certificate and provides Farsight with the public certificate. Farsight then provisions SRA subscriptions according to the subscriber's public key and a list of subscribed SIE channels. To access SIE data remotely, the subscriber will either use existing Farsight-maintained open source tools or write a custom AXA application using the C or Python APIs.
The SRA service allows a subscriber to express a set of SIE channels of interest, to set per-second rate limits, and to add IP and/or DNS watches. Based on the subscriber's commands, SRA will deliver SIE data to the subscriber in real-time over the AXA protocol.
To learn more about SRA see the SRA User Guide
AXA RESTful Interface
Purpose of Farsight AXA REST
Farsight also makes available a RESTful middleware layer in front of its AXA servers. This service, called the AXA Middleware Daemon (AXAMD) adds a standard RESTful streaming interface to AXA in order to enable developers of web-based applications to interface with Farsight's SRA and Realtime Anomaly Detector (RAD) servers. As a convenience, Farsight publishes a command line tool / Python extension library called axamd_client.
Access is controlled via an API key that is passed as the
HTTP header and upon purchase, Farsight will provision a subscriber's account and provide the api key and instructions on how to connect.
The AXAMD client is freely available
AXA REST Resources
To learn more about RESTful interface made available by AXAMD, please see the User Guide
SIE Server Rental
A server pre-configured with all of the software can be rented from Farsight. The most recent version of the hardware includes a Quad-core Intel processor, 16 GB RAM, one or two SIE cross-connects, and a 100Mbps Internet uplink. The Debian 7 operating system is pre-installed along with all SIE software components needed for accessing data, running scripts, or even performing development work.
For provisioning, the Farsight needs two items from the customer:
- the public part of an ssh key pair used to login to the server
- the IPv4 or IPv6 addresses from which remote ssh access will be allowed by our firewall
The user will be given root access to the server with the ability to modify the operating environment to suit their needs. Customers typically use this option to do their own pre-processing of data before bringing it back into their own analysis infrastructure over the Internet.
Configuring the SIE Network Interface
utility is required to connect SIE network interfaces to the SIE switch port fabric. This python utility sets up required VLAN interfaces and updates configuration files needed by libnmsg
. The MAC address of a participant server's SIE network interface must be provisioned in Farsight's system in order for sie-update
The latest version of the sie-update
script available as a Debian/Ubuntu package after installation of Farsight's package repository
. One can run in Debian/Ubuntu:
apt-get install python-daemon sie-update
For other operating systems, one can download the script and install it:
$ wget -O /usr/local/bin/sie-update https://raw.github.com/farsightsec/sie-update/master/sie-update
$ chmod +x /usr/local/bin/sie-update
# For optional "daemon" support:
easy_install python-daemon # requires python setuptools
# or install from https://pypi.python.org/pypi/python-daemon
to run properly, the name of the SIE network interface must be passed on the command line. It should be run using the --daemon
flag to periodically run in the background. For example, to use sie-update
with the eth1
interface as the SIE network interface, run:
$ sie-update -i eth1 -d
One can specify multiple interfaces on the command line (like -i eth1 -i eth3). This command must be run at system startup, for instance by adding the following line to the /etc/rc.local
$ sie-update -i eth1 -d
One might need to specify the absolute path of the script if it's installed in /usr/local/bin
instead of /usr/sbin
Note that /etc/rc.local
must be executable in order to run at startup.
program by default places the nmsg alias files into the /etc
directory, but this can be overriden by specifying the -e / --etcdir
parameter to sie-update
. Note that, when compiling nmsg
from source, --sysconfdir=/etc
should be passed to ./configure
so that libnmsg searches the correct directory for alias files, otherwise the configuration files default to being installed in /usr/local/etc
$ /usr/local/bin/sie-update -v -i eth1 -e eth3 -e /usr/local/etc
SIE Port Access
FSI customers can order a cross-connect to its SIE switches hosted at Equinix (Ashburn DC3 and Palo Alto SV8). An FSI account manager can help guide cross-connect provisioning details or hosting or colocation options.
For processing raw Passive DNS data, FSI recommends provisioning either Intel 540-T2 adapters for servers colocated near its switches. Long-range single-mode optics are recommended for participants accessing SIE from outside FSI's cage.
For provisioning, an FSI sales engineer will need the MAC address from the customer's interfaces that are connecting to the SIE switch along with an uplink IP address for the customer's server. The MAC address is utilized by an auto-configuration script to make sure VLANs are installed correctly on the server. The IP address (IPv6 and/or IPv4) is needed to allow access to the configuration data from the server.