About Security Information Exchange (SIE)
The Security Information Exchange (SIE), from Farsight Security® Inc., is a scalable and adaptable real-time data streaming and information sharing platform. SIE collects and provides access to more than 200,000 observations per-second of raw data from its global sensor network. Farsight also applies unique and proprietary methods for improving usability of the data, directly sharing the refined intelligence with SIE customers and DNSDB®, one of the world's largest passive DNS (pDNS) databases.
The diverse set of data available from SIE includes the following and is relevant and useful for practitioners in various technology roles:
- Raw and processed passive DNS data
- Darknet/darkspace telescope data
- SPAM sources and URLs
- Phishing URLs and associated targeted brands
- Connection attempts from malware-infected systems (as seen by a sinkhole)
- Network traffic blocked by Intrusion Detection Systems (IDS) and firewall devices
Each unique set of data in SIE is known as a channel and the data acquired from a specific channel can be customized to meet the needs of each customer, enabling you to subscribe to and access only the channels needed to solve your problem. A channel in SIE may be the result from analyzing the data or a subset of data from other channels.
Why Passive DNS (pDNS)?
DNS is a critical component of Internet communication and almost all Internet transactions begin with a DNS query and response.
- Visiting a website?: Your system uses DNS to resolve the IP address of the hostname for the website you are attempting to access.
- Sending an email?: Email uses DNS to resolve the IP address of the mail exchange server your message should be delivered to.
DNS serves as early warning and detection solution for phishing, spam, malicious and suspicious behaviors, and other attacks. DNS intelligence is considered the only source of "ground truth" information for the Internet.
Farsight Security's mission is to make the Internet a safer place. We provide security solutions that empower customers with meaningful and relevant intelligence. This information provides customers with insights about the network configuration of a threat and the surrounding network on the Internet for improving the value and impact of threat intelligence and research.
The Security Information Exchange (SIE), from Farsight Security Inc., is designed with privacy in mind. The passive DNS (pDNS) sensors do not collect Personally Identifiable Information (PII) from client resolvers (also known as stub) by deliberately collecting between recursive resolvers and authoritative servers.
The data from SIE enables security professionals to accurately identify, map, and protect their networks from cybercriminal activity by providing global visibility. It provides immediate access to a real-time global sensor network without the need to develop or deploy your own data collection infrastructure.
Methods to access and acquire data from SIE channels are available using SIE Direct Connect, SIE Remote Access (SRA), SIE Batch, or AXAMD. These methods are described in the SIE Technical Reference document. Due to the technical limitations of transporting high bitrate SIE channels across the Internet, some access methods are not available for specific SIE channels. These restrictions are noted below.
Based on your needs, you can subscribe to an individual channel or a bundle of commonly used channels. A Farsight sales representative or Solution Architect (SAs) can help you select the channels that will best meet your needs.
SIE Channel Guide
In the following table, the average and max bitrates (in bits per second) for each channel to indicate how much network bandwidth is required to acquire and receive data from a channel. The average and max payloads (per second) indicate the number of records that need to be processed data from a channel is acquired in real-time.
|Channel||Name||Description||Bitrate (Max)||Payloads (Max)|
|14||Darknet||Captured packets destined for unused network space. Can be used to monitor scanning activity and back-scatter from large spoofing attacks.||<1Kb/sec (<1Kb/sec)||3K/sec (5K/sec)|
|24||Spam-Full||Full copies of emails sent to spamtrap email addresses.||16Kb/sec (55Kb/sec)||2/sec (7/sec)|
|25||Spam-Select||Select fields from the emails sent to Channel 24 (Spam-Full).||16Kb/sec (55Kb/sec)||2/sec (7/sec)|
|27||Phishing URLs||PhishLabs data for malicious sites involved in phishing campaigns.||<1Kb/sec (10Kb/sec)||<1/sec (2/sec)|
|42||IDS and Firewall Log Data||ThreatStop data of blocking action from IDS and Firewall devices.||4Mb/sec (15Mb/sec)||500/sec (2k/sec)|
|80||Conficker Sinkhole||Connection attempts from infected clients to sinkholes that monitor Conficker activity.||650Kb/sec (1.2Mb/sec)||375/sec (725/sec)|
|115||DDos Events||Evidence of DDoS and DRDoS (Distributed Reflection Denial of Service) attacks based on analysis of data from Channel 14 (Darknet).||<1Kb/sec (<1K/sec)||<1/sec (1.5/sec)|
|204||Processed DNS Data||Passive DNS observations after deduplication, verification, and filtering.||37Mb/sec (70Mb/sec))||27K/sec (50K/sec)|
|206||DNSDB Rejected Records (Chaff)||Passive DNS observations that were malformed, unsuccessful queries, or otherwise fail the verification process.||28Mb/sec (40Mb/sec)||20K/sec (25K/sec)|
|207||DNSDB De-duplicated Data||Passive DNS observations after the deduplication processing phase and immediately prior to the verification phase.||120Mb/sec (150Mb/sec)||90K/sec (130K/sec)|
|208||DNSDB Verified Data||Passive DNS observations after the verification processing phase and prior to filtering.||60Mb/sec (90Mb/sec)||45K/sec (65K/sec)|
|211||Newly Active Domains (NAD)||Domains that have been observed after having not been seen for at least 10 days.||56Kb/sec (170K/sec)||50/sec (150/sec)|
|212||Newly Observed Domains (NOD)||Passive DNS observations of base domains not previously seen when compared to the DNSDB historical database.||3Kb/sec (25Kb/sec)||2/sec (20/sec)|
|213||Newly Observed Hostnames (NOH)||Fully Qualified Domain Names (FQDNs) not previously seen when compared to the DNSDB historical database.||1.5Mb/sec (3Mb/sec)||1K/sec (3K/sec)|
|214||DNS Changes||Passive DNS observations where some aspect of the query or response was not found when compared to the DNSDB historical database.||3.5Mb/sec (6Mb/sec)||2.5k/sec (5k/sec)|
|220||DNS Errors||Queries where the authoritative DNS servers answered with a non-zero error code.||220Mb/sec (240Mb/sec)||60K/sec (65K/sec)|
|221||NX Domains||Passive DNS observations where the responding server returned the "NXDomain" error.||35Mb/sec (45Mb/sec)||45K/sec (60K/sec)|
|255||Heartbeat||Repeating data used for SIE health monitoring.||1Kb/sec (1Kb/sec)||1Kb/sec (1Kb/sec)|
Note: Quoted bitrates and payloads are representative of SIE traffic as of June 2021.
Data Formats and SIE Access Methods
Intelligence data acquired from SIE is delivered in various data formats depending on the channel and access method. Data formats follow.
|NMSG||Farsight's Network Message (NMSG) Encapsulation format. See the SIE NMSG User Guide for details.|
|NDJSON||Newline delimited JSON, see https://jsonlines.org/ for details.|
|PCAP||Packet Capture format, see https://www.tcpdump.org/pcap.html for details.|
The data format for each channel and access method follow.
- SIE Direct Connect: NMSG unless noted below.
- SIE Remote Access (SRA): NMSG unless noted below.
- SIE Batch: NDJSON unless noted below.
- AXAMD: JSON unless noted below.
|Channel||Name||Direct Connect||SRA||SIE Batch||AXAMD|
|14||Darknet||Yes (PCAP)||Yes (PCAP)||No||No|
|42||IDS and Firewall Log Data||Yes||Yes||Yes||No|
|204||Processed DNS Data||Yes||Yes||Yes (NMSG)||No|
|206||DNSDB Rejected Records (Chaff)||Yes||Yes||Yes (NMSG)||No|
|207||DNSDB De-duplicated Data||Yes||No||Yes (NMSG)||No|
|208||DNSDB Verified Data||Yes||No||Yes (NMSG)||No|
|211||Newly Active Domains (NAD)||Yes||Yes||Yes||Yes|
|212||Newly Observed Domains (NOD)||Yes||Yes||Yes||Yes|
|213||Newly Observed Hostnames (NOH)||Yes||Yes||Yes||Yes|
|221||NX Domains||Yes||Yes||Yes (NMSG)||No|
SIE Access Methods
Data from SIE can be accessed and acquired using the following methods:
- Direct Connect: Connect a system to the SIE network. This 1.) requires a server to be installed in a data center where Farsight has a point of presence, and 2.) then ordering a network cross connect between your server and the SIE network. Customers can optionally, and prefer to, lease a blade server from Farsight.
- SIE Remote Access (SRA): Remotely connect to the SIE network using an encrypted tunnel from your workstation or a server in your local data center.
- SIE Batch: Provides on-demand access for downloading data from SIE channels using a RESTful API or web-based interface. You select the channel and duration of time you are interested in, and then download the data for analysis. The duration of available data is dependent on the channel, but is typically the most recent 12-18 hours.
For additional information about SIE access methods, please see the SIE Technical Overview document.
SIE Direct Connect allows a customer to physically connect a server to the Farsight SIE network for maximum data throughput. This can be done in one of two ways:
- Blade Server: Pre-configured blade servers co-located in one of Farsight's data centers that can be leased by customers for direct access to SIE channels.
- Customer Server: Customer (owned, managed, and operated) servers that can be installed in one of Farsight's data centers and physically connected to the SIE network with a network cross-connect.
If a blade server is leased from Farsight, it will be pre-installed with the essential software components needed to acquire, process, compress, buffer, and transfer data from SIE channels to the customer's data center for additional analysis, enrichment, and storage.
If a customer uses their own server, an order can be submitted for a cross-connect to the SIE switches hosted at select Equinix data centers (Ashburn DC3 and Palo Alto SV8). An FSI account manager can help guide cross-connect provisioning details, hosting, or colocation options.
For additional information about SIE connection methods, please see the SIE Technical Overview document. A Farsight's sales representatives is happy to share a copy of this document with you. This will help inform and guide you in understanding which connection method will work best for you.
SIE Remote Access (SRA)
SIE Remote Access (SRA) enables a customer to remotely connect to the Security Information Exchange (SIE) from anywhere on the Internet. SRA provides access to SIE channel data on customer's local servers, allowing their analysis and processing systems to be located in their own data centers rather than physically co-located at a Farsight's data center.
Due to the technical limitations of transporting high bitrate SIE channels across the Internet, the SRA access method is not available for all SIE channels. Please reference the SIE Channel Guide for channels that can be accessed using SRA.
SRA uses the Advanced Exchange Access (AXA) transport protocol which enables SRA sessions to perform the following:
- Select which SIE channel or channels to monitor and acquire data from
- Define user-specified search or filtering criteria to match IP or DNS traffic
- Control rate-limits and other AXA parameters
The streaming search and filtering capabilities of AXA enables SRA to access and acquire meaningful and relevant data from SIE while avoiding the costs of transporting enormous volumes of data across the Internet.
Note: For high volume channels accessed using SRA, it is expected that customer's will specify a search or filter for IP addresses and DNS domain names or hostnames of interest. The SRA service will only collect and send data matching the specified criteria across the Internet to the customer.
SIE Batch provides on-demand access for downloading data from SIE channels using a RESTful API or web-based interface. You select the channel and duration of time you are interested in, and then download the data for analysis. The duration of available data is dependent on the channel, but is typically the most recent 12-18 hours. SIE Batch allows you to acquire data from SIE channel using two (2) methods:
- API: Allows you to write tools to programmatically download data from SIE channels for analysis.
- Interactively: Web-based interface to the API that enables you to select and download SIE channel data on-demand.
Advanced Exchange Access Middleware Daemon (AXAMD)
Farsight also provides a RESTful middleware layer in front of its AXA service.
This service is called the AXA Middleware Daemon (AXAMD) and provides a RESTful
capability that adds a streaming HTTP interface on top of the AXA toolkit. This
enables web-application developers to interface with SIE using SRA. Farsight
also published a command line tool and Python extension library called
axamd_client https://github.com/farsightsec/axamd_client. This toolkit is
licensed under the Apache 2.0 license.
The Advanced Exchange Access (AXA) toolkit https://github.com/farsightsec/axa contains tools and a C library to bring Farsight's real-time data and services directly from the Farsight Security Information Exchange (SIE) to the customers network.
Advanced Exchange Access Middleware Daemon (AXAMD) is a suite of tools and library code to bring Farsight's real-time data and services directly from the Farsight Security Information Exchange (SIE) to the customers network.
Due to the technical limitations of transporting high bitrate SIE channels across the Internet, the AXAMD access method is not available for all SIE channels.
About Farsight Security
Farsight Security, Inc. is the world's leading provider of historical and real-time DNS intelligence solutions. We enable security teams to qualify, enrich, and correlate all sources of threat data and ultimately save time when it is most critical - during an attack or investigation. Our solutions provide enterprise, government, and security industry personnel and platforms with unmatched global visibility, context, and response. Farsight Security is headquartered in San Mateo, California, USA. To learn more about how we can empower your security, threat, and intelligence platforms and security organization with Farsight Security passive DNS (pDNS) and threat intelligence solutions, please visit us at www.farsightsecurity.com or follow us on Twitter at @FarsightSecInc.