Farsight's Advanced Exchange Access Internals: Understanding Accounting
By Mike Schiffman
This article explains Farsight Security's Advanced Exchange Access (AXA) accounting subsystem. This is the mechanism by which AXA tracks, logs, and communicates server-side packet information.
To get the most from this article, it is recommended that you be comfortable with the material in the following Farsight Security Blog articles:
- Farsight's Advanced Exchange Access, Part 1
- Farsight's Advanced Exchange Access, Part 2
- Farsight's Advanced Exchange Access, Part 3
What is Accounting?
Accounting is AXA's way of keeping track of traffic totals. Server-side, AXA
maintains a series of per-client packet counters (a full list is below).
The AXA protocol message
AXA_P_OP_ACCT sent from client to server is used
query this data. The command is available from
acct. It is also available from
radtunnel via the
command line option. Accounting messages are also logged server-side.
The AXA accounting counters are described below.
total-collected: On the RAD side, packets that have been successfully run through an anomaly detector.
total-congested: Packets that were unable to be forwarded to the client due to connection issues, although it could also be client load (i.e., the the connection isn't removing data from the server fast enough, or the client isn't reading and processing the data fast enough). Note that: high server load might increase
total-missed, but it would generally decrease
total-congestedby throttling the input to the congested pipe / process.
total-filtered: This is an SRA input counter. It measures packets/nmsgs that have been read on the input side (usually directly from the SIE network) and will be submitted to the filtering logic (i.e.: an AXA watch).
total-missed: This is an SRA loss counter. It measures packets that the SRA server failed to receive on the input side either because the daemon was too busy or because they were lost during transit. This counter tracks NMSG- and pcap-based loss.
total-ratelimited: This measures packets that were dropped instead of being forwarded to the SRA or RAD client to comply with the rate limits specified for the client.
total-sent: This measures packets that have been sent to an SRA or RAD client.
Accounting with sratool
Let's have a look at a common
sratool is used to
connect to an SRA server and a "fire hose" watch is set for our popular
$ sratool sra> connect tls:mschiffm@sra-server,1021 * HELLO srad version 1.2.1 sra-server AXA protocol 1 sra> ch 214 on ; 1 wa ch=214 [watch hits omitted]
This is left to run for approximately three minutes of wall clock time. Next, the output is paused and the accounting command is run.
sra> pause * OK PAUSE output paused sra> acct * OK ACCOUNTING total-filtered=66360 total-missed=0 total-collected=0 total-sent=64912 total-ratelimited=0 total-congested=0
We see that 66,360 nmsgs were read from the SIE channel. For the watch we set this should actually be a good approximation of the overall packet rate. And indeed channel 214 has an average payload rate of approximately 340 payloads per second. Over this very small three minute (180 second) window we observed ~369 packets.
There were no missed packets. In this case, this is the everything's OK alarm. We expect there to be no packet loss on the SIE input side. If there were for such a low bandwidth channel, this would be indicative of server-side network fault or server overload.
We see that 64,912 packets were sent from the SRA server to our
sratoolclient. This is good and what we expect.
Because there was no congestion or rate limiting, you may wonder why the number of sent packets is lower than the number of filtered packets. This is because when we stopped sending packets to the client, we didn't stop the server from filtering them. SRA was still reading from the input channel but wasn't forwarding any packets.
Accounting with sratunnel
Let's have a look at another example, this time using
sratunnel. Using the
sratunnel is invoked to run for three minutes. It
connects to the same SRA server and sets a fire hose watch for the
channel. Finally, the
-A 180 -d option string instructs
sratunnel to emit
accounting statistics every 180 seconds and the results are written to a file.
$ timeout 180 sratunnel -s tls:mschiffm@sra-server,1021 -w "ch=220" -c 220 -A 180 -d -o nmsg:file:test-220.nmsg connecting to tls:mschiffm@sra-server,1021 ACCOUNTING total-filtered=4183437 total-missed=44 total-collected=0 total-sent=84371 total-ratelimited=0 total-congested=4096266
This time, 4,183,437 packets were read from the SIE channel. Using the same logic as above, we find that this equates to approximately 23,241 payloads per second which is commensurate to Channel 220's searing average per second rate of ~24,000 payloads (it is sourced from our Passive DNS feed which has a per second payload rate of ~120,000).
44 packets were missed on the input side. This is a .001% rate of loss and well within acceptable limits.
Only 84,371 packets were sent to the client. To understand why, we look at the congestion number.
4,096,266 packets were lost due to congestion. In this case, this high number is likely due to the fact that the client is located on the other side of the country from the SIE data center where the SRA server is located. Additionally, the client's downstream Internet connection is just plain too slow to keep up with the high rate of channel 220.
Now that you know how to use AXA Accounting, you can use this information along with other data to learn more about your Farsight SRA data flows and in some cases, find and troubleshoot issues. If you're interested in learning more about Farsight Products and Services, please reach out.
Mike Schiffman is a Packet Esotericist for Farsight Security, Inc.