This article explains Farsight Security’s Advanced Exchange Access (AXA) accounting subsystem. This is the mechanism by which AXA tracks, logs, and communicates server-side packet information.
To get the most from this article, it is recommended that you be comfortable with the material in the following Farsight Security Blog articles:
- Farsight’s Advanced Exchange Access, Part 1
- Farsight’s Advanced Exchange Access, Part 2
- Farsight’s Advanced Exchange Access, Part 3
What is Accounting?
Accounting is AXA’s way of keeping track of traffic totals. Server-side, AXA
maintains a series of per-client packet counters (a full list is below).
The AXA protocol message
AXA_P_OP_ACCT sent from client to server is used
query this data. The command is available from
acct. It is also available from
radtunnel via the
command line option. Accounting messages are also logged server-side.
The AXA accounting counters are described below.
total-collected: On the RAD side, packets that an anomaly detector said should be sent to the RAD client are counted here.
total-congested: Packets that were unable to be forwarded to the client due to connection issues, although it could also be client load (i.e., the the connection isn’t removing data from the server fast enough, or the client isn’t reading and processing the data fast enough). Note that: high server load might increase
total-missed, but it would generally decrease
total-congestedby throttling the input to the congested pipe / process.
total-filtered: Internally, AXA stores watches in one of three tries (IPv4, IPv6, and domain). This counter measures packets that have filtered into the trie and are considered for forwarding to the SRA or RAD client. Ostensibly, this counter tracks packets that have matched a watch pattern.
total-missed: This is an SRA input counter. It measures packets that the SRA server failed to receive on the input side (usually directly from the SIE network) either because the daemon was too busy or because they were lost during transit. This counter only tracks NMSG loss, not packet loss on PCAP-based channels.
total-ratelimited: This measures packets that were dropped instead of being forwarded to the SRA or RAD client to comply with the rate limits specified for the client.
total-sent: This measures packets that have been sent to an SRA or RAD client.
Accounting with sratool
Let’s have a look at a common
sratool is used to
connect to an SRA server and a “fire hose” watch is set for our popular
$ sratool sra> connect tls:mschiffm@sra-server,1021 * HELLO srad version 1.2.1 sra-server AXA protocol 1 sra> ch 214 on ; 1 wa ch=214 [watch hits omitted]
This is left to run for approximately three minutes of wall clock time. Next, the output is paused and the accounting command is run.
sra> pause * OK PAUSE output paused sra> acct * OK ACCOUNTING total-filtered=66360 total-missed=0 total-collected=0 total-sent=64912 total-ratelimited=0 total-congested=0
We see that 66,360 packets were matched by our watch. For the watch we set this should actually be a good approximation of the overall packet rate. And indeed channel 214 has an average payload rate of approximately 340 payloads per second. Over this very small three minute (180 second) window we observed ~369 packets.
There were no missed packets. In this case, this is the everything’s OK alarm. We expect there to be no packet loss on the SIE input side. If there were for such a low bandwidth channel, this would be indicative of server-side network fault or server overload.
We see that 64,912 packets were sent from the SRA server to our
sratoolclient. This is good and what we expect.
Because there was no congestion or rate limiting, you may wonder why the number of sent packets is lower than the number of filtered packets. This is because when we stopped sending packets to the client, we didn’t stop the server from filtering them. SRA was still matching watches but wasn’t forwarding any packets.
Accounting with sratunnel
Let’s have a look at another example, this time using
sratunnel. Using the
sratunnel is invoked to run for three minutes. It
connects to the same SRA server and sets a fire hose watch for the
channel. Finally, the
-A 180 -d option string instructs
sratunnel to emit
accounting statistics every 180 seconds and the results are written to a file.
$ timeout 180 sratunnel -s tls:mschiffm@sra-server,1021 -w "ch=220" -c 220 -A 180 -d -o nmsg:file:test-220.nmsg connecting to tls:mschiffm@sra-server,1021 ACCOUNTING total-filtered=4183437 total-missed=44 total-collected=0 total-sent=84371 total-ratelimited=0 total-congested=4096266
This time, 4,183,437 packets were matched by our watch. Using the same logic as above, we find that this equates to approximately 23,241 payloads per second which is commensurate to Channel 220’s searing average per second rate of ~24,000 payloads (it is sourced from our Passive DNS feed which has a per second payload rate of ~120,000).
44 packets were missed on the input side. This is a .001% rate of loss and well within acceptable limits.
Only 84,371 packets were sent to the client. To understand why, we look at the congestion number.
4,096,266 packets were lost due to congestion. In this case, this high number is likely due to the fact that the client is located on the other side of the country from the SIE data center where the SRA server is located. Additionally, the client’s downstream Internet connection is just plain too slow to keep up with the high rate of channel 220.
Now that you know how to use AXA Accounting, you can use this information along with other data to learn more about your Farsight SRA data flows and in some cases, find and troubleshoot issues. If you’re interested in learning more about Farsight Products and Services, please reach out.
Mike Schiffman is a Packet Esotericist for Farsight Security, Inc.