Running Bulk Queries in DNSDB For A File of IP Addresses
By Joe St Sauver
I. Introduction
Most DNSDB API users are familiar with using a command line tool such as dnsdbq
to query DNSDB, while others may prefer to use the Farsight DNSDB Scout(tm) browser plugin for Chrome or Firefox.
If you use dnsdbq,
you know you can easily run simple IP address queries such as:
$ dnsdbq -i 104.244.14.108 ;; record times: 2018-09-27 03:57:22 .. 2019-07-22 14:44:31 ;; count: 4417 fsi.io. A 104.244.14.108 ;; record times: 2018-09-27 11:26:14 .. 2019-07-22 15:51:30 ;; count: 1538 www.fsi.io. A 104.244.14.108 [etc]
However, sometimes you may have a file with many IP addresses you want to check in DNSDB.
For the purpose of this blog, we'll assume that you're working from a Unix-like system (either a Mac, a Linux box, or a Windows system running the Windows Subsystem for Linux (see https://www.farsightsecurity.com/blog/txt-record/WSL-20180315/ )).
II. Using A Shell File To Run Multiple dnsdbq
queries
We'll also assume that you've got a file containing a number of IPv4 (and/or potentially IPv6) addresses such as:
104.42.128.171 128.193.4.112 128.223.32.35 128.223.60.22 131.252.115.150 134.10.50.30 140.211.209.32 140.211.127.171 198.199.109.37 2620:105:b050:1550::90 2001:468:d01:20::80df:2023
You'd like to run those through DNSDB using dnsdbq
. You could just edit your list of IP addresses to create a command file consisting of a series of dnsdbq
commands such as:
$ cat run-some-ips.sh dnsdbq -i 104.42.128.171 dnsdbq -i 128.193.4.112 dnsdbq -i 128.223.32.35 [etc]
You'd then execute that command file by saying:
$ chmod a+rx run-some-ips.sh $ sh run-some-ips.sh > run-some-ips.output $ more run-some-ips.output ;; record times: 2018-05-02 11:00:50 .. 2019-07-22 13:22:10 ;; count: 5507 qnamaker.ai. A 104.42.128.171 ;; record times: 2018-03-09 05:08:54 .. 2018-11-13 09:35:57 ;; count: 16 abma.asn.au. A 104.42.128.171 [etc]
That works, but is somewhat "clunky" and inelegant.
III. The dnsdbq -f option
A more refined option is to use the dnsdbq -f
option, which reads in and runs a file of DNSDB queries from <stdin>
. That file must be specially formatted (perhaps with your favorite editor) to look like the following for IP queries:
$ cat my-ips.dat rdata/ip/104.42.128.171 rdata/ip/128.193.4.112 rdata/ip/128.223.32.35 [etc]
You could then run those queries by simply saying:
$ dnsdbq -f < my-ips.dat > my-ips.output $ more my-ips.output ;; record times: 2018-05-02 11:00:50 .. 2019-07-22 13:22:10 ;; count: 5507 qnamaker.ai. A 104.42.128.171 ;; record times: 2018-03-09 05:08:54 .. 2018-11-13 09:35:57 ;; count: 16 abma.asn.au. A 104.42.128.171 [etc]
You're not limited to running only IP queries using the dnsdbq -f
option, although the format of the lines in your data file will be a little different.
If you were doing left hand side RRname queries instead of IP queries, your file of queries might look like:
rrset/name/www.uoregon.edu rrset/name/www.oregonstate.edu rrset/name/www.eou.edu [etc]
And if you were doing right hand side Rdata name queries, your file of queries might look like:
rdata/name/phloem.uoregon.edu rdata/name/ns1.oregonstate.edu rdata/name/mx.smtp.ucla.edu
You can also create an input file that's a mixture of IP queries, RRset name queries, and Rdata name queries, if that happens to be convenient for you – just use the appropriate prefix for each line in the file.
We should also mention that you can also use other common dnsdbq
options, such as asking for output in JSON lines format, or using time fencing, or specifying the maximum number of results to return for each query, should you desire to do so.
See the dnsdbq
manual page for details about all dnsdbq
options:
$ man dnsdbq dnsdbq(1) BSD General Commands Manual dnsdbq(1) NAME dnsdbq -- DNSDB query tool SYNOPSIS dnsdbq [-dfgmhjsShcI] [-A timestamp] [-B timestamp] [-b bailiwick] [-i ip] [-l query_limit] [-L output_limit] [-u server_sys] [-n name] [-k sort_keys] [-P page] [-p output_type] [-R raw_rdata] [-r rdata] [-t rrtype] [-J input_file] DESCRIPTION dnsdbq constructs and issues queries to the Farsight DNSDB and displays responses. It is commonly used as a production command line interface to the DNSDB API server. DNSDB is a database that stores and indexes both the passive DNS data available via Farsight Security's Security Information Exchange as well [etc]
IV. Quota Considerations
When doing bulk queries, please note that each query (whether a separate dnsdbq query run as part of a script, or a line in a bulk dnsdbq -f run), will consume one query from the user's quota.
If the user is using a free DNSDB Community Edition ("DCE") key, burst rate limitations clamp the number of bulk queries you can do per hour to no more than 25. If you accidentally attempt to do more than 25, you'll get a 429 error status with the text "Error: Burst window rate limit exceeded. (This is an intentional constraint associated with all DCE keys, but which doesn't apply to regular DNSDB API keys)
V. Conclusion
You now know two approaches you can use to easily run a "file full" of queries in DNSDB with dnsdbq
.
If you don't currently have a DNSDB API key, but would like to try DNSDB, please see https://www.farsightsecurity.com/get-started/
I would like to thank Farsight Security staff David Waitzman and Jeremy Reed for their feedback on this article.
The Farsight Security Sales Team can be reached at sales@farsightsecurity.com or give them a call at +1-650-489-7919.
Joe St Sauver Ph.D. is a Distinguished Scientist with Farsight Security®, Inc.