Running Bulk Queries in DNSDB For A File of IP Addresses
By Joe St Sauver
If you use
dnsdbq, you know you can easily run simple IP address queries such as:
$ dnsdbq -i 184.108.40.206 ;; record times: 2018-09-27 03:57:22 .. 2019-07-22 14:44:31 ;; count: 4417 fsi.io. A 220.127.116.11 ;; record times: 2018-09-27 11:26:14 .. 2019-07-22 15:51:30 ;; count: 1538 www.fsi.io. A 18.104.22.168 [etc]
However, sometimes you may have a file with many IP addresses you want to check in DNSDB.
For the purpose of this blog, we'll assume that you're working from a Unix-like system (either a Mac, a Linux box, or a Windows system running the Windows Subsystem for Linux (see https://www.farsightsecurity.com/blog/txt-record/WSL-20180315/ )).
II. Using A Shell File To Run Multiple
We'll also assume that you've got a file containing a number of IPv4 (and/or potentially IPv6) addresses such as:
22.214.171.124 126.96.36.199 188.8.131.52 184.108.40.206 220.127.116.11 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52 2620:105:b050:1550::90 2001:468:d01:20::80df:2023
You'd like to run those through DNSDB using
dnsdbq. You could just edit your list of IP addresses to create a command file consisting of a series of
dnsdbq commands such as:
$ cat run-some-ips.sh dnsdbq -i 184.108.40.206 dnsdbq -i 220.127.116.11 dnsdbq -i 18.104.22.168 [etc]
You'd then execute that command file by saying:
$ chmod a+rx run-some-ips.sh $ sh run-some-ips.sh > run-some-ips.output $ more run-some-ips.output ;; record times: 2018-05-02 11:00:50 .. 2019-07-22 13:22:10 ;; count: 5507 qnamaker.ai. A 22.214.171.124 ;; record times: 2018-03-09 05:08:54 .. 2018-11-13 09:35:57 ;; count: 16 abma.asn.au. A 126.96.36.199 [etc]
That works, but is somewhat "clunky" and inelegant.
III. The dnsdbq -f option
A more refined option is to use the
dnsdbq -f option, which reads in and runs a file of DNSDB queries from
<stdin>. That file must be specially formatted (perhaps with your favorite editor) to look like the following for IP queries:
$ cat my-ips.dat rdata/ip/188.8.131.52 rdata/ip/184.108.40.206 rdata/ip/220.127.116.11 [etc]
You could then run those queries by simply saying:
$ dnsdbq -f < my-ips.dat > my-ips.output $ more my-ips.output ;; record times: 2018-05-02 11:00:50 .. 2019-07-22 13:22:10 ;; count: 5507 qnamaker.ai. A 18.104.22.168 ;; record times: 2018-03-09 05:08:54 .. 2018-11-13 09:35:57 ;; count: 16 abma.asn.au. A 22.214.171.124 [etc]
You're not limited to running only IP queries using the
dnsdbq -f option, although the format of the lines in your data file will be a little different.
If you were doing left hand side RRname queries instead of IP queries, your file of queries might look like:
rrset/name/www.uoregon.edu rrset/name/www.oregonstate.edu rrset/name/www.eou.edu [etc]
And if you were doing right hand side Rdata name queries, your file of queries might look like:
rdata/name/phloem.uoregon.edu rdata/name/ns1.oregonstate.edu rdata/name/mx.smtp.ucla.edu
You can also create an input file that's a mixture of IP queries, RRset name queries, and Rdata name queries, if that happens to be convenient for you – just use the appropriate prefix for each line in the file.
We should also mention that you can also use other common
dnsdbq options, such as asking for output in JSON lines format, or using time fencing, or specifying the maximum number of results to return for each query, should you desire to do so.
dnsdbq manual page for details about all
$ man dnsdbq dnsdbq(1) BSD General Commands Manual dnsdbq(1) NAME dnsdbq -- DNSDB query tool SYNOPSIS dnsdbq [-dfgmhjsShcI] [-A timestamp] [-B timestamp] [-b bailiwick] [-i ip] [-l query_limit] [-L output_limit] [-u server_sys] [-n name] [-k sort_keys] [-P page] [-p output_type] [-R raw_rdata] [-r rdata] [-t rrtype] [-J input_file] DESCRIPTION dnsdbq constructs and issues queries to the Farsight DNSDB and displays responses. It is commonly used as a production command line interface to the DNSDB API server. DNSDB is a database that stores and indexes both the passive DNS data available via Farsight Security's Security Information Exchange as well [etc]
IV. Quota Considerations
When doing bulk queries, please note that each query (whether a separate dnsdbq query run as part of a script, or a line in a bulk dnsdbq -f run), will consume one query from the user's quota.
If the user is using a free DNSDB Community Edition ("DCE") key, burst rate limitations clamp the number of bulk queries you can do per hour to no more than 25. If you accidentally attempt to do more than 25, you'll get a 429 error status with the text "Error: Burst window rate limit exceeded. (This is an intentional constraint associated with all DCE keys, but which doesn't apply to regular DNSDB API keys)
You now know two approaches you can use to easily run a "file full" of queries in DNSDB with
If you don't currently have a DNSDB API key, but would like to try DNSDB, please see https://www.farsightsecurity.com/get-started/
I would like to thank Farsight Security staff David Waitzman and Jeremy Reed for their feedback on this article.
The Farsight Security Sales Team can be reached at firstname.lastname@example.org or give them a call at +1-650-489-7919.
Joe St Sauver Ph.D. is a Distinguished Scientist with Farsight Security®, Inc.