Running Bulk Queries in DNSDB For A File of IP Addresses

I. Introduction

Most DNSDB API users are familiar with using a command line tool such as dnsdbq to query DNSDB, while others may prefer to use the Farsight DNSDB Scout(tm) browser plugin for Chrome or Firefox.

If you use dnsdbq, you know you can easily run simple IP address queries such as:

$ dnsdbq -i 104.244.14.108
;; record times: 2018-09-27 03:57:22 .. 2019-07-22 14:44:31
;; count: 4417
fsi.io.  A  104.244.14.108

;; record times: 2018-09-27 11:26:14 .. 2019-07-22 15:51:30
;; count: 1538
www.fsi.io.  A  104.244.14.108
[etc]

However, sometimes you may have a file with many IP addresses you want to check in DNSDB.

For the purpose of this blog, we'll assume that you're working from a Unix-like system (either a Mac, a Linux box, or a Windows system running the Windows Subsystem for Linux (see https://www.farsightsecurity.com/blog/txt-record/WSL-20180315/ )).

II. Using A Shell File To Run Multiple dnsdbq queries

We'll also assume that you've got a file containing a number of IPv4 (and/or potentially IPv6) addresses such as:

104.42.128.171
128.193.4.112
128.223.32.35
128.223.60.22
131.252.115.150
134.10.50.30
140.211.209.32
140.211.127.171
198.199.109.37
2620:105:b050:1550::90
2001:468:d01:20::80df:2023

You'd like to run those through DNSDB using dnsdbq. You could just edit your list of IP addresses to create a command file consisting of a series of dnsdbq commands such as:

$ cat run-some-ips.sh
dnsdbq -i 104.42.128.171
dnsdbq -i 128.193.4.112
dnsdbq -i 128.223.32.35
[etc]

You'd then execute that command file by saying:

$ chmod a+rx run-some-ips.sh
$ sh run-some-ips.sh > run-some-ips.output
$ more run-some-ips.output
;; record times: 2018-05-02 11:00:50 .. 2019-07-22 13:22:10
;; count: 5507
qnamaker.ai.  A  104.42.128.171

;; record times: 2018-03-09 05:08:54 .. 2018-11-13 09:35:57
;; count: 16
abma.asn.au.  A  104.42.128.171
[etc]

That works, but is somewhat "clunky" and inelegant.

III. The dnsdbq -f option

A more refined option is to use the dnsdbq -f option, which reads in and runs a file of DNSDB queries from <stdin>. That file must be specially formatted (perhaps with your favorite editor) to look like the following for IP queries:

$ cat my-ips.dat
rdata/ip/104.42.128.171
rdata/ip/128.193.4.112
rdata/ip/128.223.32.35
[etc]

You could then run those queries by simply saying:

$ dnsdbq -f < my-ips.dat > my-ips.output
$ more my-ips.output
;; record times: 2018-05-02 11:00:50 .. 2019-07-22 13:22:10
;; count: 5507
qnamaker.ai.  A  104.42.128.171

;; record times: 2018-03-09 05:08:54 .. 2018-11-13 09:35:57
;; count: 16
abma.asn.au.  A  104.42.128.171
[etc]

You're not limited to running only IP queries using the dnsdbq -f option, although the format of the lines in your data file will be a little different.

If you were doing left hand side RRname queries instead of IP queries, your file of queries might look like:

rrset/name/www.uoregon.edu
rrset/name/www.oregonstate.edu
rrset/name/www.eou.edu
[etc]

And if you were doing right hand side Rdata name queries, your file of queries might look like:

rdata/name/phloem.uoregon.edu
rdata/name/ns1.oregonstate.edu
rdata/name/mx.smtp.ucla.edu

You can also create an input file that's a mixture of IP queries, RRset name queries, and Rdata name queries, if that happens to be convenient for you – just use the appropriate prefix for each line in the file.

We should also mention that you can also use other common dnsdbq options, such as asking for output in JSON lines format, or using time fencing, or specifying the maximum number of results to return for each query, should you desire to do so.

See the dnsdbq manual page for details about all dnsdbq options:

$ man dnsdbq
dnsdbq(1)                 BSD General Commands Manual                dnsdbq(1)

NAME
     dnsdbq -- DNSDB query tool

SYNOPSIS
     dnsdbq [-dfgmhjsShcI] [-A timestamp] [-B timestamp] [-b bailiwick]
            [-i ip] [-l query_limit] [-L output_limit] [-u server_sys]
            [-n name] [-k sort_keys] [-P page] [-p output_type] [-R raw_rdata]
            [-r rdata] [-t rrtype] [-J input_file]

DESCRIPTION
     dnsdbq constructs and issues queries to the Farsight DNSDB and displays
     responses. It is commonly used as a production command line interface to
     the DNSDB API server.

     DNSDB is a database that stores and indexes both the passive DNS data
     available via Farsight Security's Security Information Exchange as well
[etc]

IV. Quota Considerations

When doing bulk queries, please note that each query (whether a separate dnsdbq query run as part of a script, or a line in a bulk dnsdbq -f run), will consume one query from the user's quota.

If the user is using a free DNSDB Community Edition ("DCE") key, burst rate limitations clamp the number of bulk queries you can do per hour to no more than 25. If you accidentally attempt to do more than 25, you'll get a 429 error status with the text "Error: Burst window rate limit exceeded. (This is an intentional constraint associated with all DCE keys, but which doesn't apply to regular DNSDB API keys)

V. Conclusion

You now know two approaches you can use to easily run a "file full" of queries in DNSDB with dnsdbq.

If you don't currently have a DNSDB API key, but would like to try DNSDB, please see https://www.farsightsecurity.com/get-started/

I would like to thank Farsight Security staff David Waitzman and Jeremy Reed for their feedback on this article.

The Farsight Security Sales Team can be reached at sales@farsightsecurity.com or give them a call at +1-650-489-7919.

Joe St Sauver Ph.D. is a Distinguished Scientist with Farsight Security®, Inc.