Coronavirus (COVID-19) Information Read here

← Farsight Blog

Batch mode and raw query updates to dnsdbq

By

RSS

Introduction

For querying Farsight DNSDB, the world's largest historical passive DNS database, many users like our Farsight DNSDB Scout browser plugins. Others prefer a command line tool such as dnsdbq, our evolving C language tool. dnsdbq is open source software, hosted on github.com, that you build yourself.

We recently made a number of small improvements to dnsdbq, which this blog article will describe. These updates correspond to version 1.5 of the software, which is tagged in github as v1.5.0.

There are no changes to the DNSDB API. To learn more about DNSDB API, visit here.

Raw queries

The DNSDB API allows making "raw" queries using a hex representation of encoded DNS information. In the previous version, dnsdbq had a command-line switch that supported making raw hex Rdata queries (aka the right-hand side). The API also supports raw RRset queries (aka the left-hand side). The raw Rdata queries used the -R switch, even though a non-raw Rdata query used the -n switch (for name). There was no switch available for raw RRset, but batch mode did support such queries.

In this release of dnsdbq, we have renamed the raw -R switch to -N, corresponding to the non-raw -n switch.

We added a new -R switch, corresponding to the non-raw -r switch.

To recap:

What Do I Want to Search? 	Was   	 Now	  	Command Line Example 

Left-Hand Side 
       Regular			-r	-r (same)	dnsdbq -r fsi.io
       Raw Hex 			N/A	-R (new) 	dnsdbq -R 0366736902696f00

Right-Hand Side
       Regular 			-n	-n (same)	dnsdbq -n fsi.io
       Raw Hex 			-R	-N (swapped)    dnsdbq -N 0366736902696f00
       Right-Hand Side IP       -i      -i (same)       dnsdbq -i 4.0.0.0/8

We documented raw queries in 2016 in Raw Hex Rdata Queries: An Obscure (But Potentially Quite Useful) Bit Of DNSDB Functionality We wrote there that a raw query:

"is the ultimate search mode because it allows lookups for arbitrary octet strings (or prefixes of octet strings). That is, it allows you to bypass the DNS "presentation layer" and directly query the underlying data store. This works for existing RRtypes as well as unknown RRtypes, and in fact this is the only way to search the rdata of unknown RRtypes."

Raw queries also allow querying for illegally formed DNS names. These might be formed by improperly punycoded international domain names (IDNs).

The raw encoding method is described on the dnstable-encoding manual page.

Enhanced batch mode output

If you're running a big swath of queries in batch mode, it can sometimes be hard to tell what output corresponds to what query. We added an option for a enhanced (i.e. verbose) batch mode output to address that problem.

The existing output format from batch mode merely separated the results from different queries by a line with two dashes. For example, given input file input.batch containing:

rrset/name/farsightsecurity.com
rdata/ip/4.0.0.0,8

The original (and continuing) dnsdbq behavior with one -f option is:

$ dnsdbq < input.batch -j -l 1 -f

{"count":366,"time_first":1574284100,"time_last":1576847905,"rrname":"farsightsecurity.com.","rrtype":"A","bailiwick":"farsightsecurity.com.","rdata":["104.244.14.108"]}
--
{"count":4,"time_first":1574755050,"time_last":1576380492,"rrname":"4.xip.io.","rrtype":"A","rdata":"4.0.0.0"}

Now, if you specify two -f options, the outputs will be labeled with the original query after a ++ and the status of the query will be shown after the trailing --. You can specify two -f options as -f -f or -ff.

$ dnsdbq < input.batch -j -l 1 -f -f
++ rrset/name/farsightsecurity.com
{"count":366,"time_first":1574284100,"time_last":1576847905,"rrname":"farsightsecurity.com.","rrtype":"A","bailiwick":"farsightsecurity.com.","rdata":["104.244.14.108"]}
-- NOERROR (no error)
++ rdata/ip/4.0.0.0,8
{"count":4,"time_first":1574755050,"time_last":1576380492,"rrname":"4.xip.io.","rrtype":"A","rdata":"4.0.0.0"}
-- NOERROR (no error)

The two -f options feature is not (yet) compatible with the -m (merge) option. The merge option merges results from multiple batch mode queries together.

For a discussion of bulk use of batch mode, see our blog article Running Bulk Queries in DNSDB For A File of IP Addresses.

Bug fix: Handle missing newline in batch mode input file

dnsdbq previously improperly parsed a batch mode input file that was missing a newline at the end of the file. It reported an error for that line. This has been fixed.

Old buggy behavior (where the printf command is sending a two line input stream with the last line not being newline terminated):

$ printf "rrset/name/bbn.com\nrrset/name/fsi.io" |  dnsdbq -j -l 1 -f
{"count":342,"time_first":1574283947,"time_last":1576837693,"rrname":"bbn.com.","rrtype":"A","bailiwick":"bbn.com.","rdata":["192.1.122.24"]}
--
batch line too long: rrset/name/fsi.io

Fixed behavior:


$ printf "rrset/name/bbn.com\nrrset/name/fsi.io" |  dnsdbq -j -l 1 -f
{"count":342,"time_first":1574283947,"time_last":1576837693,"rrname":"bbn.com.","rrtype":"A","bailiwick":"bbn.com.","rdata":["192.1.122.24"]}
--
{"count":388,"time_first":1574271345,"time_last":1576848868,"rrname":"fsi.io.","rrtype":"A","bailiwick":"fsi.io.","rdata":["104.244.14.108"]}

You can also specify the command line options together and get more verbose batch output with command:

$ printf "rrset/name/bbn.com\nrrset/name/fsi.io" | dnsdbq -jffl 1
++ rrset/name/bbn.com
{"count":343,"time_first":1574451673,"time_last":1577013618,"rrname":"bbn.com.","rrtype":"A","bailiwick":"bbn.com.","rdata":["192.1.122.24"]}
-- NOERROR (no error)
++ rrset/name/fsi.io
{"count":396,"time_first":1574465819,"time_last":1577033230,"rrname":"fsi.io.","rrtype":"A","bailiwick":"fsi.io.","rdata":["104.244.14.108"]}
-- NOERROR (no error)

Bug fix: allow a bare . path name in a query

Curl did not allow using a bare . in a query URL. If passed a URL containing 'foo/./foo', it would change the URL to 'foo/foo'. To illustrate this, we will show a query for "com" in the root "." bailiwick. The command dnsdbq -r 'com/NS/.' sends a query for URL https://api.dnsdb.info/lookup/rrset/name/com/NS but the query should be for URL https://api.dnsdb.info/lookup/rrset/name/com/NS/.

Curl as of version 7.42.0 supports a new CURLOPT_PATH_AS_IS option to tell it to pass the . as is. If dnsdbq is built against that version or later of curl, it will use the new option to correct the misbehavior.

Curl's CURLOPT_PATH_AS_IS option is available on the command-line as the --path-as-is option. If you are generating your own queries using curl (without using dnsdbq), you should use the --path-as-is option yourself. For example, the following illustrates how curl, without the --path-as-is option, mangles the URL removing the bare "." in "NS/.":

$ curl -H 'Accept: application/json' -H "X-API-Key: $APIKEY" "https://api.dnsdb.info/lookup/rrset/name/com/NS/.?limit=2" -v 2>&1 | egrep 'GET /' 
> GET /lookup/rrset/name/com/NS/?limit=2 HTTP/1.1

Here is how the URL is passed properly by adding the --path-as-is option:

$ curl --path-as-is  -H 'Accept: application/json' -H "X-API-Key: $APIKEY" "https://api.dnsdb.info/lookup/rrset/name/com/NS/.?limit=2" -v 2>&1 | egrep 'GET /'
> GET /lookup/rrset/name/com/NS/.?limit=2 HTTP/1.1

Building dnsdbq on Unix-like systems

The dnsdbq README file describes how to install the few dnsdbq dependencies on a few Unix-like systems. After following those instructions, in a standard Unix-like environment you should be able to do the following to build, install and cleanup a dnsdbq installation:

$ sudo make all install clean

To test that it's installed and see its version number:

$ dnsdbq -v
dnsdbq version 1.5

Building dnsdbq on Windows

We have an earlier blog article Using Farsight's dnsdbq Command Line DNSDB Tool in a Microsoft Windows Environment: The "Windows Subsystem for Linux" Option that also describes how to build and install it on Windows.

Conclusion

We hope this article has explained the recent changes to dnsdbq and given you some pointers to other resources about it. You've learned that:

  • you can now do raw hex RRset queries as well as raw hex Rdata queries,

  • the option naming for raw hex queries has changed to better parallel the regular (non-raw) query argument nomenclature,

  • that bulk batch mode now is more usable because output can be tagged with the corresponding query, and

  • treatment of the root bailiwick has been cleaned up.

For your reference, you can download a PDF version of the dnsdbq manual page here.

David Waitzman is a Senior Distributed Systems Engineer for Farsight Security, Inc..

← Blog Home