Making Programmatic DNSDB Queries With libcurl



1. Introduction

Most users interact with DNSDB either through the web interface or via the sample command line interface (CLI) tools Farsight provides).

However, sometimes you may need to make complex or conditional queries that don't fit well with either of those options. In that case, DNSDB API access customers can call the DNSDB API directly from their own application.

The DNSDB API documentation describes how to make "bulk, automated DNSDB queries via the HTTP API" but it doesn't provide any actual code samples.

Rather than trying to build a complex application for this post, let's just build a basic example that uses libcurl to make calls to the DNSDB API.

2. libcurl

libcurl is the API version of the curl command line web client we all know and love.

Installation instructions for libcurl are available here and documentation is available at here.

Some alternatives to libcurl are available here should you want to check them out.

Please note that libcurl is under active development. The version I installed and used for the following example was:

$ curl-config --version
libcurl 7.50.3        [released on Sept 14, 2016]

For details about libcurl's release history and changes, go here and here. I strongly encourage you to use the most recent version of libcurl, and be sure that your installation of openssl is fully up-to-date, too.

3. Sample simple skeleton C code to make DNSDB queries

For this example, we just want to build a small C program that will take a list of fully qualified domain names from stdin and pump them through DNSDB, returning the results to stdout. We could easily do this with the existing CLI clients available for use with DNSDB, but we wanted a simple example that nicely illustrates the basics of making DNSDB API calls with libcurl.

The full range of API command options described at in the documentation can be easily added to this example, if desired.

Note: Speaking of extending this example, this skeleton code is just that, an example, and does not purport to be "production-grade" (e.g., it embeds the user's API key, it assumes the input is trusted and doesn't explicitly sanitize it, it does not do extensive error checking, it only takes the first dozen results for each domain, etc.).

Anyone using this example as the basis for actual production code should enhance/rewrite the sample as may be appropriate for their environment and their unique requirements.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <curl/curl.h>

int main (int argc, char **argv)
        CURL *curl;
        CURLcode res;
        char mydomain[1024], tempstring[1024], fullcommand[1024];

        /* initialize curl. must be called once and only once. */
        if (curl_global_init(CURL_GLOBAL_ALL) != 0)
                fprintf(stderr, "curl_global_init() failed\n");
                return (EXIT_FAILURE);

        /* get base domain from stdin */
        while (scanf("%s", mydomain) == 1)
                /* build the command we want to pass to curl */
                /* all our commands use the same basic RESTFUL API endpoint... */
                strcpy(fullcommand, "");

                /* now tack on the domain */
                strcpy(tempstring, mydomain);
                strcat(fullcommand, tempstring);

                /* just give me a token dozen results */
                strcpy(tempstring, "?limit=12");
                strcat(fullcommand, tempstring);

                /* get curl ready for action */
                curl = curl_easy_init();

                /* pass the API key */
                struct curl_slist *chunk = NULL;
                chunk = curl_slist_append(chunk, "X-API-Key: [elided]");
                res = curl_easy_setopt(curl, CURLOPT_HTTPHEADER, chunk);

                /* do the actual curl command */
                curl_easy_setopt(curl, CURLOPT_URL, fullcommand);
                res = curl_easy_perform(curl);
                if (res != CURLE_OK)
                        return (EXIT_FAILURE);

        return (EXIT_SUCCESS);

Normally we'd use a makefile to build the code, but it in this case it's such a small and simple sample we can build via:

$ gcc -Wall -O3 -o libcurl-sample libcurl-sample.c -I/usr/local/include -L/usr/local/lib/ -lcurl

Next, assume that we have a small test file that perhaps looks something like:

$ cat test-domains.txt

We can run the test by saying:

$ ./libcurl-sample < test-domains.txt > output.txt

4. Performance

This is a very simple single-threaded and untuned example. Nonetheless, how does it perform?

If we take 1,000 domains from and time how long it takes to run those domains through DNSDB with this simple application, we see:

$ time ./sample < top-1000.txt > /dev/null
real	1m17.653s     (e.g., 77.653 seconds)
user	0m22.417s
sys		0m0.828s

That implies 1,000/77.653=12.87 queries/second even for just this very simple code. How long would it take to run a million queries with the simple code? Hypothetically, by extrapolation, it should take roughly 1,000 times the time it took to run 1,000 queries: 1,000*77.653/(60*60)=21.57 hours

Obviously, parallelization (e.g., using ten concurrent threads, or simply running multiple parallel jobs), could reduce that time by an order of magnitude. [Farsight generally asks that users limit their queries to no more than ten concurrent threads unless special arrangements have been made in advance.]

5. Conclusion

You've now learned how to write C code with libcurl that will let you run basic DNSDB queries using your DNSDB API credentials.

If you aren't currently a Farsight DNSDB API customer, and want to lean more, please go here for information on how to purchase DNSDB API service.

Joe St Sauver, Ph.D. is a Scientist with Farsight Security, Inc.