Poison Pivots



I. Introduction

Pivoting is a fundamental skill that passive DNS users inevitably use during the course of the analyses they conduct. Stripped to its essence, pivoting is a three step process:

  1. Find an initial clue (such as a suspicious domain name)

  2. Identify a potentially-shared resource associated with that initial clue (such as the IP address that the initial "clue domain" uses, or the name servers that the initial "clue domain" uses)

  3. Search DNSDB to find other domain names that also use that shared resource

Pivoting will often be highly productive, yielding new names that are genuinely related to your initial clue. The process works best when an initial clue lead to resources that are shared, but shared only with other domains belonging to the same person as the initial clue.

Unfortunately, sometimes you may inadvertently run into a "poison pivot" that doesn't work out the way you'd hoped. This article describes some of those problem areas, and explains strategies for how to identify and overcome those obstacles.

II. Non-existent Resources: The Road To Nowhere

At the risk of stating the obvious, you'll only be able to pivot on a resource if that resource exists. For example, you won't be able to pivot via an IPv6 address if the "clue domain" doesn't use IPv6, and you won't be able to pivot via a "clue domain's" MX record if one has never been defined for the "clue domain."

The only "solution" to this problem is to try pivoting on something else.

III. Completely Unshared Resources: "Cul De Sacs" Off The "Pivot Highway"

In order for pivoting to work, there must be a shared resource associated with the "clue domain."

For example, if we're trying to pivot via an IP address, that pivot will only work if the IP address is shared – it MUST be used by more than one domain name in order for pivoting via that IP to be feasible. If a bad guy is careful to only host a single domain name on an IP at a time, there's "nothing to pivot to." That IP basically becomes a "blind alley" or "dead end" for pivoting purposes.

When you run into that sort of scenario, you should consider alternative analytic approaches:

  • "Exploring The Neighborhood:" If a bad guy is only putting one domain on each IP to avoid potential passive DNS pivoting, he may end up needing a large number of IPs to accommodate all of his domains on a one domain-per-IP basis.

    More often then not, providers will simply assign contiguous ranges of IPs for their customers, and that range of IPs can easily be found and enumerated by investigating the other IP's in the same SWIP'd netblock. Begin by checking IP Whois/Rwhois for the encompassing netblock, or if that data's lacking, begin by just exploring the encompassing IPv4 /24 netblock by default.

    If you discover a bad guy with a highly fragmented IP allocation (with one IP in this netblock, another IP in another netblock, a third IP in a third netblock, etc.), particularly if contiguous blocks of network addresses appear to be available, that may be a sign that the provider is colluding with/supportive of its customer in avoiding investigations/takedown/blocklisting.

  • "Passive DNS As Time Machine:" Some bad guys may use the same IP for a succession of evil domains, but running with only one domain on each IP at any given time.

    When that happens, use passive DNS as a sort of "time machine" to review the domains that have historically been on an IP over time. This approach will allow you to find potentially-related domains that are being sequentially rotated in-and-out of play by a bad guy.

  • "Try Pivoting on Something Else:" While pivoting on a domain's IPv4 address is very common and often quite productive, it's not the only potentially shared resource a domain might have. For example, even if the IPv4 address isn't shared, how about its IPv6 address? Or how about the domain's name servers?

IV. Eponymous Name Servers (Sometimes Hosted On Shared IPs)

Speaking of strategies for pivoting by name servers, there may be times when you run into a domain that has eponymous name servers (e.g., www.example.com may have name servers using the "same domain," e.g., ns1.example.com and ns2.example.com).

Even if those name servers are really only used for a single domain, check out the IP address(es) that those name servers sit on: often there may be more than one eponymous name server hosted on a shared IP address!

V. Massively-Shared Name Servers: The DNS Version of Playing "Where's Waldo?"

When you register a domain name, you need to specify the name servers that are going to answer queries for it. You can run your own name servers, or you can use third party name servers that someone else has made available for your use. For example, a growing number of registrars and web hosting companies routinely offer free authoritative name service for their customers in an effort to gain (or retain) customers. When a registrar or web hosting provider offers free authoritative name service, they will normally do so via massively-shared name servers servicing tens of thousands of different customers and hundreds of thousands (or even millions!) of domains.

If you attempt to use passive DNS to pivot through these massively-shared name servers, you may find yourself staring at an overwhelming number of results, all of which are virtually totally unrelated to your initial clue (except for the fact that all those domains have taken advantage of a registrar's or hosting company's free name server services). Many bad guys know this, and routinely attempt to "hide in the crowd."

As an analyst, you need to be prepared to overcome this strategy and avoid collateral damage (you really don't want to seize/takedown/block a DNS server that's supporting hundreds of thousands or even millions of domains!)

The first step is learning to recognize name servers that are likely to be massively-shared. But how can you know "ahead of time" what name servers are likely to be massively-shared? As you gain experience working with DNSDB over time, you'll quickly come to recognize many of the more popular massively-shared name servers.

But let's see if we can help bootstrap that process for new analysts working with passive DNS.

What shared name servers are most likely to exhibit this sort of mass scale? Let's check DNSDB. We begin by extracting all NS records from the MTBL file for the month of January 2019:

$ dnstable_dump -r /path-to-mtbl-files/dns.201901.M.mtbl | grep ' NS ' | awk '{print $4}' | sed 's/\.$//' > 201901-nameservers.txt

There were over 1.3 billion NS records in the January 2019 MTBL file (commas added for readability):

$ wc -l 201901-nameservers.txt
1,313,705,482 201901-nameservers.txt

Counting each individual name server, the individual name server fully qualified domain names that appeared most often in the January 2019 data were:

Rank     NS Records		Fully Qualified NS Domain Name
1)	  9,337,435 		dns1.registrar-servers.com
2)	  9,333,606 		dns2.registrar-servers.com
3)	  9,323,648 		nsg2.namebrightdns.com
4)	  9,323,623 		nsg1.namebrightdns.com
5)	  7,949,869 		f1g1ns2.dnspod.net
6)	  7,942,571 		f1g1ns1.dnspod.net
7)	  6,759,936 		ns1.parkingcrew.net
8)	  6,759,054 		ns2.parkingcrew.net
9)	  5,493,074 		ns01.cashparking.com
10)	  5,493,024 		ns02.cashparking.com
11)	  4,043,074 		ns1.dns.com
12)	  4,042,831 		ns2.dns.com
13)	  3,523,623 		jm1.dns.com
14)	  3,523,619 		jm2.dns.com
15)	  3,393,529 		ns1.bluehost.com
16)	  3,392,485 		ns2.bluehost.com
17)	  2,897,092 		ns2.123-reg.co.uk
18)	  2,895,703 		ns.123-reg.co.uk
19)	  2,684,893 		ns13.domaincontrol.com
20)	  2,683,380 		ns14.domaincontrol.com

We then looked at the effective 2nd-level domains for those name servers, aggregating individual FQDN name servers (e.g., ns1.example.com, ns2.example.com and ns3.example.com would all be summed up as part of example.com's name server counts).[Naturally, if those name servers all offer the same content, this aggregation has the potential to inflate the apparent number of domains associated with that base domain, but in cases where a provider offers scores of name servers, most domains will only appear on two or three of those.]

This resulted in a top 20 list changes to include a number of domains that might otherwise not be on the analyst's "radar," such as cloudflare.com and googledomains.com:

Rank       NS Records		Base Domain Name
1)	    205,554,269 	domaincontrol.com
2)	    29,089,017 		cloudflare.com
3)	    25,507,242 		hichina.com
4)	    23,708,084 		registrar-servers.com
5)	    23,299,345 		googledomains.com
6)	    22,031,879 		namebrightdns.com
7)	    19,886,273 		dns.com
8)	    17,186,546 		wixdns.net
9)	    16,250,081 		dnspod.net
10)	    14,966,287 		dnsdun.com
11)         14,935,796 		dnsdun.net
12)	    13,524,207 		parkingcrew.net
13)         12,726,977 		rzone.de
14)         12,574,097 		ovh.net
15)	    12,171,402 		name-services.com
16)	    10,986,102 		cashparking.com
17)	    10,462,397 		ui-dns.org
18)	    10,462,360 		ui-dns.com
19)	    10,462,242 		ui-dns.de
20)	    10,462,047 		ui-dns.biz

If your "clue domain" uses any of those domains (or similarly massively-shared name servers), it may be hard for you to do a productive straightforward pivot via those name servers.

Some strategies for coping with massively-shared name servers include:

  • Ask For The Most Results You Can Get If You Feel You MUST Try Pivoting on a Massively-Shared NS: DNSDB's command line clients (such as dnsdbq) can return up to a million results for a given query, but you'll only get a fraction of those results by default (typically no more than 10,000). To ask for the most results you can get from DNSDB API while using the dnsdbq client:
    $ dnsdbq -l 1000000 -n ... > output.txt

    (that's dash ell one million). Note that this sort of query can take a long time to run for some particularly massive shared name servers. Also note that I show output from that command getting piped into a file because I doubt that you'll want to have a million results scroll across your screen…

  • If You Are Only Interested in NS records, Say So…: By default you'll get all records matching your query. When dealing with massively-shared name servers, if you only care about NS records, restrict your query appropriately:
    $ dnsdbq -l 1000000 -n ns1.ieee.org/NS > output.txt
  • Leverage Timefencing: If you know that a bad guy registered and began to use a new set of hundreds of bad domains all at once, there may be little point to looking at NS records from far before that time. Set an appropriate "time after" value. For example, in dnsdbq, you might say:
    $ dnsdbq -l 1000000 -A "2019-01-14 14:00:00" -n ns1.ieee.org > output.txt
  • Ask For Results For Specific Name Servers: While it may be tempting to use wildcards when pivoting via name servers, don't do that if you're working with massively-shared name servers. Investigate SPECIFIC name servers as-actually-used by the "clue domain." For example:
    $ dnsdbq -r ieee.org/NS
    ;; record times: 2013-05-01 14:59:38 .. 2019-02-15 17:20:50
    ;; count: 15590228; bailiwick: org.
    ieee.org.  NS  ns1.ieee.org.
    ieee.org.  NS  ns2.ieee.org.
    ieee.org.  NS  ns3.ieee.org.
    $ dnsdbq -l 1000000 -n ns1.ieee.org > ieee.txt
    $ dnsdbq -l 1000000 -n ns2.ieee.org >> ieee.txt
    $ dnsdbq -l 1000000 -n ns3.ieee.org >> ieee.txt
  • Explore EACH Of The Name Servers That A "Clue Domain" Uses: Normally a domain name will have two or three authoritative name servers defined when it gets registered.

    While all of the name servers for a domain will normally return consistent data for the domains they answer for, and those name servers will often share query load on a roughly equal basis, some (typically small) differences may still exist in the data from those servers.

    As a matter of due diligence, check EACH of the domain names that a "clue domain" uses, don't just check one of them and consider your work to be done. Check EACH of them.

  • Consider DNSDB Export: If you consistently find yourself needing more results than you can easily get via DNSDB API, consider upgrading to DNSDB Export. DNSDB Export will allow you to extract an arbitrary number of matching NS records.

  • "What About Limiting By Bailiwicks?" You may be tempted to try limiting results by bailiwick when pivoting via a name server, but note that this will NOT work/is not allowed:

    $ dnsdbq -l 1000000 -n ns1.ieee.org/NS/org > parent.txt
    error: -n must be NAME[/TYPE] only
    Try dnsdbq -h   for a short description of program usage. 

VI. Fast Flux: An Example of Extreme Pivot Complexity Management

Traditional web hosting (and traditional name server service) often doesn't work very well for bad guys: security researchers often quickly notice what's going on, and then the bad guy's web sites or name servers quickly get blocklisted or taken down (or the bad guys end up needing to pay premium prices to buy so-called "bulletproof hosting" from companies that are willing to ignore complaints).

One alternative to that traditional hosting has been something called "fast flux," where content is hosted on malware infected ("botted") consumer PCs. (Both a content server and its name servers can exploit fast flux – when that happens, this is often referred to as "double fast flux.")

To understand how fast flux (or double fast flux) is different from traditional hosting models:

  • Traditional hosting is purchased:

    • It uses dedicated server hardware
    • That hardware is normally hosted in large data centers
    • Servers are connected to the Internet via dedicated Ethernet links
    • The company providing the hosting is a public entity
    • The identity of the entity purchasing hosting is known (at least to the hosting company)
    • The hosted entity has no issue leaving a "financial trail," paying with a check or credit card.
  • "Fast flux" hosting is taken without authorization:

    • Malware-infected consumer computers ("bots") are used without the owner's permission
    • The bots are connected to the Internet over broadband connections (cable, DSL, etc.)
    • The bad guy typically tunnels traffic from the publicly visible botted host to a hidden backend server the bad guy controls
    • The identity of the entity offering the fast flux hosting may be unknown
    • The user of the fast flux hosting may also be anonymous
    • The user may attempt to remain financially anonymous by paying with a cryptocurrency.

Bad guys have historically found fast flux suited them because it was hard to take down or block.

Blocking any individual botted host will not take down the fast flux domain – the botnet will just automatically repoint the fast flux hosted domain(s) away from any unavailable botted hosts to other ones that are still up and available.

That ability to move, or "flux," is important, because victimized consumers turn their computers on and off all the time, and ISPs actively hunt and block malware-infected systems as they may be able to find them.

That sort of "hostile operational environment" means that the bad guys have had to work hard to keep fast flux hosted content "reliably available." They've done so by using more-than-the-typical-number of hosts for a given domain name and by keeping TTLs short, thereby allowing the fast flux-hosted-domain to change IPs whenever a botted system isn't working as expected.

A good review of fast flux domains and fast flux name servers can be found in the 2009 ICANN "Final Report on Fast Flux Hosting." One public report identifying some domains that used fast flux can be found in Akamai's report "Digging Deeper – An In-Depth Analysis of a Fast Flux Network."

If you find yourself pivoting through fast flux infrastructure, the good news is that you WILL uncover shared resources, and you WON'T run into massively shared centralized infrastructure.

The bad news is that fast flux-related investigations are highly prone to quickly chaining from one set of IPs to another, and to using lots of different domain names. This means that you need to be disciplined when it comes to documenting your investigation and being thorough chasing IPs and domain names. Some key things to remember include:

  • If searching for fast flux name servers, search EACH of the fast flux name servers associated with a domain name, and expect to see LOTS of different IPs for each nameserver. In some cases the RRsets may differ only in the order of the nameservers in the RRsets.

  • Time fence appropriately (IPs that haven't been seen in use for a month have probably already been identified and mitigated by an antivirus product or the relevant ISP; unless you have time to handle even stuff that's now "ancient history," focus on the current stuff).

  • If you decide to try chaining from:

    --  (initial "clue domain") --> 
       -- (fast flux name servers) --> 
          -- (IPs of the fast flux name servers) --> 
             -- (other hosts sharing the IPs used by the fast flux nameservers) --> 

    be SURE to be prepared to work with a LOT of domains and IP addresses. Some fast flux botnets may have tens of thousands or even hundreds of thousands of different botted hosts, and often there will also be thousands of domain names to "chase down," too.

VII. Partially Working A Fast Flux Example From the Akamai Report As An Example

Consider farmocnevnest[dot]net as mentioned in the Akamai report. That domain has now been sinkholed:

;; record times: 2018-09-23 04:53:44 .. 2019-02-15 18:56:41
;; count: 167; bailiwick: net.
farmocnevnest[dot]net.  NS  ns1.torpig-sinkhole.org.
farmocnevnest[dot]net.  NS  ns2.torpig-sinkhole.org.

but prior to that it was seen using the name servers:

;;   zone times: 2017-06-01 16:02:31 .. 2017-07-26 16:02:27
;; count: 56; bailiwick: net.
farmocnevnest[dot]net.  NS  ns11.ameriabank[dot]at.
farmocnevnest[dot]net.  NS  ns22.ameriabank[dot]at.
farmocnevnest[dot]net.  NS  ns33.ameriabank[dot]at.
farmocnevnest[dot]net.  NS  ns44.ameriabank[dot]at.

Looking at just one RRset associated with one of those name servers and working "depth first," we can see it used six different IP addresses associated with six different ASNs, although there appears to be a heavy concentration of IPs from ISPs in Crimea.

;; record times: 2017-06-07 21:47:18 .. 2017-06-07 21:47:18
;; count: 1; bailiwick: ns11.ameriabank[dot]at.
ns11.ameriabank[dot]at.  A		<-- AS6910, Digital Cable Systems S.A., Bucharest, RO [See Appendix A]
ns11.ameriabank[dot]at.  A		<-- AS5650, Frontier Communications, Rochester NY, US
ns11.ameriabank[dot]at.  A		<-- AS42919, LTD Darya, Kamensk-Shahtinskiy, RU
ns11.ameriabank[dot]at.  A		<-- AS6849, JSC UKRTELECOM, Kiev, UA
ns11.ameriabank[dot]at.  A		<-- AS6789, CRELCOM LLC, Simferopol, UA
ns11.ameriabank[dot]at.  A		<-- AS13188, CONTENT DELIVERY NETWORK LTD, Kiev UA

On another occasion, this name server used:

;; record times: 2017-06-07 21:30:57 .. 2017-06-07 21:30:57
;; count: 1; bailiwick: ns11.ameriabank.at.
ns11.ameriabank[dot]at.  A		<-- AS35816, Lancom Ltd., Sevastopol, UA
ns11.ameriabank[dot]at.  A		<-- AS3326, DATAGROUP PJSC, Kiev, UA
ns11.ameriabank[dot]at.  A		<-- AS42322, MTS PJSC, Moscow, RU
ns11.ameriabank[dot]at.  A		<-- AS48004, kct.net.ua, Simferopol, UA
ns11.ameriabank[dot]at.  A		<-- AS15895, Kyivstar PJSC, Kiev, UA
ns11.ameriabank[dot]at.  A		<-- AS12389, Rostelecom PJSC, Moscow, RU  

If we were doing this "for real" (e.g., in 2017 and not years later just as an example), we'd look at ALL the IP addresses associated with the name servers, AND all the IPs associated with farmocnevnest[dot]net as well.

We'd then look to see what was on each of those IPs, discovering new domains to "chase down:"

$ dnsdbq -i | grep -v ";;" | grep -v "^$" | sort -u
asdkjaskdjwejhsd[dot]com.        A
benumbermaytheappcontht[dot]ru.  A
empires[dot]cc.       		 A
fludgwererqo[dot]at.  		 A
ns1[dot]boarnoir.ru.  		 A
ns1[dot]folortrestergam.at.      A
ns1[dot]marvdns.ru.  		 A
ns1[dot]meetson.at.  		 A
ns1[dot]playtree.ru.  		 A
ns1[dot]pricorn.ru.              A
ns1[dot]urdupeople.ru.           A
ns1[dot]valorema.ru.             A
ns11[dot]ameriabank.at.          A
ns11[dot]klyatimoskali.at.       A
ns2[dot]aweasy.org.              A
ns2[dot]boarnoir.ru.             A
ns2[dot]folortrestergam.at.      A
ns2[dot]marvdns.ru.              A
ns2[dot]meetson.at.              A
ns2[dot]playtree.ru.             A
ns2[dot]pricorn.ru.              A
ns2[dot]urdupeople.ru.           A
ns2[dot]valorema.ru.             A
ns22[dot]ameriabank.at.          A
ns22[dot]klyatimoskali.at.       A
ns3[dot]aweasy.org.              A
ns3[dot]boarnoir.ru.             A
ns3[dot]folortrestergam.at.      A
ns3[dot]marvdns.ru.              A
ns3[dot]meetson.at.              A
ns3[dot]playtree.ru.             A
ns3[dot]pricorn.ru.              A
ns3[dot]teozaeppck.at.           A
ns3[dot]valorema.ru.             A
ns33[dot]ameriabank.at.          A
ns33[dot]klyatimoskali.at.       A
ns4[dot]aweasy.org.              A
ns4[dot]boarnoir.ru.             A
ns4[dot]folortrestergam.at.      A
ns4[dot]marvdns.ru.              A
ns4[dot]meetson.at.              A
ns4[dot]playtree.ru.             A
ns4[dot]pricorn.ru.              A
ns44[dot]ameriabank.at.          A
ns44[dot]klyatimoskali.at.       A
nxbcsjdghwyehywe[dot]net.        A
paysell[dot]bz.                  A
paysell[dot]me.                  A
ppeppeppety[dot]com.             A
rahmatulahh[dot]at.              A
royaldumps[dot]org.              A
rztrack[dot]com.                 A
unclesam[dot]ws.                 A

The process of investigating ns1[dot]boarnoir.ru, ns1[dot]folortrestergam.at, ns1[dot]marvdns.ru, ns1[dot]meetson.at, etc. will inevitably lead to more IPs, and likely then more domain names, and then more IPs, etc. Eventually that process will "converge" and you won't see any new leads to chase, but you can sometimes begin to feel as if this process is endless.

We'd then repeat the process for the next IP from our original set, e.g.:, and then the one after that, etc.

Obviously, this can quickly get complex. "Half the game" when investigating fast flux (and double fast flux) domains is keeping careful notes about how you chained from one discovery to another.

Fortunately, fast flux is less common than it once was.

VIII. Reverse Proxy Networks (Such As Cloudflare)

Perhaps the biggest challenge you'll face when trying to pivot arises when dealing with reverse proxy networks such as Cloudflare. Reverse proxy networks work by interposing proxy servers between Internet users and backend servers, and by comingling content from numerous entities. When the bad guys use a reverse proxy network:

  • You can't pivot via name servers, because ten domains from the same user may use twenty different shared name servers

  • You can't pivot via IP addresses, because just as domains are randomized onto different name servers, domains are also randomized onto different shared IP addresses

Your best bet when encountering reverse proxies is usually to try using DNSDB as a sort of "time machine," looking back and trying to find out where a given domain name lived before it "went behind the reverse proxy service."

Other times, you may find intriguing hints of netblocks to check in things like the domain's SPF records1. Visit https://web.archive.org/web/20190225203208/http://www.openspf.org/ or https://www.getmailbird.com/what-spf-resources-are-available-now-that-openspf-org-is-gone/

If the domain names that a party of interest uses are distinctive, you may also want to consider enumerating all domains used by all name servers associated with the reverse proxy network, and then pulling the relevant domains from that consolidated exhaustive enumeration.

IX. Summary

You've now seen some examples of how pivoting can get tricky:

  • In Section II, you were cautioned that for pivoting to work, the to-be-pivoted-upon resource must actually exist
  • In Section III, you learned that pivots must be shared in order for pivoting to be possible
  • In Section IV, you learned that sometimes you may think that a resource (such as a domain's name servers) may be unshared, only to find that unique name servers are hosted on a shared IP address, thereby allowing pivoting to still happen
  • In Section V, we explained the problem that massively-shared name servers can pose
  • In Section VI, we introduced passive DNS pivot complexity, and in Section VII we scratched the surface when it comes to illustrating this problem by partially working a fast flux example
  • In Section VIII, we briefly discussed the problem that reverse proxy networks can pose.

Throughout, we've attempted to share at least basic strategies that you can try to overcome these challenges when pivoting using passive DNS.

We hope you've found this to be an interesting topic to explore.

And isn't it time you talked to Farsight about how passive DNS can transform your online investigations? For more information contact our sales team at sales@farsightsecurity.com.

Appendix A

We map IP addresses to ASN using a little shell script that leverages data published by Oregon Routeviews.

$ cat /usr/local/bin/ip2asn
origip=`echo $1`
revip=`echo $1 | sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)/\4.\3.\2.\1/'`
listing=`host -w -t txt ${revip}.asn.routeviews.org 2>/dev/null | tail -1 `
#echo "${listing}\n"
listing2=`echo ${listing} | awk '{print $4}' | sed 's/"//g'`
cho "${listing2} ${origip}"

Sample run:

$ ip2asn

We then use Whois to map the AS numbers to organizational names. For example:

$ whois AS6910
org-name:       Digital Cable Systems S.A.
org-type:       LIR
address:        Bd. Dimitrie Pompeiu nr. 10A
address:        020337
address:        Bucharest
address:        ROMANIA

Joe St Sauver Ph.D. is a Distinguished Scientist with Farsight Security, Inc.

1 Previous reference was to http://www.openspf.org/. However, as this site is no longer available, we're now offering https://web.archive.org/web/20190225203208/http://www.openspf.org/ and https://www.getmailbird.com/what-spf-resources-are-available-now-that-openspf-org-is-gone/ as replacements.