Doh! No, not the Homer Simpson Doh!, but DNS over HTTPS- that DoH…

 

When you open a browser now, you may see a message like this:

What does this mean?

This means your browser, mostly without telling you, has configured DNS over HTTPS (DoH) or, less likely, the other IETF standard, DNS over TLS (DoT), which are both new features to enhances browsing privacy. Now, if you’re an end user at a company or @home worker, this will be great for you since you can now browse anywhere without your boss knowing what you’re doing or where your going, but for someone like myself in the Network Security NGFW business, this changes a lot for us.

When you type a web address or domain name into your address bar, your browser sends a clear request over the Internet to look up the IP address for that website, for example, “what is the IP address of cisco.com?”. DoH hides this information from anyone between you and your DNS server (Firefox uses CloudFlare now by default with a TLS connection, so as a quick solution you can always just block 1.1.1.1 and 1.0.0.1: port 853, which is CloudFlare, but this only solves the FireFox issue).

When you used the standard DNS queries (UDP/53), all your header information was/is sent in the clear, so this header information would show someone every website you visit, and this information is available to just about anyone that bothered to look, and in addition, ISP’s sells this information to marketing companies, even though they promised they wouldn’t do that. DoH, when enabled, ensures that your ISP cannot collect and sell personal information related to your browsing behavior, because DoH sends the domain name you typed to a DoH-compatible DNS server using an encrypted HTTPS connection instead of a plain text one.

DoH’s benefits are listed as preventing eavesdropping and manipulation of DNS data by man-in-the-middle attacks.”Fantastic you say! This will stop the government from watching me!” Well, maybe, but other issues abound here. First, if you are blocking malicious website with some sort of security intelligence, or simply trying to implement parental/schools/corporation URL controls, this will no longer work, which is a problem for security engineers working on NGFW trying to implement the corporate security policies.

Before I move on to a short term solution, there is another transport protocol you need to understand created by google that Chrome uses called QUIC. “Quick UDP Internet Connections” is a new encrypted-by-default Internet transport protocol, with the intended goal of eventually replacing TCP and TLS on the web, and creates the same problem we see with DoH.

In short:

  • DoH uses 443 tcp
  • QUIC uses udp 443

So by understanding the difference here between DoH and QUIC, you can see the conundrum we are in because now we have to handle both protocols.

Here’s the problem for NGFWs when DoH/QUIC is in use:

First, notice in this previous Network Analysis output, with the URL, category and reputation of the users URL requests, all shown, and the NGFW admin can filter based on this information.

 

 

 

 

 

 

Looks great! We can stop users from going to malicious sites, etc by filtering on known URL information. However, let’s see what the Network Analysis looks like with DoH/QUIC enabled:

In the output above, notice that we now get no useable information in my NGFW network analysis. How are supposed to control our users without that information, which is our God-given right as a network admin!

With all that said, let’s look at all our options here, none of them are pretty right now. However, no matter what I come up with here, in the long run there isn’t anything that will help us when TLS 1.3 is used. I’ll come back to this…

Using your Current ACP Rules to Solve this

First, you can use your Firepower ACP rules, at no additional cost, to block the protocols. Take a look at the top two rules in my Cisco Firepower ACP. I’ve always used the same few rules on top of my ACP, but I’ve added a new one to the very top. Take a look:

First, understand that if the domain name lookups using QUIC (UDP/443) fail for any reason, the browsers will fall back and use the default DNS configured by the operating system (OS) instead of displaying an error. So my first rule blocks the QUIC UDP/443, and this request will be denied, so the DNS query will be sent in the old, clear text way using TCP/443. So we’ll see this again for now:

By doing this, the Chrome browser using QUIC knows that the DNS request did not make it through, so they’ll revert to the defaults. This will be the only way to force the clients to not use QUIC if they try to, knowingly or not. The QUIC solutions was fast, easy and quick (pun intended), but what about DoH, which is the title of this post?

Additionally, and staying with the free solutions, you could always block DNS completely from going outside from your inside zone, and allow only DNS queries to your name server that you manage yourself. This will help you with QUIC and DoH for internal users.

Notice that I am using the Layer 7 DNS Application as a filter, which is more overhead, but the new NGFW’s can handle that now. Just add the destination network to your DNS server, and then block all other DNS packet completely. This is tight.

Another solution for DoH, is to disable DoH on FireFox with  “use-application-dns.net”, and if the browser sees this, it will not enable DoH by default.  The admin would have to enable this on all FireFox browsers in the organization, but this maybe doable, however, the user possibly could reenable this in FireFox if they know how.

Good, free, easy solutions, but no long term solutions by a long shot.

So where are we at now with Free solutions?

  1. Block CloudFlare: 1.1.1.1 and 1.0.0.1: port 853 to stop DoH
  2. Create rules in your Layer 7 ACP block UDP/443 to stop QUIC
  3. Create rules in your Layer 7 ACP to only allow DNS to your known servers and block all other DNS
  4. Disable DoH on FireFox browsers with  “use-application-dns.net”

Cisco Umbrella

Let’s now move onto Umbrella, which maybe our best solution

The better solution  is to use Umbrella to block these DoH/QUIC requests, because Umbrella includes known DoH servers in the “Proxy/Anonymizer” content category, and Umbrellas supports the “use-application-dns.net” domain, if the Umbrella resolvers are being used by the client, then Firefox will not enable DoH by default, but again, the users can override this.

So going back to the ACP rule example, if you are using Umbrella than the easiest and best solution is to force DNS to Umbrella

Lt’s take a look at the ACP rules now:

The first rule forced DNS to Umbrellas server, and the second rule blocks all TCP/UCP 53, which restrict all other DNS traffic to any other IPs. If you do these rules, and enable the Proxy/Anonymizer content category in Umbrella, this will work great!

SSL

However, if you don’t use Umbrella, than there is another solution for now, and that is using a SSL policy to decrypt the DNS request. Yes, perfect, our network is saved. For now yes, but since we can’t decrypt TLS 1.3, this is only a short term solution, isn’t it?

In the near future, we can do a downgrade to 1.2 to decrypt it, but this does cause some issues as cryptography is complex, with a lot of moving parts, that are constantly changing, so that is not an end-all solution either.

Solution Summary

  1. Block CloudFlare: 1.1.1.1 and 1.0.0.1: port 853 to stop DoH
  2. Create rules in your Layer 7 ACP block UDP/443 to stop QUIC
  3. Create rules in your Layer 7 ACP to only allow DNS to your known nameservers and block all other DNS
  4. Disable DoH on FireFox browsers with  “use-application-dns.net”
  5. Use Umberella and redirect all DNS traffic to only those servers, block all other DNS traffic
  6. SSL decryption, which works great for now.

Long Term Solution?

Umbrella is probably a long term solution, maybe, we’ll see what TLS 1.3 brings. But the longer term solution for NGFW would be some type of application detector such as an app-id for QUIC/DoH, which Cisco’s can’t do at this time, but I’m sure they’ll work on it shortly.

What do you think about all this? Let me know!

6 Comments

  1. Todd,

    I would advocate a more targeted approach instead of wholesale blocking of udp/443. Here are several alternatives:
    1. A URL Filtering rule to block the initial DNS lookup to mozilla.cloudflare-dns.com that Firefox uses to establish subsequent communication with DoH.
    2. If you have Umbrella in place, the the default security policy will block Proxy/Anonymizers which includes the service Firefox uses.
    3. We could also use measures such as a GPO to push a policy to the Firefox clients to disable DNS over HTTPS as described here:
    https://github.com/mozilla/policy-templates/blob/master/README.md#dnsoverhttps

    Blocking udp/443 wholesale prevents other uses such as remote access VPN over DTLS (which provides significantly better performance than the older TLS method).

    1. Thanks, Marvin, appreciate your post. I updated the post and would appreciate any feed back.

  2. Todd,

    Your revised approach is more in line with what I’m thinking. Thanks for the update.

    I’ve also been thinking about the viability of a NAT rule for all outbound DNS queries to change the target destination to one of the Umbrella servers. Paul Stewart mentioned this approach a couple of years ago in a post on his blog and I’ve been meaning to try it outside of a lab environment.
    https://packetu.com/2018/07/10/redirecting-dns-requests-to-umbrella-with-asa/

    1. Interesting, I’ll see if I can try that next week in my lab environment.

      I wrote this post so I could get feedback, and I got some good feedback too, on this subject.
      I noticed this issue last last year when doing 6.6 beta. Chrome didn’t provide URL info but my Safari browser did. I had opened a ticket, but then found out about QUIC and started doing some research. This certainly needs to be addressed with anyone using any type of NGFW.

      Thanks, Marvin, really appreciate your thoughts on this, and I hope we can continue to provide labs and solutions this year.

  3. Todd

    Where you ever able to try the workaround of redirecting the traffic rather than blocking. I discovered yesterday that 90+GB of our traffic was QUIC. I implemented your rules above and I now have the visibility into what the applications are but not sure if that’s a long term solution.

    1. Blocking the udp 443 is efficient, your other option is an SSL policy to decrypt the 443 udp, but thats harder and slightly more overhead, depending on devices. The best you can do is use Umbrella to perform this if you have umbrella.

Leave a Reply to Todd Lammle Cancel reply

Your email address will not be published. Required fields are marked *