DNS Security: DNSSEC, DNS over HTTPS, and Enterprise DNS Filtering

DNS is the protocol that makes the internet usable — and one of the most persistently underdefended components of enterprise network infrastructure. DNS queries leave your network on every browser request, every application call, every email lookup. They traverse multiple recursive resolvers, traverse potentially hostile networks, and return results that your systems act on without question. For attackers, DNS is both an attack vector and an exfiltration channel. For defenders, DNS telemetry is one of the richest, most actionable security data sources available.

This guide examines the full DNS security stack: the attack surface that DNS exposes, the cryptographic controls that protect DNS infrastructure integrity, enterprise DNS filtering architecture, and the operational techniques for detecting DNS-based threats in your environment.

The DNS Attack Surface

Cache Poisoning

DNS cache poisoning attacks inject fraudulent DNS records into a resolver's cache, causing subsequent queries for legitimate domains to return attacker-controlled IP addresses. The Kaminsky attack (2008) demonstrated how birthday-paradox probability calculations could reliably poison resolvers without requiring privileged access to the resolver itself. Protections include source port randomization (now standard), DNSSEC validation, and 0x20 encoding. Organizations running internal DNS resolvers that lack these mitigations remain vulnerable to cache poisoning from internal attackers or compromised network segments.

DNS Hijacking

DNS hijacking attacks manipulate DNS records at the authoritative level — typically through compromise of the domain registrar account or the DNS hosting provider. Once authoritative records are modified, all resolvers globally serve the fraudulent records. Notable incidents include the Sea Turtle campaign (2019), which targeted DNS registrars and TLD operators across the Middle East and North Africa to redirect traffic for government, ISP, and technology targets. For enterprise domains, registrar account security — strong passwords, hardware MFA, registry lock for critical domains — is foundational DNS security.

DNS Tunneling

DNS tunneling exploits the fact that DNS queries and responses can carry arbitrary data in labels and resource records. Tools like dnscat2 and Iodine encode arbitrary protocol data within DNS requests, enabling command-and-control communications and data exfiltration through DNS queries that bypass firewall policies. Detection relies on traffic analysis rather than signature matching: tunneling traffic exhibits characteristic patterns including high query volumes to single domains, anomalous query lengths, high-entropy subdomains, and unusual resource record types.

DNSSEC: The Chain of Trust

DNSSEC (DNS Security Extensions) adds cryptographic authentication to DNS responses. The DNSSEC chain of trust begins at the root zone, where ICANN maintains the Root Zone Signing Key (KSK). Each TLD operator signs their zone with their own KSK, which is validated against the root KSK. Each domain's zone is signed by the domain operator's Zone Signing Key (ZSK), validated against the TLD's record. When a resolver validates a DNSSEC-signed response, it follows this chain back to the root, verifying each signature.

DNSSEC protects against cache poisoning and provides data integrity verification — but it does not encrypt DNS traffic. Queries and responses remain visible to anyone on the network path. DNSSEC tells you the answer came from the legitimate authority; it does not prevent others from observing that you asked the question or what answer you received.

Enterprise DNSSEC deployment considerations: enabling DNSSEC validation on recursive resolvers is the highest-impact action (it validates responses from DNSSEC-signed zones). Signing your own authoritative zones provides data integrity protection for your domain but requires operational commitment to key rotation — a DNSSEC configuration error that results in failed validation effectively breaks your domain's resolution for validating resolvers.

DNS over HTTPS and DNS over TLS: Enterprise Implications

DoH (DNS over HTTPS) and DoT (DNS over TLS) encrypt DNS traffic between the resolver client and the recursive resolver, preventing interception and manipulation on the network path. For enterprise environments, these protocols present a dual challenge:

• Pro — client protection: Endpoints using DoH to a trusted enterprise resolver receive privacy and integrity benefits on untrusted networks (coffee shops, home networks, hotel WiFi).
• Con — visibility loss: DoH to external resolvers (Cloudflare 1.1.1.1, Google 8.8.8.8) bypasses enterprise DNS monitoring entirely. DNS-based filtering, DNS telemetry collection, and threat intelligence queries all depend on traffic traversing enterprise DNS infrastructure. Endpoints resolving DNS directly over HTTPS to external providers are invisible to enterprise security monitoring.

The enterprise DNS security architecture recommendation: deploy an enterprise DoT or DoH resolver that endpoints are directed to via MDM policy. This provides encrypted DNS transport for endpoint protection on untrusted networks while maintaining enterprise DNS visibility and filtering. Block direct DoH/DoT connections to non-enterprise resolvers at the firewall layer (TCP/UDP 853, and filtering of known DoH provider IPs).

Enterprise DNS Filtering Architecture

Cisco Umbrella

Cisco Umbrella (formerly OpenDNS) provides cloud-delivered DNS security that applies threat intelligence categorization to every DNS query. Queries for known malware command-and-control infrastructure, phishing domains, newly registered domains associated with DGA activity, and other threat categories are blocked at the DNS layer before any HTTP connection is established. Umbrella deploys as the recursive resolver — endpoints are directed to Umbrella via DHCP or MDM, and Umbrella forwards clean queries to authoritative servers. The platform includes Investigate for threat intelligence research and integrates with Cisco SecureX for correlated incident response.

Cloudflare Gateway

Cloudflare Gateway provides DNS filtering as part of the Cloudflare One SASE stack. Gateway applies Cloudflare's threat intelligence (derived from visibility into a significant fraction of global internet traffic) to block malicious domains, enforce acceptable use policies, and log DNS telemetry. Integration with Cloudflare Access makes Gateway a natural choice for organizations standardizing on the Cloudflare stack. The DNS-over-HTTPS endpoint supports managed endpoint clients for encrypted enterprise DNS.

Detecting DNS Tunneling

DNS tunneling detection relies on behavioral analysis of DNS traffic patterns rather than signature matching. Key detection indicators:

• Query length analysis: Legitimate DNS queries for standard domains typically have subdomains under 30 characters. Tunneling tools encode data in DNS labels and generate queries with unusually long subdomains — often 60-100 characters of base64 or hex-encoded data. Alert on queries where the total query name length exceeds 150 characters.
• Query volume per domain: DNS tunneling requires many queries to a single domain (the attacker's nameserver) to encode a meaningful amount of data. A workstation generating 1,000 queries per minute to a single second-level domain it has no business relationship with warrants investigation.
• Shannon entropy of subdomains: Random base64 or hex subdomains have entropy characteristics distinct from legitimate hostnames. High-entropy subdomain labels (entropy > 3.5 bits per character) are a reliable indicator of encoding. Zeek's DNS log provides the raw data for entropy analysis.
• Unusual resource record types: DNS tunneling tools often use TXT, NULL, MX, CNAME, or A records for data carriage. Workstations generating high volumes of TXT record queries are anomalous — legitimate TXT queries (SPF, DKIM lookups) are generated by mail servers, not workstations.

NXDOMAIN response analysis is equally valuable: endpoints generating high volumes of NXDOMAIN responses may be running DGA (Domain Generation Algorithm) malware that generates random domain names and queries them until it finds an active C2 controller. A workstation generating more than 100 NXDOMAIN responses per hour warrants investigation.

For enterprise organizations building DNS security into their detection pipeline, managed network infrastructure services from Fortress MSSP include DNS monitoring, filtering deployment, and threat intelligence integration. Network penetration testing engagements include DNS security assessment — testing for cache poisoning susceptibility, DNSSEC validation, and DNS filtering bypass opportunities that real attackers exploit. Contact us to assess your DNS security posture.