Detection is only as good as the telemetry feeding it. A Security Operations Center running the most sophisticated SIEM platform available, staffed by skilled analysts, produces alerts of limited value if the underlying log sources are incomplete, poorly normalized, or missing the event types that make detection possible. Building an effective security detection pipeline requires deliberate decisions about what to log, how to collect it, where to store it, and how to turn raw log data into actionable alerts — before you purchase a single SIEM license.
This guide covers the log sources that matter most for security detection, log volume and storage cost estimation, SIEM vendor evaluation, detection engineering principles, alert fatigue mitigation, and the retention requirements imposed by the compliance frameworks most relevant to NYC enterprises.
What to Log: The Essential Sources
Firewall Deny Logs
Every packet that hits your explicit-deny rule should generate a log entry. Firewall deny logs reveal: internal hosts attempting to reach prohibited destinations (potential C2 communication that your DNS filtering blocked but the endpoint is still trying to reach by IP), scanning activity from internal hosts (compromise indicator), and policy violations that indicate misconfigured applications. Permit logs for traffic to sensitive destinations (database servers, management interfaces) are equally valuable for establishing normal access baselines. Raw firewall log volume for a mid-size organization (500 users) typically ranges from 50 to 500 million events per day depending on logging verbosity.
DNS Query Logs
DNS logs are among the highest-signal security data sources available. Every DNS query and response reveals what hosts are communicating with and when. Detection use cases: DGA malware generating random domain queries, DNS tunneling (high-entropy subdomain queries), newly registered domains (high-risk for phishing), C2 communication patterns (regular queries to the same domain), and internal DNS reconnaissance (LDAP, SRV, and PTR queries that reveal AD enumeration). Enable logging on your internal recursive resolvers (Windows DNS debug logs or Zeek DNS logs from a network tap). Volume: approximately 100,000-500,000 DNS queries per day per 100 users.
DHCP Logs
DHCP logs provide IP-to-MAC and IP-to-hostname mappings over time — essential context for correlating IP-based alerts with specific devices and users. When a firewall deny log shows an alert for IP 10.0.0.145, DHCP logs answer: what device was assigned that IP at that time, and what was its hostname? Without DHCP logs, IP-based alerts require manual investigation to attribute. Volume is modest — DHCP lease events number in the thousands per day for most organizations.
Authentication Logs
Active Directory authentication events (Windows Event IDs 4624, 4625, 4648, 4768, 4769, 4771) are the primary data source for detecting credential attacks, lateral movement, and privilege escalation. VPN authentication logs reveal remote access patterns and impossible travel scenarios. Cloud identity provider logs (Entra ID sign-in logs, Okta system log) capture MFA usage, conditional access evaluation, and anomalous authentication patterns. Authentication log volume is proportional to user count and sign-in frequency.
NetFlow
NetFlow (or IPFIX/sFlow) provides summarized network traffic records — source IP, destination IP, source port, destination port, protocol, byte count, packet count, start time, end time. NetFlow does not contain packet payload but provides comprehensive visibility into network communication patterns at much lower volume than full packet capture. Detection use cases: lateral movement (new internal-to-internal communication patterns), data exfiltration (large outbound transfers to unexpected destinations), and beaconing (regular communication to external IPs at consistent intervals). Collecting NetFlow from all routers and layer 3 switches provides full network visibility. Volume: approximately 50,000-500,000 flows per gigabit of network traffic per day.
Proxy and Web Filter Logs
Web proxy logs capture every HTTP and decrypted HTTPS request from managed endpoints — URL visited, user identity, bytes transferred, response code, and user-agent. Detection use cases: visits to known malicious URLs (blocked and alerted), downloads of suspicious file types, web-based C2 communication (HTTP requests to attacker infrastructure), and data exfiltration via HTTP POST to external services.
SIEM Vendors: Capability and Cost Comparison
Splunk Enterprise Security
Splunk is the SIEM market leader with the deepest ecosystem of content (detection rules, dashboards, integrations) and the most flexible query language (SPL). Splunk's strength is its ability to handle diverse log formats, perform complex correlation across billions of events, and support custom detection development. The primary limitation is cost: Splunk's ingest-based licensing ($50-100+ per GB/day) makes it expensive for organizations with high-volume log sources. Splunk Cloud reduces infrastructure management overhead. For organizations with the budget and technical staff to develop custom detections, Splunk provides the most capable platform.
Microsoft Sentinel
Sentinel is Microsoft's cloud-native SIEM built on Azure Log Analytics. Pricing is consumption-based ($2.46/GB ingested after the first 10 GB/day free) — substantially lower than Splunk at equivalent volumes for organizations already in the Microsoft ecosystem. Sentinel's native integration with Microsoft 365, Entra ID, and Defender products provides immediate detection coverage for Microsoft-centric environments. The content hub provides hundreds of pre-built detection rules. Query language (KQL — Kusto Query Language) is less mature than Splunk SPL but adequate for most detection use cases. For organizations with significant Microsoft infrastructure, Sentinel is often the most cost-effective SIEM choice.
Elastic Security
Elastic Security (SIEM built on the Elastic Stack — Elasticsearch, Logstash, Kibana) provides a capable open-source foundation with commercial SIEM features in Elastic Cloud or self-hosted deployments. Elastic has the lowest licensing cost of the major vendors — the core log aggregation and search are open source, and SIEM detection capabilities are included in Elastic Security commercial subscriptions. Deployment and operational complexity is higher than SaaS options; organizations deploying self-hosted Elastic need strong Elasticsearch administration expertise.
Detection Engineering Workflow
Detection engineering is the discipline of translating threat intelligence and adversary behavior models into actionable detection logic. A mature detection engineering workflow follows this lifecycle:
- Threat model development: Define the adversary techniques relevant to your organization based on industry, technology stack, and threat intelligence. Map to MITRE ATT&CK to ensure comprehensive technique coverage.
- Data source identification: For each technique in your threat model, identify which log sources contain the signals needed to detect it. T1558.003 (Kerberoasting) requires Windows Security Event Logs with Kerberos auditing enabled. T1071.004 (DNS as C2) requires DNS query logs.
- Rule development and testing: Write detection logic against historical log data. Tune for false positive rate before production deployment — a detection that fires 500 times per day for legitimate activity will be immediately ignored and disabled.
- Deployment and monitoring: Deploy to production with appropriate alert severity, response guidance, and assignment to the analyst team. Monitor false positive rate in the first two weeks and tune thresholds accordingly.
- Ongoing maintenance: Detection rules require maintenance as the environment changes, as attackers adapt their techniques, and as log source configurations change. A detection that relied on a field in Windows Event Logs that your logging configuration does not actually capture will silently fail to detect attacks.
Alert Fatigue: Causes and Solutions
Alert fatigue — the condition where analysts stop meaningfully investigating alerts because volume exceeds capacity — is the primary operational failure mode in SIEM deployments. The causes are consistent: too many rules enabled without adequate tuning, detection of low-confidence indicators without correlation, and insufficient investment in false positive reduction. Alert fatigue transforms a security detection investment into expensive log storage with a checkbox — alerts fire, nobody investigates them, and real attacks pass undetected.
Solutions: prioritize high-confidence, high-fidelity detections over comprehensive coverage. One detection that reliably fires only on real attacks is more valuable than fifty detections that each have a 95% false positive rate. Implement alert severity tiers (critical/high/medium/low) with explicit analyst response expectations by tier. Track mean time to investigate and close by analyst and by rule — use this data to identify consistently noisy rules that need tuning or retirement. Use behavioral analytics and machine learning baselines to detect anomalies rather than static thresholds.
Log Retention Requirements by Compliance Framework
Log retention is both a security requirement (you cannot investigate incidents in data that has been deleted) and a compliance requirement. Key retention requirements for NYC enterprises:
- PCI DSS: Requirement 10.7 mandates at least 12 months of audit log retention, with a minimum of three months immediately available for analysis. All log types that could impact the security of the CDE are in scope.
- HIPAA: The HIPAA Security Rule does not specify a minimum log retention period, but the HIPAA documentation retention requirement (6 years from creation or last effective date) is interpreted to apply to audit logs as well by most compliance counsel. Six years is the practical standard for HIPAA-covered entities.
- NY DFS 23 NYCRR 500: Section 500.06 requires covered entities to maintain audit trails for a minimum of five years. This is one of the more demanding retention requirements in the US regulatory landscape and must be factored into log storage architecture and costs for DFS-regulated entities.
- NYDFS vs. SEC: For SEC-registered entities, Exchange Act Rule 17a-4 requires broker-dealer record retention for three to six years depending on record type, with an increasing emphasis on electronic communications retention following recent enforcement actions.
Log storage architecture for compliance requires tiered storage: hot storage (immediately queryable, typically 90 days for operational investigation) and cold storage (archived for compliance retention, typically object storage at significantly lower cost). The cost difference between hot and cold storage makes tiered architecture essential for organizations with multi-year retention requirements.
For organizations building or maturing their logging and SIEM infrastructure, Fortress MSSP provides log collection architecture design, SIEM deployment, and detection engineering services. Network penetration testing engagements validate your detection coverage by simulating real attack techniques and confirming whether your logging and SIEM infrastructure generates the alerts that should fire. Contact us to assess your logging and detection pipeline maturity.