Why Detection Engineers Focus on Cost
Modern SIEMs commonly charge by ingestion volume, making unnecessary logs a direct cost multiplier. Public guidance consistently stresses prioritising high-value logs and filtering noise before ingestion — a principle codified in CISA's SIEM/SOAR guidance and NIST SP 800-92.
Key Concept: Value-Based Filtering
Many logs — routine sign-outs, health checks, redundant events — add little detection value yet account for a large share of SIEM volume. Industry research confirms that routine, low-risk data should be routed to cheaper storage tiers while high-fidelity security events remain in the SIEM.
Five Strategies for Reducing SIEM Ingestion
1. Filter Low-Value Events
Drop routine sign-outs, heartbeat pings, and irrelevant metadata while archiving full logs to cold storage for compliance.
2. Summarise High-Frequency Activity
Create rollups for high-volume categories (successes, DNS, proxy flows) with first/last timestamps. Preprocessing at source reduces volume dramatically.
3. Prune Unused Fields
Map detections to required fields and drop unused ones. CISA and NIST highlight standardised schemas and necessary-field retention as best practice.
4. Tiered Retention
Hot (0–30 days) in SIEM; Warm (30–90 days) in Elastic or similar; Cold (up to 7 years) in Azure Data Lake or equivalent. Compliance intact, SIEM costs slashed.
5. Route Logs by Purpose
Send identity, endpoint, network-security, and cloud control-plane events to SIEM. Route verbose operational logs to log-management platforms.
6. Hot vs Cold Placement
Keep in SIEM: Auth failures, MFA challenges, EDR alerts, IDS/IPS alerts, cloud IAM changes.
Move out: Auth successes (summarised), debug logs, health checks, compliance retention copies.
30-Day Implementation Plan
- Baseline GB/day by source; identify the top cost drivers.
- Map detection rules to fields/sources; eliminate non-referenced data.
- Deploy pipeline preprocessing — filter, dedupe, enrich.
- Shift retention to cold/archive for non-SIEM-critical logs.
- Track KPIs: cost/day, fidelity, false-positive rate, MTTD.
FAQ
Does filtering reduce incident investigation quality?
No — archive full logs in cheap storage and retrieve when needed. This matches best practices recommended by public guidance and widely referenced in detection-engineering workflows.
How do you ensure detection coverage stays intact?
Maintain an ATT&CK-mapped detection catalog and ensure required fields remain unfiltered. CISA emphasises coverage mapping and field relevance.