Overview
Alert fatigue is one of the biggest challenges in network operations. This article explains how to tune NetOp thresholds to minimize false positives while ensuring real issues are caught.
Understanding the Problem
Too many alerts typically stem from:
- Thresholds set too low — triggering on normal behavior
- Wrong interface roles — client port disconnects generating alerts
- Missing baselines — static thresholds that don't account for normal peak times
- Notification over-configuration — per-metric anomaly emails enabled for everything
Step 1: Review Interface Roles
Go to Settings → Network Interfaces and ensure:
- All end-user switch ports (workstations, printers) are marked as USER role
- Trunk and uplink ports are marked as INTERCONNECT
- ISP-facing ports are marked as ISP
Setting client ports as USER prevents interface-down events from generating anomalies when a laptop is unplugged.
Step 2: Adjust Anomaly Thresholds
Go to Settings → Anomalies:
For CPU:
- If your devices regularly run at 60%+ normally, raise the High threshold to 80%
- Keep Critical at 90% (genuine performance impact)
For Latency:
- For satellite or long-distance WAN links, raise the threshold — 50ms may be normal
- For local LAN or fiber, keep at 20–30ms
For Packet Loss:
- A small amount (0.1–0.5%) may be acceptable on some ISP links
- Set High threshold at 1%, Critical at 3–5%
Step 3: Configure Escalation Times
In Settings → Sites, adjust the escalation behavior per device type:
- Increase "escalate after" time for Access Points from 1 minute to 5 minutes — brief AP disconnects are normal during client roaming
- Keep SD-WAN/Firewall escalation tight (1–2 minutes) — these are service-affecting
Step 4: Disable Notifications for Low-Priority Metrics
In Settings → Anomalies, the Notify Admin toggle is disabled by default for all metrics. Only enable it for:
- Metrics with a direct SLA impact (WAN latency, ISP packet loss)
- Metrics with a history of causing real incidents
Avoid enabling notifications for CPU, Memory, and Wifi-errors on all devices simultaneously.
Step 5: Use Incident-Based Monitoring
The most effective way to reduce noise is to focus on incidents rather than anomalies. Incidents only fire when correlated anomalies indicate a real service impact. Configure your notification rules to send emails only for incident events (Creation, Severity Change, Closing) rather than individual anomaly triggers.
Step 6: Leverage AI Baselines
NetOp's AI baselines automatically learn each device's normal behavior over time. Ensure:
- The platform has been running for at least 2 weeks before fine-tuning — baselines need data to stabilize
- Do not reset baselines unnecessarily; this discards learned behavior
Validation
After tuning, monitor for 1–2 weeks:
- The anomaly count should decrease by 30–60%
- Real incidents should still appear (the tuning should not suppress genuine issues)
- Check the Incidents list — if incidents are missing for known outages, a threshold was set too high
Getting Help
If alert noise persists after tuning, contact NetOp support with:
- The customer name
- The specific anomaly type generating excess noise
- Current threshold configuration
- Sample anomaly descriptions
Comments
0 comments
Please sign in to leave a comment.