Detecting Outbound SMTP Spam and Malicious Traffic with NetFlow and sFlow

Summary

One common risk for data centers, ISPs and operational teams is one or more hosts sending mass email (botnet/spam) and degrading their IP reputation. NetFlow and sFlow do not carry packet payloads; detection is therefore based on behavioral metrics rather than content inspection. This article explains flow-based detection signals, example metrics, anomaly detection approaches, and practical pre-blacklist response actions.

Typical Flow Characteristics of Outbound SMTP Traffic

• High connection counts: Many outbound TCP connections from a single source to destination ports 25, 465, 587 in short time windows.
• Short TCP sessions: Flow duration values are short and packet counts low due to failed or half-open handshakes.
• High packets-per-second (pps): Anomalous pps increase per source or on an egress interface.
• Destination IP diversity: A single source targeting many different destination IPs on the same port (high dst IP entropy).
• SYN intensity and SYN/ACK ratio: Many SYNs with few ACKs or a low SYN/ACK ratio is typical.
• Packets/bytes-per-flow profile: Low bytes/packets per session indicating only handshake or tiny transmissions.

What NetFlow and sFlow Can and Cannot Do

NetFlow and sFlow do not provide payload content or content filtering. They primarily provide:

• Source/destination IP and ports, protocol
• Flow durations, packet and byte counts
• TCP flags (in NetFlow and in sampled sFlow headers)
• Interface and direction metadata, timestamps

Therefore spam detection relies on flow-based metrics: connection counts, unique destination counts, pps, and SYN behavior.

Key Metrics and Example Rules for Detection

Compute these metrics at regular intervals (e.g. 1, 5, 15 minutes). Thresholds must be tuned to your environment.

Core metrics

• flows/s per source: Number of flows per source. Example alert: >= 200 flows/host in 5 minutes.
• unique dst IPs per source: Unique destination count. Example alert: >= 100 different dst IPs in 5 minutes (especially for dst port 25/587).
• pps (packets per second): High pps on interface or per source. Example: source pps is 10x normal.
• avg packets per flow / avg bytes per flow: Very low averages indicate short failed handshakes.
• SYN rate and SYN/ACK ratio: High SYN counts and low SYN/ACK ratio are indicative.
• flow duration distribution: A high proportion of short flows suggests anomaly.

Example detector rules (pseudocode)

1. if flows_from(src, window=300s, dst_port in {25,465,587}) >= 200 then alert
2. if unique_dst_ips(src, window=300s, dst_port=25) >= 100 then alert
3. if syn_count(src, window=60s) >= 500 and syn_ack_ratio(src) < 0.1 then suspect_bots
4. if pps_on_interface(iface, window=60s) >= baseline(iface)*10 then investigate

Note: If sFlow sampling is in use (e.g. 1:1000), scale these counts by the sampling factor or apply statistical adaptation.

Anomaly Detection Approaches

Simple threshold rules are effective, but more advanced approaches reduce false positives and adapt to change.

Time-series thresholds

• Compute a 7/30-day median and IQR per source or per egress interface.
• Trigger alerts when real-time measurements exceed median + 3xIQR.

Entropy and diversity analysis

• Compute destination IP entropy per source; high entropy can indicate broad scanning or spam distribution.
• Port entropy: unusual distribution across SMTP ports may be suspicious.

Clustering and machine learning

• At small scale: use K-means or DBSCAN to form normal vs anomalous flow clusters.
• More advanced: per-host feature vectors (flows/s, unique dsts, avg duration, syn_rate) produce anomaly scores.

Reducing False Positives and Exceptions

• Whitelist known mail servers, bulk senders and monitoring/backup traffic.
• Consider scheduled sending windows: campaign mailings concentrate at predictable times.
• Adjust for sampling: if using sFlow, apply the sampling ratio to estimate true counts.
• Perform account-level anomaly detection: historical behavior per user/VM matters.

Priority Response Steps for NOC/SOC (Before Blacklisting)

When a host shows suspicious outbound SMTP behavior, NOC and SOC must act quickly and in coordination. Steps are listed by priority.

1) Detection and verification

• Validate flow metrics over 5–15 minute windows (flows, unique dsts, syn/ack).
• Check inventory/CMDB: is this IP a mail server?
• Examine firewall/SMTP relay logs and authentication logs for corroboration.

2) Immediate mitigations (low-impact)

• Egress rate-limiting: apply a policing rate for the host/source (e.g. limit to 100 conn/min).
• Temporary ACL: allow only known mail relays and block direct outbound 25/465/587.
• Port-based quarantine: restrict the suspect host to management traffic only.

3) Isolation and detailed investigation

• Restrict host access and notify system owners for forensic inspection.
• Run on-host process and connection analysis (netstat/ss, ps) and search for malware indicators.
• Take a VM snapshot if needed for forensic preservation.

4) Network-level aggressive measures (if necessary)

• Use BGP Flowspec to push targeted upstream filters (edge-level port-based blocking).
• Deploy permanent ACL/firewall rules after assessing operational impact.

5) Coordination and delisting preparation

• After remediation, retain logs and flow records and contact blacklist providers/upstream recipients as needed.

Automation and Integration Recommendations

Automation reduces human latency and helps prevent blacklisting. Typical automation elements:

• Playbooks that push ACLs or rate-limits to network devices in response to flow detector alerts.
• SIEM/playbook integration: auto-create incidents, assign owners and notify via chatops.
• Flowspec playbooks: with approval, push temporary upstream filters when thresholds are exceeded.

Note: Ensure approvals and rollback mechanisms for aggressive automated actions (Flowspec/ACL) to avoid impacting legitimate services.

Best Practices and Prevention

• Egress filtering: restrict direct port 25 egress to authorized mail relays only.
• Enforce SMTP authentication and apply outbound rate-limits for authenticated senders.
• Endpoint security: timely patching, EDR/AV and application whitelisting.
• Retain flow data: keep at least 30 days of flow records for investigation and delisting (adjust for policy/regulations).

Conclusion and Call

Although NetFlow and sFlow do not reveal payloads, behavioral metrics can detect outbound spam and botnet activity with high fidelity. With the right metrics (flows/s, unique dst IPs, pps, SYN rates) and mature operational playbooks, NOC/SOC teams can remediate incidents before IP reputation is damaged. Account for sampling effects and false positives when tuning thresholds and automations.

Flowtrion offers technical consulting to evaluate your NetFlow/sFlow anomaly detection posture and help optimize NOC/SOC processes. Contact us for a short technical assessment if you would like assistance.