Pay it forward: Firewall Log Monitoring
Table of contents for pay-forward
- Pay it forward: Blocking applications in AD
- Pay it forward: Firewall tips
- Pay it forward: Firewall Log Monitoring
- Pay it forward: Know Your Network
- Pay it forward: Wrap Up
- Pay it forward: Success!
In yesterday’s series I talked about firewall best practices and I mentioned automatically parsing the logs. Well today’s post is a lengthy examination of how I’ve done that in the past. The system was crude but extremely effective and efficient. Like I mentioned in my post yesterday, it has the potential to alert you to zero-day compromises in your network.
The key is to look for ports commonly ‘wormed’ like RPC (will be wormed for the foreseeable future), SMTP, LSASS, etc. This way when hosts start scanning, say port 445 for vulnerable hosts, the script catches it regardless of what worm is causing it.
I called it WES for Worm Early warning System. I did a logic diagram that helps to visualize how the process works.
Below is a write-up I did on the first iteration of WES. I since upgraded it to a full MySQL back end and PHP front end. All configuration is done through the PHP front end and all variables are saved to a MySQL table. All logging is also done in a separate MySQL table. I was also toying with jpgraph to generate a cool graph showing the number of alerts per month generated by WES.
Architecture:
WES is the name for the whole system, which is comprised of several individual scripts, called sentrys.
Each sentry looks for a particular alert in the firewall logs. For example, there are sentrys scanning for violations specific to the following ports:
- SMTP port 25
- IRC port 6667
- Sasser port 445
There is also a sentry that is port-independent and simply looks for the number of ‘denied’ alerts per host. This sentry is good at detecting P2P applications, as they normally light up the firewall with several port attempts.
Alert Process:
Every five minutes the sentrys first scan the firewall logs and generate a list of the top 20 internal violators. This makes the scan more efficient by scanning only 20 IP addresses.
It then scans for port violations per IP in the top twenty list. If the number of port violations per IP meets or exceeds one of three predetermined thresholds, an alert is generated. Only one alert per threshold, per host can be sent per 24 hours, to prevent a flood of alerts during an outbreak.
As the alert is generated it is saved as a temp file. This temp file is the actual email alert and also serves to stop additional alerts from being generated. At the beginning of the day, at 1201a all temp files are purged and the system resets itself.
Phases:
There are three phases per sentry:
- Phase I (early warning)
- Phase II (Mid-level warning, host continues to scan)
- Phase III (Aggresive scanning, time for action)
Phase I is designed to warn very early, usually after 200 packets, depending on the sentry. This is early enough to allow a network manager to prevent an outbreak, but is also a low enough threshold that its possible to encounter false-positives.
Phase II is designed to notify the network manager that the activity persists. Phase II indicates a dramatic increase from phase I, usually around 1,000 packets. This means if a network manager receives a phase II alert, there is little possibility that this is a false positive and someone should look at the host to find out what is generating the scans.
Phase III is the last alert generated for a given host, per sentry. Phase III alerts on aggressive scans (5,000 packets or more). If a Phase III alert is generated, its time to disconnect the host. To date, every single time a host generates 5,000 or more violations against a firewall it has been a worm or virus.
In certain situations it is possible for WES to skip phases. If a host is scanning at a rate of 1,000 packets or more per minute, WES will go straight to phase III alerting. This happens because WES only checks the logs every five minutes. If a host generates 5,000 or more violations between WES log checks, WES moves immediately to phase III. This is designed to reduce email flooding during aggressive outbreaks.
Logging:
WES currently has basic logging features. Everytime an alert is generated WES adds an entry to an alert log so that it can track the date, time, sentry, host, and recepients of every alert generated. At the end of the week, when the weekly firewall reports are generated, the WES alert log is added. This will show the troubled hosts from the previous week and also indicate if additional people should be notified on the alerts.
