Configuring Splunk Universal Forwarder on Forefront TMG 2010
Aggregating logged data from security devices such as the Forefront Threat Management Gateway (TMG) 2010 firewall is a top priority for many security engineers. Forefront TMG and its predecessor, ISA Server, have always lacked an integrated facility to forward logged data to an external event management system. Often the administrator will have to devise an elaborate process that consists of batch files or scripts that collect firewall and web proxy logs and copy them to another location where they can be consumed. In the past I’ve demonstrated how third-party utilities can convert firewall log data to the syslog format as well.
Splunk is one of the more popular log management systems in use today, and to make it easier to get Forefront TMG log data in to Splunk we can use the Splunk Universal Forwarder. The Universal Forwarder is a utility that installs on the Forefront TMG firewall and monitors the folder containing W3C formatted text log files. The Universal Forwarder has a small footprint and consumes few resources, making it the ideal method to collect Forefront TMG log data and deliver it to the Splunk indexing server for analysis and archiving. The Splunk Universal Forwarder can be downloaded here.
Configuring Forefront TMG 2010
Before installing the Universal Forwarder, the Forefront TMG firewall must be configured to log to text file format. To change the log file format, open the Forefront TMG management console and highlight the Logs & Reports node in the navigation tree, select the Logging tab in the center console window, and then click Configure Firewall Logging in the Tasks pane on the right.
Select the option to log to File and choose the W3C extended log file format from the drop down box below. Repeat these steps to configure web proxy logging.
When the option to log to text file format is chosen, native Forefront TMG reports cannot be generated and access to historical log data in the Forefront TMG management console is no longer possible. Clicking Ok will generate the following warning message:
Reports cannot be generated with the currently selected logging method. To generate reports, use logging to SQL Server Express databases (on the local server).
An access rule must be created to allow the Splunk Universal Forwarder to communicate with the Splunk indexing server. The source will be the local host network, the destination will be the Splunk indexing server, and the protocol will be TCP 9997 (outbound), which is the default port used by the Splunk Universal Forwarder.
Configuring Splunk Universal Forwarder
Next, install the Splunk Universal Forwarder on the Forefront TMG firewall. When prompted, enter the hostname, FQDN, or IP address of your indexing server and specify a TCP port to use (the default is TCP port 9997).
Select the option to forward Local Data Only.
The Forefront TMG firewall will create new text log files each day and store them in the specified log files folder. Specify a Path to monitor by clicking Directory… and selecting C:\Program Files\Microsoft Forefront Threat Management Gateway\Logs (or the path where your log files are stored, if different from the default).
Configure Splunk Indexing Server
Once the installation is complete, open the Splunk Manager and click Forwarding and receiving.
Click the Add new link next to Configure receiving.
Configure the indexing server to Listen on this port and enter 9997.
Once you’ve configured Splunk to receive data from the forwarder, Forefront TMG firewall and web proxy log data should appear on the indexing server.
Connect
Richard Hicks' Forefront TMG Blog 
I remembered I showed you some Splunk work using TMG 2010 back in 2010 with a company called Satisnet. You can do some pretty funky stuff with it, it might be worth mentioning that you can dynamically extract all the fields with ease using Splunk, just take a look at the documentation or just use the field extractor wizard.
Hi Thomas,
Good to hear from you! I do recall when I visited you at Satisnet a while back that you were doing some great things with Forefront TMG firewall logs and Splunk. Splunk is a wonderful took, but the scope of this article was simply to describe how to get firewall and web proxy logs in to Splunk. Also, I have limited experience with Splunk so I don’t know all of the cool things you can do with it. 😉
Thanks for sharing that information though. 🙂
Can you use this simply to relocate log files from the TMG server to a remote storage server for long term storage? I get confused when a whole Splunk server comes into play. We just need to relocate these files so they can be stored for about a year and obviously don’t want to store all that on the TMG server (just 30-60 days on the TMG server itself).
And would you recommend W3C files as opposed to database files? Because right now we are storing the logs in the local SQL express database.
Thank you!
Not really. If you simply need to archive your log files, you can use W3C text files and write a simple batch file that copies the files to a remote file share. If you want to continue using the local SQL express database, I have a script that will export data from the SQL database to text file which you can then copy to a remote file share. Send me an e-mail if you’d like a copy of the script.
What setting do you use for your web proxy files? Do you log those to flat file as well? Or, do you allow those to go to the local SQL Server?
Will the TMG Splunk Forwarder work with Splunk installations earlier than version 4.2?
Typically I would use the W3C file format for both Forefront TMG 2010 web proxy and firewall logs.
Not sure. You’d have to test to confirm. Let me know if you can make that work. 🙂
W3C logs are written with a GMT time stamp, how would you tackle this considering we are situated GMT+10 and all our logs appear in Splunk 10 hours in the past.
Interesting dilemma, Alex. I’m not really sure how to resolve this, as I’m not that familiar with Splunk. I expect there’s some way to resolve this, however. I’d suggest posting your question on the Splunk user forums. Someone there will most likely have an answer for you. 🙂
In Splunk, you’d want to configure a props.conf entry for your data – something like this:
props.conf
[my_forefront_log_sourcetype]
TZ=UTC
This would tell Splunk that the events themselves are timestamped with UTC time, but Splunk will render them (in the UI) in whatever local time you have configured.
http://docs.splunk.com/Documentation/Splunk/latest/admin/propsconf
Hi Richard, U have a great blog!.. Im trying to integrate tmg 2010 with the RSA envision well I have the problem to create a user to create a connection using ODBC, do u know what is the pass from sa in the sql express db? Maybe u have some experience with RSA envision.
The SA password is managed by the system and isn’t known to anyone. You should, however, be able to read the logs as long as you have administrative privileges in TMG. I don’t have any experience with RSA Envision, but I can tell you that if you are trying to query the logs remotely that won’t work by default. Typically you’ll need to install an agent on the TMG firewall for this.
Hello.-
I know this is an old post, but i am not sure if understood. Does the forward read directly the Express SQL database ??.
Thanks.
No, it does not. It reads the text files generated by TMG instead.