Blue Flower

In this article we'll explore the installation of Splunk in a Linux environment, you'll know by now that Splunk is a powerful tool that helps you get intelligence of what is happening on your network in long term basis. Before you start syslogging your data into Splunk, read this article about how to logically prepare a plan to gather the logs of your infrastructure:

Splunk could be quite confusing due to the nomenclature that they use. There is plenty info in http://docs.splunk.com as well as in http://answers.splunk.com, but both could you be an absolute nightmare, as they are full with links within links, within links, with not a clear solution at hand when something goes wrong. Let's revise some of the terminology that Splunk uses, so we get to an agreement in this article in regards to the nomenclature:

Indexer = This is the server where you install Splunk Enterprise, if you have use a single installation only, this server will also be known as  the "Indexer", "Search Head" and "Deployment Server", quite a few titles for a single machine, ah? From now on we'll refer to this server as Indexer only, the name of Splunk should not be use to designated this server (e.g. "the splunk server") as this is both confusing and incorrect ("do you mean the Indexer, the Search Head, etc?").

Forwarder = This is a client (windows and/or unix system) that sends data to the Indexer; it is in the clients where you install the Universal Forwarder, then use the Forwarder Management option in the Indexer to deploy add-on apps

In addition, the different packages that Splunk uses should be installed as follows: 

  • App = They go into the Indexer only
  • Add-on = They go to both the Indexer and the Forwarder
  • Supporting add-on = They go to both the Indexer and the Forwarder

Here a summary of the sections I've covered in this article, hope you like it and above all that you enjoy it!

  1. Installing Splunk
  2. The Indexes
  3. Installing Splunk App for Windows Infrastructure
    • 3.1 Configuring the deployment-apps
    • 3.2 Configuring the Splunk App for Microsoft Windows
    • References
  4. Installing app: Windows Events Logs Analysis
  5. Installing an app: Avaya Call

 

The Splunk Infrastructure

I'd recommend you to get 2 x licenses for Splunk, yes, you head me well, 2 x licenses (unless you're illegal and decide to install the same license on two machines), and get one Splunk -Indexer- instance to welcome all the logs from Servers, while the other one can be configure to receive logs from Client and peripherals. 

1. Installing Splunk

Get your VM ready with CentOS (I have with 6GB and 300GB to start with), then either create a personal account in Splunk or login to this site if you already have an account: https://www.splunk.com/en_us/download/splunk-enterprise.html  and download the release for whichever version you have the license for, and be brave and install it. 

To obtain the release you can also use wget; of the example below I'm using the version 6.5.3, and this is the wget URL taking from the download page:

wget -O splunk-6.5.3-36937ad027d4-linux-2.6-x86_64.rpm 'https://www.splunk.com/page/download_track?file=6.5.3/linux/splunk-6.5.3-36937ad027d4-linux-2.6-x86_64.rpm&ac=&wget=true&name=wget&platform=Linux&architecture=x86_64&version=6.5.3&product=splunk&typed=release'

Get privileges rights on your VM and issue the above command on it (ensure before that you get proper DNS resolution!). After that download finished, verify that you have the .rpm file on your area:

Then, do the following:

chmod o+x splunk.....rpm  ;ensure that file is executable by the root
su
groupadd splunk
useradd -d /opt/splunk -g splunk splunk
rpm -i splunk...rpm

After you "install" splunk, visit the location /opt/splunk/bin ,then issue:

chown -R splunk: /opt/splunk
firewall-cmd --set-default-zone=trusted
firewall-cmd --zone=trusted --add-port=8000/tcp --permanent
firewall-cmd --reload
 ./splunk start

Enter "Y" to agree the license, and if you're installing version 7.2 enter the "admin" and "yourNewPasswd" for the Splunk web interface. If you're installing older version, ensure that the splunk services are up, then visit http://yoursplunkserver:8000 to choose your super-duper password

If you have your splunk.license file at hand, install it using the web interface. Notice that you'll have two very different locations:

  • Path to installation: /opt/splunk
  • Path to indexes:     /opt/splunk/var/lib/splunk
  • Path to apps:         /opt/splunk/etc/apps

And, of course, ensure splunk is enable at boot start

./splunk enable boot-start

2. The indexes

Splunk will use the "main" index by default, putting all data in there, which inevitably will become messy in the future, so please ensure from the start that you carefully created a dedicated index for each App that you're going to install on Splunk. Create a naming convention for your indexes and stick to it!

3. Installing Splunk App for Windows Infrastructure

This should be a nice one, as I take we are all running the popular Active Directory from Microsoft. Before you install this app on your Splunk, you would need to download the following packages from the Splunk Base:

Once you unzip all of these goodies, use WinSCP an put all of them in the mighty folder /opt/splunk/etc/apps (on the screenshot below I've only shown a few)

Again, once the folders have been copied, ensure you run the following in the apps folder to give full permission to the Splunk account

chown splunk:splunk -R *

Guess what? Yes! after all above... restart Splunk 

Now that you have copied all the apps into the .../etc/apps, it is time to do the same thing but from the clients point of view 

 

3.1 Configuring the deployment-apps 

The Universal Forwarder is the application that we install in the Forwarder, so that it sends the logs and data to the Indexer. Get the download from here (https://www.splunk.com/en_us/download/universal-forwarder.html) and install it in all the Windows Servers from where you want to gather information, domain controller, printer servers, wsus servers, etc 

Install the Splunk Universal Forwarder on the FRDC and configure its "deploymentclient" file under C:\Program Files\SplunkUniversalForwarder\etc\system\local to point to the Splunk server IP address and port 8089

  1. Copy the "Splunk_TA_windows" app into the /opt/splunk/etc/deployment-apps of your Splunk Server

 

After you have added the app, visit Settings > Forwarder Management and verify that the client FRDC is listed, it should be listed because you install the Universal Forwarded on it and it reported to Splunk using 8089

Create a new "Server Class" and give it a meaningful name to host the App and the machines related to the app, on my example I called "Windows_AD"

 

3.2 Configuring the Splunk App for Microsoft Windows

Before jumping into the configuration, do as follows:

  1. Visit you Active Directory and create a standard user account under the OU "Users", I called that account "Splunk", and it will be used to search AD data on behalf of Splunk
  2. Configure Splunk to received data by creating a stanza under opt/splunk/etc/system/local
[splunktcp://9997]
disabled = 0

Now we are ready to configure the app, 

  1. Visit the configuration tab of this app and configure the details for your domain as follows, on my example my domain could have been "london.mydomain.local"
    1. The "Alternate domain name" means the NETBIOS of the domain
    2. For the hostname of the LDAP server you can use the name of the FRDC (Forest-Root DC), but I decided to use its IP instead, fail proof!
  2. On your Splunk web GUI, visit Settings > Data Inputs >

 Once this is done, visit the "Splunk App for Windows Infrastructure" and ensure all the prerequisites are met

 

On the next windows, you most likely will get an error of some sort saying that the " Search "sourcetype="WinPrintMon*" | head 5" did not return any events in the last 24 hours"; this means that your Indexer is not getting the data related to those events. For example, on the screenshot below I've highlighted the "WinPrintMon" source type, indicating that the Indexer is not receiving any print monitoring data from, let's say, your print server. If you know for sure you have installed the Universal Forwarder on your print server, you might wonder: why my expected sourcetype data is not getting into the Indexer?

One of the most common problems is that the client is not actually configured to send to the Indexer the relevant data. To fix this, ensure that the inputs.conf file on the client (located in C:\Program Files\SplunkUniversalForwarder\etc\system\local) is configure to send the data that you need; in the example below I've added the stanzas needed to send the relevant data to the Indexer

[default]
host = MY-PRINT-SERVER

[WinPrintMon://printer]
type=printer
interval=600
baseline=1
disabled=0

[WinPrintMon://driver]
type=driver
interval=600
baseline=1
disabled=0

[WinPrintMon://port]
type=port
interval=600
baseline=1
disabled=0

[WinPrintMon://jobs]
type=job
interval=60
baseline=0
disabled=0

 To fix the warning related to the "WinHostMon", add the following entries into the inputs.conf file of the client forwarders

# Queries computer information.
[WinHostMon://computer]
type = Computer
interval = 300

# 'interval' set to a negative number tells Splunk Enterprise to run the input once only. 
[WinHostMon://os]
type = operatingSystem
interval = -1

[WinHostMon://processor]
type = processor
interval = -1

[WinHostMon://disk]
type = disk
interval = -1

[WinHostMon://network]
type = networkAdapter
interval = -1

# This example runs the input ever 5 minutes.
[WinHostMon://service]
type = service
interval = 300

[WinHostMon://process]
type = process
interval = 300 

 For the network monitor, you can add this entry in the inputs.conf if you like, that should gather plenty info in regards to your network traffic

 [WinNetMon://winnetmon]
 direction = inbound;outbound
 disabled = 0
 index = windows
 packetType = accept;connect

 Any problems... visit the Splunk Tips & Tricks, it is really awesome :) https://www.splunk.com/blog/tag/splunk-universal-forwarder.html 

  

References

Great article this one: https://www.splunk.com/blog/2014/04/21/windows-print-monitoring-in-splunk-6.html  

https://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorWindowshostinformation  

 

4. Installing app: Windows Events Logs Analysis

Another good one to have is this: https://splunkbase.splunk.com/app/3067/ from EventID.net, particularley useful is the Windows Event Summary dashboard, specially the "Errors" and "Warnings" boxes, that tells you exactly where to look in the tons of logs you receive from your servers

 

As you should know by now, to install this app simply extract the folder from the zipped file and drop it into /opt/splunk/etc/apps using WinSCP, thereafter...you guess it... restart Splunk

 

 

5. Installing Splunk app for Unix

Now that we got all the Windows servers under control, let's go and add our Linux servers to Splunk too. Download this app to get started:

Install both of these packages (the app and the add-on) in your Indexer, by copying them into the so famous /opt/splunk/etc/apps, then issue this command to ensure the splunk account have got access to it:

chown splunk:splunk -R *

Needless to say that you should have already configured your Indexer to receive data on the port 9997

 

 

 

 

 

Installing an app: Avaya Call

Let's start by installing the Avaya Call (obviously, do it only if you happen to have an Avaya device on your network), and get experience with the app installation into Splunk. Download the app from this link https://splunkbase.splunk.com/app/1889/ and then using the WinSCP uploading (drag it) to the /opt/splunk/etc/apps location in your Linux Splunk VM

 

After you have copied the folder, gave the full ownership to it to the "splunk" user as well as the "splunk" group

chown splunk:splunk -R /opt/splunk/etc/apps/AVAYA_CALL

Restart your Spluk service for the installation to take effect

systemctl restart splunk
systemctl status splunk

Alternatively, visit Server Control > Restart Splunk 

Once the service has successfully restarted, logon to Splunk webserver and visit Settings > Indexes > and create a new index called "App_[the app name]" for the new app that you have installed, set it to 100GB maximum size, 500GB which is the default is a bit crazy and pretty useless to be honest, it would take too long (and eventually time out) for splunk to search data that is older than 6 months, we're talking about thousands of logs!

Once you got the index done, visit Settings > Data Inputs > Local Inputs TCP and create a new entry for port 6969 (the same port that you've configured in the Avaya PBX box)

Select source type > Uncategorize > tcp-raw and set the settings as advise on the configuration of the app in Splunk website

 

 

Install the NetFlow gadgets

Start by installing Splunk Stream from this link: https://docs.splunk.com/Documentation/StreamApp/7.1.2/DeployStreamApp/InstallSplunkAppforStream

Edit the file /etc/sysctl.conf and add the following:

# increase kernel buffer sizes for reliable packet capture
net.core.rmem_default = 33554432
net.core.rmem_max = 33554432
net.core.netdev_max_backlog = 10000

After you have added those entries, run the following to reload the settings: /sbin/sysctl -p

Installing the net flow add-on

 

 

Print Friendly, PDF & Email

Comments powered by CComment