Nazaudy, a spark in your curious mind

Disk monitoring with Splunk Enterprise

Disk monitoring with Splunk Enterprise can tell you how many dodgy hard drive you have running on your network so you can be proactive. This is al thanks to Luke Murphey and its Disk Monitoring app for Splunk Enterprise. In this article we'll explore how to do it

First of all , go ahead and download the app for your Splunk Enterprise from this link:

https://splunkbase.splunk.com/app/3279/

Splunk Disk Monitoring app

 

Installing and configuring the app on your Splunk Enterprise Server

 

Installation for Windows

Once you have installed the app on your Splunk Enterprise server, you need to install the Splunk Universal Forwarder on your windows client machines

1.Check to accept the license

Splunk Enterprise Universal Forwarder

2. Leave the "Deployment Server" blank, we'll do it later with a command, but enter the "Receiving Indexer" with the IP of your Splunk Enterprise

Universal Forwarder Receiving

3.Once you finish installing the Universal Forwarder, copy the Disk Monitoring app (the "smartmon" folder) from the location SPLUNK_HOME$\etc\app on the Splunk Enterprise to the Windows client. Use "netstat" to verify that you are connected to the Splunk server by port 9997

smartmon in Disk monitoring with Splunk Enterprise

4. Open the Command Prompt with Administrative privileages, and navigate to the "smartmon" folder > bin and run the 2 x windows scripts that are there, this will generate some data that will be passed to the Splunk Disk monitoring app. Then issue these two commands to ensure that splunk starts at boot up

C:\Program Files\SplunkUniversalForwarder\bin\splunk enable boot-start

C:\Program Files\SplunkUniversalForwarder\bin\splunk restart

 If the cmd files don't work, copy the "systemct.exe" to C:\windows and modify the script files to launch the utility from the C:\windows path

Once you copy the systemctl.exe  to the C:\Windows, edit the .cmd files located in C:\Program Files\SplunkUniversalForwarder\etc\apps\smartmon\bin and configure them as follows:

smartmon_results.cmd

@echo off

:: Determine if this is a 32-bit or 64-bit OS
Set ARCH=win64
IF %PROCESSOR_ARCHITECTURE% == x86 (
  IF NOT DEFINED PROCESSOR_ARCHITEW6432 Set ARCH=win32
  )

:: Create a reference to the smartctl command
SET SMARTCTL=C:\Windows\smartctl.exe

:: Get the list of drives
SET COMMAND_TO_RUN="%SMARTCTL%" --scan

:: Run smartctl on each disk
for /f "usebackq tokens=*" %%a in (`%COMMAND_TO_RUN%`) do (
	for /f "tokens=1 delims= " %%b in ("%%a") do (
		"%SMARTCTL%" -a %%b
	)
)

 

smartmon_short_test.cmd

@echo off

:: Determine if this is a 32-bit or 64-bit OS
Set ARCH=win64
IF %PROCESSOR_ARCHITECTURE% == x86 (
  IF NOT DEFINED PROCESSOR_ARCHITEW6432 Set ARCH=win32
  )

:: Create a reference to the smartctl command
SET SMARTCTL=C:\Windows\smartctl.exe

:: Get the list of drives
SET COMMAND_TO_RUN="%SMARTCTL%" --scan

:: Run smartctl on each disk
SET TEST="short"
for /f "usebackq tokens=*" %%a in (`%COMMAND_TO_RUN%`) do (
	for /f "tokens=1 delims= " %%b in ("%%a") do (
		"%SMARTCTL%" -t %TEST% %%b
	)
)

 

Installation on iMacs

This is the fun part! For the MAC OSX to talk to Splunk, be very much aware that the port 9997 (which is the beloved port for Splunk) is in use on the macOSX world for some kind of chat program called "palace". You can verify that by open the "Network Utility" on a Mac OSX and select to scan for the ports of your Splunk Enterprise server

Port 9997 in use

 

That means that we have to use another port for communicating to Splunk Enterprise. So go ahead and visit this website that list all ports assigned on the Internet and use a completely free one of your choice

https://www.adminsub.net/tcp-udp-port-finder/

For my example, I'm using port 10587, which as you can see if not being assigned to anything:

Port 10587 is free

You need to add that port as well (of course) on the Splunk Enterprise. To do that visit Settings >Forwarding and Receiving > Receive Data and add port 10587 so the clients can send data to that portSplunk forwarding and receiving

We need to create a Server Class as well for this app, so in your Splunk Enterprise go to Settings > Forwarder Management > Server Class and create a new one called "DiskMon"

New Server Class

Now copy the "smartmon" folder to the location /opt/splunk/etc/deployment-apps in the Splunk Enterprise and yes... restart Splunk for this last change to be effective

Choose any iMac that you like, and copy to the /Applications/SplunkForwarder/etc/deployment-apps the same folder. On that give iMac, create a file called deploymentclient.conf and add this info to it:

[target-broker:deploymentServer]
targetUri = 192.168.0.44:8089

You need to do the above only once, and just on one iMac, this is so that the .sh scripts are detected and manageable by Splunk Enterprise. Once you finished go to Data Inputs > Scripts > and clone any of the .sh scripts, pointing them to the right location in the /Applications/SplunkForwarder folder in the iMacs. Finally, visit Data Inputs > Remote Scripts and enable the two newly detected .sh scripts in your Splunk Enterprise:

Splunk Enterprise remote scripts

 

Moving ahead, and once you got the port configured, let's start by downloading the Splunk Universal Forwarder for macOS and install it ;visit this link and logon to your Splunk account to proceed: https://www.splunk.com/en_us/download/universal-forwarder.html#tabs/macos  Remember to do this job only when logging on as admin

 Install Splunk Universal Forwarder

After the installation, start the service

Splunk Little Helper

If you're prompted to install additional code, go ahead and install it

Install additional code

If installing for the first time, you should get a "successfully" notification

The installation was successfull

Then click on the "Splunk" icon on your desktop, and logon using the admin details you use to logon to your Splunk Enterprise server

 Splunk bash

And yes to the next window, start Splunk:

 Disk monitoring with Splunk Enterprise

You are done (for now) with Splunk

 

 

 

Install Smartmontools in macOS

Visit this link and install the smartmontools package: https://sourceforge.net/projects/smartmontools/files/

Install Smartmontools in macOS

 You should received a green tick after the install, well done! :)

SMART monitoring tools

The installation of smartmoontools places the "smatctl" program and associated in the folder /usr/local/sbin, which your iMac won't find on its path unless you add it to your environment. I find much easier to just visit that folder (Open Finder > Go > Go to Folder...) and copy the files to the location /usr/local/bin, which will be found by your iMac and therefore by Splunk. To facilitate all this copy process, open the terminal and run this command:

sudo cp /usr/local/sbin/* /usr/local/bin

smartctl path location

Just like on Windows, copy the "smartmoon" folder into /Applications/SplunkForwarder/etc/apps

smartmon folder

 Once you copy, open the Terminal and run this command:

sudo chmod uo+x /Applications/SplunkForwarder/etc/apps/smartmon/bin/*.sh

 

 

Configure the Splunk Forwarder

Using the Terminal, visit the location /Applications/SplunkForwarder/etc/system/local and run the following:

sudo vi inputs.conf

Populate the inputs file with this, change the IP address of your Splunk Enterprise host accordingly, and remember to change the NAME to match the name of the client!

[default]
host = NAME

[script://$SPLUNK_HOME/etc/apps/smartmon/bin/smartmon_results.sh]
disabled = false 
host = NAME
index = main
interval = 300 
source = smartmon
sourcetype = smartmon


[script://$SPLUNK_HOME/etc/apps/smartmon/bin/smartmon_short_test.sh]
disabled = false 
host = NAME
index = main
interval = 300 	 
source = smartmon
sourcetype = smartmon

Now either create or edit the outputs.conf file to look like this:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 192.168.0.40:10587

[tcpout-server://192.168.0.40:10587]

 

Now open another terminal and execute

sudo /Applications/SplunkForwarder/bin/splunk enable boot-start

sudo /Applications/SplunkForwarder/bin/splunk restart

 Job done, you should be able to see the hard disk reporting to the Splunk app, good luck with that!

 

 

 

Troubleshooting commands

netstat -ap tcp  //**verify that ports 10587 is open to the splunk enterprise server

./splunk list monitor //**the scripts should be listed there, if it doesn't use:

sudo ./splunk add monitor /Applications/SplunkForwarder/etc/apps/smartmon/bin/smartmon_results.sh

sudo ./splunk add monitor /Applications/SplunkForwarder/etc/apps/smartmon/bin/smartmon_short_test.sh

./splunk list forward-server //**it should be active

GIVE A GOOD REBOOT TO THE iMAC TO ENSURE THAT THE SPLUNK SERVICE STARTS AUTOMATICALLY

 

If you have enjoyed reading this article about Disk monitoring with Splunk Enterprise, you will also enjoy reading this other one:

 

 

 London, 16 August 2019