splunk - a search engine for your systems logs

Splunk is an Search Engine for your system logs. Information is fed to splunk via data inputs such as FIFO queues, log files, network ports and scripts that generated output. Information that is fed into splunk generates events which can be searched and reported on using the splunk interface.

Data inputs feeds that are fed into splunk can be fed into other splunk instances so you could easily create a daisy chained splunk environment. I wasnt able to do this as i was using the free license which allowed splunk to send data to other splunk instances though not able to receive. The free license also prohibits users being added to the splunk web interface forcing a single account which is the admin account.

Im not one to install applications on servers without using the distributions package management, so i looked at how i could run splunk as a normal user without it installing too much cruft sprawled over my filesystem. Im also not in favour of running applications as root when they dont need to.

To be able to run splunk as a normal user i had to add the user to the adm group in order to be able to read log files located in /var/log/. I also had to set the following environment variables before running spunk which i put into a wrapper bash script. The script assumes that splunk is found in the directory /home/rene/local/splunk.

#!/bin/bash
export SPLUNK_HOME=/home/rene/local/splunk
export SPLUNK_DB=${SPLUNK_HOME}/var/lib/splunk
export LD_LIBRARY_PATH=${SPLUNK_HOME}/lib
# run splunkd
${SPLUNK_HOME}/bin/splunk start

Once i had splunk up and running importing logs was simple. It was a simply a matter of adding the paths of the files or directories that i wanted splunk to watch. Splunk does come with a sample data set which can be referenced as index::sampledata when searching.

Splunk has a neat search interface which provides extensive features such as suggestive search, filtering on hosts, sourcetype and users, tagging, highlighting certain strings of a search result and saved searches. The splunk dev team have created their own searching language which resembles SQL with the power of the UNIX philosophy where you can direct output from one command into another.

Some examples of search syntax within Splunk.

error OR (success NOT login)
sourcetype::access_combined | where [search sourcetype::access_combined | top 4 clientip | fields clientip | format]
smtp NOT success hoursago::1

Splunk certainly is a new and innovative way for system admins to review their logs. Splunk also supports reporting, alerts and graphing which we all know managers love.