Fix stripping of domainname from hostname for events of sourcetype “syslog”

Problem: Splunk does not display the FQDN of a UniversalForwarder for logfiles of sourcetype “syslog”, even though the forwarder is configured with its FQDN.
Cause: Splunks default props and transforms strip away the domain part of the hostname.
/opt/splunk/etc/system/default/props.conf:

[syslog]
pulldown_type =true
maxDist = 3
TIME_FORMAT =%b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = False

/opt/splunk/etc/system/default/transforms.conf:

[syslog-host]
DEST_KEY = MetaData:Host
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(\w[\w\.\-]={2,})\]?\s
FORMAT = host::$1

Fix: Create or edit /opt/splunk/etc/system/local/props.conf and add the following lines:

# Override default TRANSFORM for syslog sourcetype to avoid domainname stripping from host field
[syslog]
TRANSFORMS =

Automatic index routing of multiple instances logging to /opt/app/$instancename/log/

transforms.conf:

[override-app-index]
SOURCE_KEY = MetaData:Source
REGEX = ^source\:\:\/opt\/app\/(\S+?)\/log(s)?\/.*$
DEST_KEY = _MetaData:Index
FORMAT = app-$1

props.conf:

TRANSFORMS-override-app-index = override-app-index

Automatic field extraction of arbitrary field-value pairs separated by “:”

props.conf:

REPORT-autofields = autofields

transforms.conf:

[autofields]
REGEX = (?m-s)^[s]+:(.*)$
FORMAT = $2::$3
MV_ADD = true

Find logfiles that can’t be read due to permission errors

index=_internal source="/opt/splunk/var/log/splunk/splunkd.log" "*Permission*" | rex field=message "error.*path (?<path>.*): Permission.*" | stats count by host,path

Check Index Volume

index=_internal source=*metrics.log group=X | eval MB=kb/1024

Possible values for X:

• per_sourcetype_thruput
• per_source_thruput
• per_host_thruput

For example:

index=_internal source=*metrics.log group="per_index_thruput" | eval GB=kb/1024/1024 | stats sum(GB) by series

If you want to get the volume for a specific index, you can use the following search:

index=_internal source=*metrics.log group="per_host_thruput" [search index=APP_INDEX earliest=-15min latest=now | dedup host | rename host as series | table series] | rex field=series "(?[a-zA-Z0-9]*).*" | eval GB=kb/1024/1024 | timechart span=1d sum(GB) by hostname

Narrowing down issues with monitor inputs

(index=_internal AND message="*permission*") OR (index=APP_INDEX) OR ...
| fields _indextime,_time,host,source,message
| eval path=source
| eval indexdate=strftime(_indextime, "%d.%m.%Y %H:%M:%S")
| eval date=strftime(_time, "%d.%m.%Y %H:%M:%S")
| rex field=message "Insufficient permissions to read file=\'(?.*)\' \(hint: (?.*)\)"
| rex field=message "error getting attributes of path \"(?.*)\": (?.*)"
| eval reason=reason." on ".date
| search NOT path="*splunk*"
| eval MB=len(_raw)/1024/1024
| stats first(host) as host, first(indexdate) as latest-indexdate, last(indexdate) as earliest-indexdate, first(date) as latest-eventdate, last(date) as earliest-eventdate,sum(MB) as total-mb, values(reason) as problems by path