【实践成果】Splunk 9.0 Configuration Change Tracking

Splunk 9.0 引入了新的功能，一个很重要的一个，就是跟踪conguration 文件的变化：

这个很重要的特性，在splunk 9.0 以后才引入，就看server.conf 配置中，9.0 以后的版本才有：

server.conf - Splunk Documentation

Configuration Change Tracker

[config_change_tracker]
disabled = <boolean>
* Whether or not splunkd writes configuration changes to the configuration change log at $SPLUNK_HOME/var/log/splunk/configuration_change.log.
* If set to "false", configuration changes are captured in$SPLUNK_HOME/var/log/splunk/configuration_change.log.
* If set to "true", configuration changes are not capturedin $SPLUNK_HOME/var/log/splunk/configuration_change.log.
* Default: falsemode = [auto|diff|track-only]
* Determines the method used by 'config_change_tracker' to track and recordchanges to .conf files.
* A value of "auto" or "diff" means splunkd logs all configuration changes made to.conf files, including changes to setting values. In this mode, config changetracking only includes changes that could have an effect on your environment.For example, if a file with a stanza and setting-value pair is created, updated,or deleted, splunkd logs the change. But if an empty file or a stanza without anysetting-value pairs is added or deleted, splunkd does not log the change since itwill not have an impact. Similarly, splunkd does not track any comments that areadded to or removed from files.
* A value of "track-only" means splunkd logs .conf file changes, but excludesconfiguration setting values. In this mode, config change tracking includeschanges whether or not they can have an effect on your environment. For example,splunkd logs a change for any updates to file content, or that come from a changeby the operating system. Splunkd also sees a comment that has been added to a .conffile as a change, because that change results in a different file checksum.
* Splunkd tracks all .conf files under the following directories:* $SPLUNK_HOME/etc/system* $SPLUNK_HOME/etc/apps* $SPLUNK_HOME/etc/users* $SPLUNK_HOME/etc/peer-appsIt also tracks changes to the following:* $SPLUNK_HOME/etc/instance.cfg
* The values "auto" and "diff" have the same behavior at this time. Setting the value to "auto" ensures that the instance will always use the latest feature set.
* Default: autodenylist = <regular expression>
* If set, splunkd does not monitor files for configuration change tracker if their path matches the specified regex.
* No default.log_throttling_disabled = <boolean>
* Describes whether or not splunkd logs config changes to a .conf filethat occur within the 'log_throttling_threshold_ms' time span as a single event.
* A value of "false" means that splunkd logs all changes to a conf file withinthe time span 'log_throttling_threshold_ms' as a single event.
* A value of "true" means that splunkd logs all changes individually assoon as it detects them.
* This setting requires a Linux system with the "inotify" API forfile system event monitoring.
* Do not change this setting without first consulting with Splunk Support.
* Default: truelog_throttling_threshold_ms = <positive integer>
* The span of time, in milliseconds, during which splunkd logs multiplechanges to a .conf file as a single configuration change event.
* If multiple changes are made to a conf file within the time span'log_throttling_threshold_ms' milliseconds, splunkd logs those changesas a single event.
* Default: 10000exclude_fields = <comma-separated list>
* If set, splunkd excludes the stanza key that you specify when it writes to the configuration_change.log file.
* The format for each entry is '<conf-file>:<stanza>:<key>'. Separate multiple entries with commas.
* To exclude all keys under a stanza, use the '<conf-file>:<stanza>:*' format.
* This setting has no effect when mode is set to "track-only".
* Example setting: 'server.conf:general:pass4SymmKey, authentication.conf:authentication:*'
* No default.* NOTE: The [config_change_audit] stanza, which was previously mentioned in the Splunk version 8.2.0 documentation and configuration specification files,is now DEPRECATED.-

上面明确提到splunk8.2 是没有这个功能的。

下面实践一下这个功能：

先创建一个alert:

这样，就在如下的path 下面产生文件：

/opt/splunk/etc/users/sheng/search/下面产生savedsearches.conf 文件。

Everyweek / expire: 24 / is equal to=0

后来的修改：

Everyday / expire: 48 / is equal to=2

这个时候：看到如下的跟踪log 变化：

/opt/splunk/var/log/splunk/configuration_change.log

{"datetime":"09-30-2023 13:32:23.725 +0800","log_level":"INFO ","component":"ConfigChange","data":{"path":"/opt/splunk/etc/users/admin/search/local/savedsearches.conf","action":"update","modtime":"Sat Sep 30 13:32:22 2023","epoch_time":"1696051943","new_checksum":"0xd41d8cd98f00b204","old_checksum":"0x7e9b3519314c7f2b","changes":[{"stanza":"coubase_alert","properties":[{"name":"action.email","new_value":"","old_value":"1"},{"name":"action.email.to","new_value":"","old_value":"shp@163.com"},{"name":"action.email.useNSSubject","new_value":"","old_value":"1"},{"name":"action.webhook.enable_allowlist","new_value":"","old_value":"0"},{"name":"alert.expires","new_value":"","old_value":"48h"},{"name":"alert.suppress","new_value":"","old_value":"0"},{"name":"alert.track","new_value":"","old_value":"0"},{"name":"counttype","new_value":"","old_value":"number of events"},{"name":"cron_schedule","new_value":"","old_value":"0 0 * * *"},{"name":"dispatch.earliest_time","new_value":"","old_value":"-1d"},{"name":"dispatch.latest_time","new_value":"","old_value":"now"},{"name":"enableSched","new_value":"","old_value":"1"},{"name":"quantity","new_value":"","old_value":"2"},{"name":"relation","new_value":"","old_value":"equal to"},{"name":"request.ui_dispatch_app","new_value":"","old_value":"search"},{"name":"request.ui_dispatch_view","new_value":"","old_value":"search"},{"name":"search","new_value":"","old_value":"index=abc_base"}]}]}}

很好：下面开始change alert 来对比一下：

index=_configtracker "data.path"="*savedsearches.conf"
| spath output=modtime data.modtime,
| spath output=path data.path,
| spath output=stanza data.changes{}.stanza,
| spath output=name data.changes{}.properties{}.name,
| spath output=new_value data.changes{}.properties{}.new_value,
| spath output=old_value data.changes{}.properties{}.old_value,
| table modtime path name prop_name new_value old_value