Splunk UF 9.0 and POSIX Capabilities

Sorry this has taken so long to post. I caught a (thankfully very mild) case of covid at .cough2022 and between then and now life has not found a way (sorry Jurassic Park). Hopefully this is just the first of a few posts on stuff I’ve been working on and learning about since then.

Anyone who reads this very infrequently updated blog might have seen the over 3 year old post now, https://www.duanewaddle.com/splunk-and-posix-capabilities/. This was mostly a rant about what didn’t work, why it didn’t work, and the operational issues that it brought for Splunk administrators trying to run a “best practices” system where different best practices were in conflict:

  • Don’t run daemons as root
  • Do try to collect all of the logs from your systems
  • Don’t make sensitive log files world readable

Finally we have some good news resolving this. Splunk 9.0 includes running the Universal Forwarder in a “least privileged” mode. The docs say much more about this in detail, but the short version is that Splunk made the UF able to use POSIX capabilities in a way that enables admins to run Splunk “as splunk” (not as root), but still be able to read (and only read) all of the files on the system. On new installations this is the default (yay!)

Here’s one of my personal machines:

 [root@stinky local]# ps -fu splunk
UID          PID    PPID  C STIME TTY          TIME CMD
splunk     43027       1  0 16:28 ?        00:00:04 splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd
splunk     43052   43027  0 16:28 ?        00:00:00 [splunkd pid=43027] splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd [process-runner]

[root@stinky local]# lsof -p 43027 | egrep /var/log/audit
splunkd 43027 splunk   89r      REG              252,1  7014280  12924618 /var/log/audit/audit.log

[root@stinky local]# ls -l /var/log/audit/audit.log 
-rw-------. 1 root root 7024453 Nov 12 16:50 /var/log/audit/audit.log

Observe Splunk is running as user splunk, but it has a file open (/var/log/audit/audit.log) that is only readable by root. Witchcraft? Nah, CAP_DAC_READ_SEARCH. We can see that the systemd unit file for SplunkForwarder enables CAP_DAC_READ_SEARCH as an AmbientCapability, so that when the process starts it is blessed with this ability.

[root@stinky local]# systemctl cat SplunkForwarder
# /etc/systemd/system/SplunkForwarder.service
#This unit file replaces the traditional start-up script for systemd
#configurations, and is used when enabling boot-start for Splunk on
#systemd-based Linux distributions.

Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'

ExecStart=/opt/splunkforwarder/bin/splunk _internal_launch_under_systemd
SuccessExitStatus=51 52
ExecStartPre=-/bin/bash -c "chown -R splunk:splunk /opt/splunkforwarder"
ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/system.slice/%n"


CAP_DAC_READ_SEARCH means that Discretionary Access Control (the normal Linux filesystem permissions model) is bypassed for “read” and “search” operations. From the Linux man pages:

      Bypass file read, write, and execute permission checks.  (DAC is an abbreviation of "discretionary access control".)

      * Bypass file read permission checks and directory read and execute permission checks;
      * invoke open_by_handle_at(2);
      * use the linkat(2) AT_EMPTY_PATH flag to create a link to a file referred to by a file descriptor.

So the UF can read any file – including sensitive ones like /etc/shadow. But, it has no other “root” characteristics or abilities. Its (or its children) cannot change system configuration. While I’ve not tried it yet, I feel like you should be able to use the standard linux audit system to keep an eye on any files the UF should not be accessing.

Since I’m making Jurassic Park references today…

there it is

Leave a Reply

Your email address will not be published. Required fields are marked *