Sorry this has taken so long to post. I caught a (thankfully very mild) case of covid at .cough2022 and between then and now life has not found a way (sorry Jurassic Park). Hopefully this is just the first of a few posts on stuff I’ve been working on and learning about since then.
Anyone who reads this very infrequently updated blog might have seen the over 3 year old post now, https://www.duanewaddle.com/splunk-and-posix-capabilities/. This was mostly a rant about what didn’t work, why it didn’t work, and the operational issues that it brought for Splunk administrators trying to run a “best practices” system where different best practices were in conflict:
- Don’t run daemons as root
- Do try to collect all of the logs from your systems
- Don’t make sensitive log files world readable
Finally we have some good news resolving this. Splunk 9.0 includes running the Universal Forwarder in a “least privileged” mode. The docs say much more about this in detail, but the short version is that Splunk made the UF able to use POSIX capabilities in a way that enables admins to run Splunk “as splunk” (not as root), but still be able to read (and only read) all of the files on the system. On new installations this is the default (yay!)
Here’s one of my personal machines:
[root@stinky local]# ps -fu splunk
UID PID PPID C STIME TTY TIME CMD
splunk 43027 1 0 16:28 ? 00:00:04 splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd
splunk 43052 43027 0 16:28 ? 00:00:00 [splunkd pid=43027] splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd [process-runner]
[root@stinky local]# lsof -p 43027 | egrep /var/log/audit
splunkd 43027 splunk 89r REG 252,1 7014280 12924618 /var/log/audit/audit.log
[root@stinky local]# ls -l /var/log/audit/audit.log
-rw-------. 1 root root 7024453 Nov 12 16:50 /var/log/audit/audit.log
Observe Splunk is running as user splunk, but it has a file open (/var/log/audit/audit.log) that is only readable by root. Witchcraft? Nah, CAP_DAC_READ_SEARCH. We can see that the systemd unit file for SplunkForwarder enables CAP_DAC_READ_SEARCH as an AmbientCapability, so that when the process starts it is blessed with this ability.
[root@stinky local]# systemctl cat SplunkForwarder
# /etc/systemd/system/SplunkForwarder.service
#This unit file replaces the traditional start-up script for systemd
#configurations, and is used when enabling boot-start for Splunk on
#systemd-based Linux distributions.
[Unit]
Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
Restart=always
ExecStart=/opt/splunkforwarder/bin/splunk _internal_launch_under_systemd
KillMode=mixed
KillSignal=SIGINT
TimeoutStopSec=360
LimitNOFILE=65536
LimitRTPRIO=99
SuccessExitStatus=51 52
RestartPreventExitStatus=51
RestartForceExitStatus=52
User=splunk
Group=splunk
NoNewPrivileges=yes
AmbientCapabilities=CAP_DAC_READ_SEARCH
ExecStartPre=-/bin/bash -c "chown -R splunk:splunk /opt/splunkforwarder"
Delegate=true
CPUShares=1024
MemoryLimit=1861214208
PermissionsStartOnly=true
ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/system.slice/%n"
[Install]
WantedBy=multi-user.target
CAP_DAC_READ_SEARCH means that Discretionary Access Control (the normal Linux filesystem permissions model) is bypassed for “read” and “search” operations. From the Linux man pages:
CAP_DAC_OVERRIDE
Bypass file read, write, and execute permission checks. (DAC is an abbreviation of "discretionary access control".)
CAP_DAC_READ_SEARCH
* Bypass file read permission checks and directory read and execute permission checks;
* invoke open_by_handle_at(2);
* use the linkat(2) AT_EMPTY_PATH flag to create a link to a file referred to by a file descriptor.So the UF can read any file – including sensitive ones like /etc/shadow. But, it has no other “root” characteristics or abilities. Its (or its children) cannot change system configuration. While I’ve not tried it yet, I feel like you should be able to use the standard linux audit system to keep an eye on any files the UF should not be accessing.
Since I’m making Jurassic Park references today…
