Back in June, along with the release of Splunk 9.0, Splunk dropped several security advisories. I’m spending a little time digging in on SVD-2022-0607. Come along with me as we learn together.
The first thing of interest to me about this one is … we’ve been here before. Go back to https://www.duanewaddle.com/splunk-pass4symmkey-for-deployment-client-deployment-server/ and read the update from Martin:
Discussing this feature with Martin Müller, there are some limitations here. The
pass4SymmKeyonly protects the DS “control channel” API endpoint. There are other API endpoints, like the app download endpoint, that will not require
pass4SymmKey. What this means is that an attacker who knows the names of your serverclasses and apps will still be able to download those apps from your DS without authentication. See Splunk Ideas https://ideas.splunk.com/ideas/EID-I-391 which discusses this.
The good news I guess is that this is fixed? Now we’ll need to review the docs suggested by the SVD in terms of how to implement the fixes. Also go back and read my old blog post as a refresher – there’s some useful stuff in there. At first glance, I don’t think the docs are accurate / sufficient – I’ll have to do some docs feedback on that.
In my test environment I have a DS and 2 UFs. The DS (ds) is running Splunk 9.0.2. One UF (uf9) running Splunk 9.0.2, the other (uf8) is running Splunk 8.2.9. To make understanding what is going on a little easier, I have disabled TLS on the DS REST API as follows in
[sslConfig] enableSplunkdSSL = false
Now I can use
tcpdump to help understand the protocol between the UFs and DS, something tells me this will be important. On the DS I will enable the exact settings from the docs page in
[broker:broker] authKeyStanza=deployment requireAuthentication = true [streams:deployment] authKeyStanza=deployment requireAuthentication = true
On the UFs I won’t do anything special but set them up to use the DS in
deploymentclient.conf, but in HTTP mode:
[target-broker:deploymentServer] targetUri= http://ds:8089
Now we will set up a trivial app (testapp) and
serverclasses.conf for it, and watch what happens on both uf8 and uf9. First from splunkd.log on uf9:
11-16-2022 03:38:43.840 +0000 INFO HttpPubSubConnection [185 HttpClientPollingThread_29B28178-CD4D-4534-BE89-3D7C5D003BE5] - Running phone uri=/services/broker/phonehome/connection_10.89.0.6_8089_uf9.dns.podman_888d73cf5fde_29B28178-CD4D-4534-BE89-3D7C5D003BE5 11-16-2022 03:38:43.845 +0000 INFO DeployedApplication [185 HttpClientPollingThread_29B28178-CD4D-4534-BE89-3D7C5D003BE5] - Checksum mismatch 0 <> 15970752176927723036 for app=testapp. Will reload from='http://ds:8089/services/streams/deployment?name=default:test:testapp' 11-16-2022 03:38:43.849 +0000 INFO DeployedApplication [185 HttpClientPollingThread_29B28178-CD4D-4534-BE89-3D7C5D003BE5] - Downloaded url=ds:8089/services/streams/deployment?name=default:test:testapp to file='/opt/splunkforwarder/var/run/test/testapp-1668569894.bundle' sizeKB=10 11-16-2022 03:38:43.849 +0000 INFO DeployedApplication [185 HttpClientPollingThread_29B28178-CD4D-4534-BE89-3D7C5D003BE5] - Installing app=testapp to='/opt/splunkforwarder/etc/apps/testapp' 11-16-2022 03:38:43.859 +0000 INFO ApplicationManager [185 HttpClientPollingThread_29B28178-CD4D-4534-BE89-3D7C5D003BE5] - Detected app creation: testapp 11-16-2022 03:38:43.868 +0000 WARN DC:DeploymentClient [185 HttpClientPollingThread_29B28178-CD4D-4534-BE89-3D7C5D003BE5] - Restarting Splunkd...
Now in the
splunkd_access.log on the ds:
10.89.0.6 - splunk-system-user [16/Nov/2022:03:37:43.837 +0000] "POST /services/broker/phonehome/connection_10.89.0.6_8089_uf9.dns.podman_888d73cf5fde_29B28178-CD4D-4534-BE89-3D7C5D003BE5 HTTP/1.1" 200 407 "-" "Splunk/9.0.2 (Linux 5.15.0-52-generic; arch=x86_64)" - - - 1ms 10.89.0.6 - splunk-system-user [16/Nov/2022:03:38:43.842 +0000] "POST /services/broker/phonehome/connection_10.89.0.6_8089_uf9.dns.podman_888d73cf5fde_29B28178-CD4D-4534-BE89-3D7C5D003BE5 HTTP/1.1" 200 471 "-" "Splunk/9.0.2 (Linux 5.15.0-52-generic; arch=x86_64)" - - - 2ms 10.89.0.6 - splunk-system-user [16/Nov/2022:03:38:43.848 +0000] "POST /services/streams/deployment?name=default:test:testapp HTTP/1.1" 200 265 "-" "Splunk/9.0.2 (Linux 5.15.0-52-generic; arch=x86_64)" - - - 1ms
This is pretty reasonable. We see the “phone home” event on both the UF and the DS, and the “download” from the DS and the install to the UF. The app is out there and all is well. But on uf8 running Splunk 8.2.9, things are not looking so great:
11-16-2022 03:51:17.044 +0000 INFO HttpPubSubConnection [139 HttpClientPollingThread_EDC9AFAC-E27B-4D71-B824-9C4D1CDD2AD0] - Running phone uri=/services/broker/phonehome/connection_10.89.0.5_8089_uf8.dns.podman_6f42c58a52ce_EDC9AFAC-E27B-4D71-B824-9C4D1CDD2AD0 11-16-2022 03:51:17.049 +0000 INFO DeployedApplication [139 HttpClientPollingThread_EDC9AFAC-E27B-4D71-B824-9C4D1CDD2AD0] - Checksum mismatch 0 <> 15970752176927723036 for app=testapp. Will reload from='http://ds:8089/services/streams/deployment?name=default:test:testapp' 11-16-2022 03:51:17.051 +0000 WARN HTTPClient [139 HttpClientPollingThread_EDC9AFAC-E27B-4D71-B824-9C4D1CDD2AD0] - Download of file /opt/splunkforwarder/var/run/test/85d323c966bd0bee failed with status 401 11-16-2022 03:51:17.051 +0000 WARN DeployedApplication [139 HttpClientPollingThread_EDC9AFAC-E27B-4D71-B824-9C4D1CDD2AD0] - Problem downloading from uri=ds:8089 to path='/services/streams/deployment?name=default:test:testapp' 11-16-2022 03:51:17.051 +0000 ERROR DeployedServerclass [139 HttpClientPollingThread_EDC9AFAC-E27B-4D71-B824-9C4D1CDD2AD0] - name=test Failed to download app=testapp
Oof. HTTP 401 errors coming back from attempts to download the app. But, this makes sense! We enabled authentication on the DS side for
streams:deployment above, and Splunk has already told us that Splunk < 9.0.0 as a deployment client does not send pass4SymmKey authenticated requests for that endpoint. So this is not really a surprise – we are just setting the stage. Let’s see what’s in the pcap! From uf8 (Splunk 8):
POST /services/streams/deployment?name=default:test:testapp HTTP/1.1 User-Agent: Splunk/8.2.9 (Linux 5.15.0-52-generic; arch=x86_64) TE: trailers, chunked Host: ds:8089 Accept-Encoding: gzip Content-Length: 0
From uf9 (Splunk 9):
POST /services/streams/deployment?name=default:test:testapp HTTP/1.1 User-Agent: Splunk/9.0.2 (Linux 5.15.0-52-generic; arch=x86_64) TE: trailers, chunked Host: ds:8089 Accept-Encoding: gzip Content-Length: 0 x-splunk-lm-nonce: 1f6cc1fd7ef379df64d6709823d383d9 x-splunk-lm-timestamp: 1668569923 x-splunk-lm-signature: LIKvYvPhmg9vkgSh/qMyuX0XxrhEeT3Sc5cr/o3Fqvlqp6oDUFaesPwiOXM= x-splunk-digest: v2,VgAXgTboCvGPJh8qIdg+MhDy18W1UkAMPPb3Uh75JWwIiu0j/zkub8adNs7t/E8JZdzozwHGGwRdOoCRgMWl0Q==
There’s some new request headers here. The
pass4SymmKey is not DIRECTLY in the HTTP headers, but there’s definitely some new stuff here. With enough time, we might be able to figure out exactly what these headers are trying to say or how to validate them. But, the key thing here is that the key code change in Splunk 9.0.0 for SVD-2022-0607 was to add the
pass4SymmKey authentication headers to this API call. Once that was done and your environment has rolled out Splunk >= 9.0.0 to every DS client, then the Splunk Administrator can enable the DS to require the headers via
restmap.conf. To keep this story going, I’m going to upgrade uf8 to Splunk 9.0.2.
Mysterious Missing pass4SymmKey
If we look back at our DS, we had configured
restmap.conf to look for
pass4SymmKey in the
deployment stanza of
btool command will confirm this, but it will also confirm that no
pass4SymmKey is set there.
[root@33e28c67e142 local]# /opt/splunk/bin/splunk btool --debug restmap list broker:broker | egrep authKeyStanza /opt/splunk/etc/system/local/restmap.conf authKeyStanza = deployment [root@33e28c67e142 local]# /opt/splunk/bin/splunk btool --debug server list deployment /opt/splunk/etc/system/local/server.conf [deployment]
So what pass4SymmKey is it using? Thankfully the README/restmap.conf.spec gives us a hint for the DS-side of the connection.
pass4SymmKeyis available, authentication is done using the
We could test this, but I’m going to take it at faith and do what seems to be the “right thing.” I’ll add a
pass4SymmKey under the
[deployment] stanza in
[deployment] # myfirstpass4symmkey pass4SymmKey = $7$TZxP0ci6l/e88I7IcRGJuhJR/N1eLgNuUlv3wQqbi29ueVps/9D8SSuSLOMTka8ZAkks
After we do this and restart, both of our UFs are now failing to connect because they don’t have the right
pass4SymmKey. There are HTTP 401 responses to their attempts to connect in the
splunkd_access.log. I can add that
pass4SymmKey to each of them using a
[deployment] stanza. Once I do that they’re both able to connect.
Thinking about this some, this means that if I do nothing more than follow the procedure in the Splunk Docs above, my environment may be configured in a way that all of the UFs are authenticated to the DS using the default pass4SymmKey of “
changeme“. The docs do suggest that setting a
pass4SymmKey is a necessary prerequisite to this, but it turns out a default one for the
[general] stanza is already set. A Splunk Admin has to be careful when implementing this in order to not present a thin veneer of “security”.
Also, this complicates UF deployment a bit. If I do configure a robust, non-default
pass4SymmKey on the DS then my UFs need to know it before they can ever connect to the DS. Because the DS is usually how UFs are configured, this is a circular dependency. This means that my UFs need to be pre-seeded with the correct
pass4SymmKey. This is extra work for the admin as part of deployment. For example, on Windows, the MSI does not include “deployment
pass4SymmKey” as one of the supported arguments for the MSI. For Linux, the additional effort is probably not too large because Linux usually requires extra effort to configure the UF to phone home to a DS anyway. But, this definitely adds a new complexity to be handled as part of the UF deployment.
Changing the pass4SymmKey
So by now, we’ve ideally solved the deployment problems. We’ve installed Splunk 9.0.x UFs everwhere, we’ve pre-seeded them with a robust
pass4SymmKey using the
[deployment] stanza, and we’ve enabled authenticated access to our DS using
restmap.conf. Of course, now that we’re done someone accidentally does a git commit with our
pass4SymmKey in it and pushes it to a public GitHub repo. We’ll need to rotate the
pass4SymmKey. This is a problem. If we first change the
pass4SymmKey at the DS, that will cut all of the UFs off until we visit each one to give it the new value. If we start visiting UFs first, then we’ll cut them off one at a time until all have been visited. Neither of these are great.
We could turn off authentication until all of the UFs have been updated, and then re-enable it. That might work, but if we went to the effort of enabling authentication then this might not make sense. We added authentication for a reason, and just arbitrarily turning it off just to rotate a password seems less than ideal. Surely we didn’t enable authentication just to put a check mark in a compliance mandated security theater spreadsheet?
It turns out that Splunk anticipated this in
restmap.conf, but you only see the hint of it from the spec file. In the
authKeyStanza key, you can list multiple
server.conf stanzas that contain
pass4SymmKey values. So we can do this on the DS:
[root@33e28c67e142 local]# cat restmap.conf [broker:broker] authKeyStanza=deployment, deployment2 requireAuthentication = true [streams:deployment] authKeyStanza=deployment, deployment2 requireAuthentication = true [root@33e28c67e142 local]# /opt/splunk/bin/splunk btool server list deployment [deployment] pass4SymmKey = $7$TZxP0ci6l/e88I7IcRGJuhJR/N1eLgNuUlv3wQqbi29ueVps/9D8SSuSLOMTka8ZAkks [deployment2] pass4SymmKey = $7$+N40XRehCqEaLsWuaVQntroku8ekkskZ7IqG2xSkTt6Ejkal5/rdb5bsBswF0IIgXyPGyHgdKHv2cShtTkwb2X8xZg== [root@33e28c67e142 local]# /opt/splunk/bin/splunk cmd splunkd show-decrypted --value '$7$TZxP0ci6l/e88I7IcRGJuhJR/N1eLgNuUlv3wQqbi29ueVps/9D8SSuSLOMTka8ZAkks' myfirstpass4symmkey [root@33e28c67e142 local]# /opt/splunk/bin/splunk cmd splunkd show-decrypted --value '$7$+N40XRehCqEaLsWuaVQntroku8ekkskZ7IqG2xSkTt6Ejkal5/rdb5bsBswF0IIgXyPGyHgdKHv2cShtTkwb2X8xZg==' mysecondkeybecausethefirstwasleaked
Now, the DS supports either one of these
pass4SymmKey values. We can put either of them on a UF and do a transition from one to the other. Check out uf8 and uf9 using different
[root@888d73cf5fde local]# /opt/splunkforwarder/bin/splunk btool server list deployment [deployment] pass4SymmKey = $7$FhVu9cYZfT0eAZX42cOYtfNJDIV5WIDUsi1dtczbFKhbQud5EiksI/+rMNO+o0pjfPIbQU76XkNgQI2tFAckG1tm4w== [root@888d73cf5fde local]# /opt/splunkforwarder/bin/splunk cmd splunk show-decrypted --value '$7$FhVu9cYZfT0eAZX42cOYtfNJDIV5WIDUsi1dtczbFKhbQud5EiksI/+rMNO+o0pjfPIbQU76XkNgQI2tFAckG1tm4w==' mysecondkeybecausethefirstwasleaked [root@6f42c58a52ce testapp]# /opt/splunkforwarder/bin/splunk btool server list deployment [deployment] pass4SymmKey = $7$RLJg9Y2qsPjYkHZmXdiZ9GYHXBDPU2xfutAy97GTozXsCOvYpOtIz+Nr7JekYu4iTQSw [root@6f42c58a52ce testapp]# /opt/splunkforwarder/bin/splunk cmd splunkd show-decrypted --value '$7$RLJg9Y2qsPjYkHZmXdiZ9GYHXBDPU2xfutAy97GTozXsCOvYpOtIz+Nr7JekYu4iTQSw' myfirstpass4symmkey
And if we look on the DS, both are getting HTTP 200 responses in splunkd_access.log for their DS phonehomes:
10.89.0.5 - splunk-system-user [17/Nov/2022:03:08:56.344 +0000] "POST /services/broker/phonehome/connection_10.89.0.5_8089_uf8.dns.podman_6f42c58a52ce_EDC9AFAC-E27B-4D71-B824-9C4D1CDD2AD0 HTTP/1.1" 200 24 "-" "Splunk/9.0.2 (Linux 5.15.0-52-generic; arch=x86_64)" - - - 1ms 10.89.0.6 - splunk-system-user [17/Nov/2022:03:09:24.132 +0000] "POST /services/broker/phonehome/connection_10.89.0.6_8089_uf9.dns.podman_888d73cf5fde_29B28178-CD4D-4534-BE89-3D7C5D003BE5 HTTP/1.1" 200 471 "-" "Splunk/9.0.2 (Linux 5.15.0-52-generic; arch=x86_64)" - - - 1ms
So we did a bit of a deep dive into how UFs (and other deployment clients) talk to DSes. We enabled DS authentication of the UFs and saw how code changes in Splunk 9.0.0 made it possible to enable
pass4SymmKey authentication for app downloads from the DS. We even learned how to do
pass4SymmKey secret rotation without shooting ourselves in the foot.
One important thing to (again) note. This is not mutual authentication. Having the DS authenticate clients using
pass4SymmKey does not guarantee that clients are talking to a legitimate DS. Without correct TLS certificates and configurations, an attacker can still stand up a fake DS that does not require client
pass4SymmKey authentication. In this scenario, clients will download and execute arbitrary apps from the attacker DS. Always perform a realistic assessment of your threat model and take the right steps to reduce risk in your environment.