Everything you need to know about SSL for Splunk in 2022

This blog post provides the definitive answer to all questions regarding SSL usage in the Splunk Enterprise product suite. The blog describes every possible SSL configuration in the Splunk configurations and helpful tips and tricks. With the release of Splunk 9.0 on June 14, 2022, new configuration options were introduced. The configuration that only applies to Splunk versions after 9.0 are prefixed with an according prefix.

A huge thank you goes to Duane Waddle & George Starcher for their .conf 2015 talk
Best Practices for Splunk SSL (TheSSLippery Slope Revisited) that served as a reference book for all things SSL for many years.

Configuring strict SSL in Splunk, including verifying all certs that a Splunk instance sees, is a pain point for many Splunkers. This is due to some quirks in how SSL is configured for Splunk and undocumented particularities that have to be known by the Splunker. This blog post wants to resolve any ambiguities left after reading the Splunk documentation and help anyone configuring SSL successfully.

What is SSL?​​

Secure Socket Layer (SSL) is the defacto standard for encrypting data on the internet. The term SSL is still widely used even though Transport Layer Security (TLS) replaced SSL in 1999 with the release of TLS 1.0. Therefore, SSL and TLS can be used interchangeably for this blog.

SSL is based on Public Key cryptography and uses multiple keys and a Public Key Infrastructure (PKI). There are different Public Key cryptography algorithms with distinct advantages and disadvantages. The process of secure key generation is not explained in this post.

Every system that wants to communicate securely using SSL needs to possess two keys:

• Public Key
• Private Key

Usage of these two keys can solve the problem of Confidentiality. First, a Public-Key Infrastructure (PKI) has to get established for the communicating parties to verify each other. A PKI consists of a Root-CA and one to many intermediate CAs. A CA is nothing different than a Public/Private Key pair that signs other Public/Private Key pairs. Before a CA signs another Public Key, the CA checks if the requestor is legitimate and the system/domain claiming to be.

The whole chain is presented to a communication partner after a CA signs the public key. First, the communication partner can verify if a trusted CA signed the public key and whether the public key is legitimate. If the communication partner trusts the CA, the legitimacy of the communication partner is taken for granted.

Where is SSL used in the Splunk product?​​

Splunk uses the Confidentiality and Integrity provided by a PKI to encrypt communication and verify the communication partners.

Every communication inside a Splunk environment can be secured using SSL. This chapter gives an overview of all communication to, from, and between Splunk servers. The next chapter provides example configurations for each communication type, including configuration snippets. Splunk provides a table-based overview of all communication and the default setting in the Splunk Documentation. As pointed out by the documentation, Splunk only enables SSL on the splunkd API port by default.

Following ports are used in the table and should be secured by a certificate in Splunk:

• 8000 (Splunk Web)
• 8089 (splunkd API)
• 9997 (splunkd splunktcp-ssl)
• 8191 (mongod)
• 9887 (splunkd replication_port-ssl)
• 8088 (splunkd HEC)

Additionally following ports that are not managed by a Splunk component should be secured:

• 443 (LB)
• 636 (LDAPS)

How is SSL configured in the Splunk product?​​

This chapter shows the default configuration when installing a new instance of Splunk Enterprise. Afterward, we offer what is recommended based on our experience managing large-scale Splunk Enterprise installations.

Default SSL Version and Cipher Suites​​

Splunk enforces the usage of TLS1.2 beginning with version 6.6. TLS1.3, released in 2018, is not yet supported. Following default configuration is found in server.conf, web.conf, and every other file that supports SSL.

sslVersions = tls1.2 
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ecdhCurves = prime256v1, secp384r1, secp521r1

This configuration drops support for SSL3.0, TLS1.0 and TLS1.1 and defines a set of cipher suites to be used.

Default splunkd API​​

By default, Splunk enables SSL on port 8089 with a self-signed certificate created at installation.

excerpt of $SPLUNK_HOME/etc/system/default/server.conf 
[sslConfig] enableSplunkdSSL = true 
serverCert = $SPLUNK_HOME/etc/auth/server.pem 
sslPassword = password 
caCertFile = $SPLUNK_HOME/etc/auth/cacert.pem 

excerpt of $SPLUNK_HOME/etc/system/local/server.conf 
[sslConfig] sslPassword =

As we can see, the Splunk installation provides a default CA cert available for all Splunk installations and is located in $SPLUNK_HOME/etc/auth/cacert.pem. During the initial startup, Splunk creates a certificate stored at $SPLUNK_HOME/etc/auth/server.pem using the default CA. The sslPassword is encrypted using the splunk.secret of the instance and saved in
$SPLUNK_HOME/etc/system/local/server.conf. The certificate server.pem contains the whole certificate chain, including cacert.pem.

Default Splunk Web​​

Additionally to the splunkd API port, Splunk prepares some default configurations for Splunk Web. The following configuration for Splunk Web exists but is not yet active:

excerpt of $SPLUNK_HOME/etc/system/default/web.conf 

[settings] 
privKeyPath = $SPLUNK_HOME/etc/auth/splunkweb/privkey.pem 
serverCert = $SPLUNK_HOME/etc/auth/splunkweb/cert.pem

Splunk creates an additional set of Public and Private Keys to be used with Splunk Web. They reside in a different directory, $SPLUNK_HOME/etc/auth/splunkweb, and the two keys are split up into public and private keys. If SSL is enabled using the GUI following additional configuration is written to $SPLUNK_HOME/etc/system/local/web.conf, which enables the preconfigured certificates:

[settings] 
enableSplunkWebSSL = 1

Compared to the certificate used by splunkd, the Splunk Web certificate is stored in plaintext and does not contain the whole chain.

Updating Default Certs​​

Splunk ships with a script that allows renewing the default certs using the Splunk CA from the CLI. To update the certificate used by splunkd API, use the following command:

$SPLUNK_HOME/bin/splunk createssl server-cert 4096 -d etc/auth -n server -c <hostname>

If you changed the location of the default certificate from $SPLUNK_HOME/etc/auth/server.pem, you can copy over the created server.pem. You can adapt the -n parameter if your custom location is in $SPLUNK_HOME/etc/auth. If you want to use a different password for the certificate, you can use the parameter -p for additional options. Do not forget to update the sslPassword parameter in $SPLUNK_HOME/etc/system/local/server.conf.

To update the certificates used by Splunk Web, you can use the following command:

$SPLUNK_HOME/bin/splunk createssl web-cert 4096 -n <hostname>

This command will generate new certificates in $SPLUNK_HOME/etc/auth/splunkweb.

SSL all the things!​​

General​

Some key points to consider for using certificates with Splunk:

• Always check the requirements of the customer regardingcertificates. Some customers might have the policy that all certificates have tobe signed by their enterprise CA and others only need certificates presented toend-users to be valid.
• Always use encrypted private keys and set restrictive access permissions
• Usually, sslCommonNameToCheck andsslAltNameToCheck do not need to be set as the default mechanism oftrusting a certificate if it is from the same CA already provides goodprotection from 3rd parties.
• Store certificates in a different folder as $SPLUNK_HOME/etc/auth,e.g., as used for this guide, $SPLUNK_HOME/local/certs
• In case of a certificate chain in a single file, ensure the orderLEAF > INTERMEDIATE2 > INTERMEDIATE1 > ROOT
• Keep certificate configuration local to a system and do notdeploy (except for forwarders)
• Keep an inventory of all certificates and, if possible, getalerts when one is about to expire
• Add the certificates used by Splunk to your root CA using.cat /opt/splunk/etc/auth/apps* /opt/splunk/etc/auth/cloudCA.pem >> ca.pem
• Create OpenSSL.cnf files for eachsystem for easy renewal or use a tool like XCA
• Automate automate automate

SSL Version and Cipher Suites​​

The used cipher suites sometimes have to be adapted depending on requirements posed by regulators or standards (e.g.
PCI DSS). The default configuration configures some ciphers marked as “weak”. To enforce the usage of only modern ciphers, add the following configuration everywhere a certificate is configured:

cipherSuite = TLSv1.2:!eNULL:!aNULL
ecdhCurves = secp521r1

Splunk Web​​

1. copy your certificate to the required location
2. configure $SPLUNK_HOME/etc/system/local/web.conf

[settings] 
enableSplunkWebSSL = 1
sslPassword = <password> 
privKeyPath = local/certs/splunkweb/<cert>.key 
serverCert = local/certs/splunkweb/<cert>.pem

3. restart Splunk
4. validate the certificate used by Splunk Web and make sure the entire certificatechain is presented

openssl s_client -connect localhost:8000 | openssl x509 -noout -text

Splunkd​

1. copy your certificate to the required location
2. configure $SPLUNK_HOME/etc/system/local/server.conf

[sslConfig]
sslPassword = <password>
serverCert = $SPLUNK_HOME/local/certs/splunkd/<server>.pem
sslRootCAPath = $SPLUNK_HOME/local/certs/ca/ca.pem
requireClientCert = true # Only clients with certificate signed by the configured root CA are allowed to connect
sslVerifyServerCert = true # Only connects to servers which use a certificate signed by the configured root CA
# Splunk 9+ | sslVerifyServerName = true
# Splunk 9+ | cliVerifyServerName = true
# Optional | sslCommonNameToCheck = <list all systems this server is connecting to (LM, CM, SH, etc.) and the server itself> 
# Optional | sslAltNameToCheck = <list all altnames this server is connecting to (LM, CM, SH, etc.) and the server itself>

# Splunk 9+ | [pythonSslClientConfig]
# Splunk 9+ | sslVerifyServerCert = true
# Splunk 9+ | sslVerifyServerName = true

3. Add the following line to $SPLUNK_HOME/etc/splunk-launch.conf

PYTHONHTTPSVERIFY = 1

4. restart Splunk
5. validate the certificate used by splunkd and make sure the entire certificatechain is presented

openssl s_client -connect localhost:8089 | openssl x509 -noout -text

6. If requireClientCert is enabled, the certs for Splunk Web have tobe configured as well, see Splunk Web on howto do this
7. Check for errors in splunkd.log and web_service.log

Deployment Server​

1. make sure splunkd uses SSL on the DS
2. make sure the forwarder has a generic forwarder certificate from the same CA asthe certificate on the DS as well as the CA certificate configured (e.g., in example_forwarder_ssl)
3. configure the deploymentclient.conf on the forwarder using an app(e.g., in example_forwarder_deploymentclient)
4. restart the forwarder
5. ensure the app containing the certificates and deploymentclient.conf isoverwritten by the DS

Forwarding – classic​​

1. make sure the forwarder has a generic forwarder certificate from the same CA asthe certificate on the indexers and intermediate forwarders as well as the CAcertificate configured (e.g., in example_forwarder_ssl)
2. ensure the indexers and intermediate forwarders have a cert present locally andinstalled at Splunkd with a Subject Alternative Name identical for all indexersand intermediate forwarders or uses an identical subdomain (e.g.,indexer.splunk.datapunctum.ch)
3. configure outputs.conf on the forwarder using an app (e.g., in example_forwarder_outputs)
4. configure inputs.conf on the indexers and intermediate forwarders alocal app (e.g., in example_indexer_inputs)

Forwarding – indexer discovery​​

1. make sure the forwarder has a generic forwarder certificate from the same CA asthe certificate on the indexers and intermediate forwarders as well as the CAcertificate configured (e.g. in
   example_forwarder_ssl)
2. ensure the indexers and intermediate forwarders have a cert present locally andinstalled at Splunkd with a Subject Alternative Name identical for all indexersand intermediate forwarders or uses an identical subdomain (e.g.indexer.splunk.datapunctum.ch).
3. configure outputs.conf on the forwarder using an app (e.g. in example_forwarder_outputs_idxc)
4. check that the common name presented by the CM is given in theserver.conf of the forwarder if sslCommonNameToCheck is set
5. configure inputs.conf on the indexers and intermediate forwardersusing a local app (e.g. in example_indexer_inputs)

Indexer Replication​

1. make sure the indexers have a cert present locally and installed at Splunkd
2. configure the following in server.conf of the clustering app

[replication_port-ssl:9887]
disabled = false
sslPassword = <password>
serverCert = $SPLUNK_HOME/local/certs/splunkd/<server>.pem
requireClientCert = true compress = true

3. add the SAN that is identical for all indexers to sslAltNameToCheckor all common names to sslCommonNameToCheck in$SPLUNK_HOME/etc/system/local/server.conf if sslVerifyServerCert is set and either of the parameters are used
4. restart Splunk
5. look for errors in $SPLUNK_HOME/var/log/splunk/splunkd.log
6. check cluster status on the cluster manager runningsplunk show cluster-status --verbose

KV Store​​

Distinct configuration is required for the KVStore starting with Splunk 9.0.0. This configuration has to be added to server.conf. The certificate configured for the KVStore
has to be encrypted. Therefore, if the certificate for Splunkd is not encrypted, an additional, encrypted certificate is required.

1. Create a second, encrypted .pem out of an existing.pem containing the certificate chain and the private key, run thefollowing code snippet:

export CERT_NAME=<pem name without ending>
openssl rsa -in $CERT_NAME.pem -out $CERT_NAME.enc.key -aes256
openssl x509 -in $CERT_NAME.pem > $CERT_NAME.enc.pem
cat $CERT_NAME.enc.key >> $CERT_NAME.enc.pem

2. Add following stanza to $SPLUNK_HOME/etc/system/local/server.confafter creating an encrypted .pem certificate:

[kvstore]
sslVerifyServerCert = true
sslVerifyServerName = true
sslPassword = <password>
serverCert = $SPLUNK_HOME/local/certs/kvstore/<server>.enc.pem

3. Restart Splunk

Http Event Collector (HEC)​

HEC uses the same certificate as configured for Splunkd. Therefore no additional configuration is required if the splunkd cert can be used for HEC.

Template Apps​​

example_forwarder_ssl​

The initial connection from the forwarder to its deployment server must be secured using SSL. Therefore an app containing the used CA and a generic forwarder cert is rolled out at the installation of the forwarder. The DS should deploy this app and overwrite the local version with the first download of forwarder apps. The app contains the following files:

• ssl/ca.pem
• ssl/forwarder.pem
• local/server.conf

[sslConfig]
sslPassword = <password>
serverCert = $SPLUNK_HOME/etc/apps/<example_forwarder_ssl>/ssl/forwarder.pem
sslRootCAPath = $SPLUNK_HOME/etc/apps/<example_forwarder_ssl>/ssl/ca.pem

• local/app.conf

# 
# example_forwarder_ssl 
# Deployed by: ds.splunk.datapunctum.ch 
# 
[install] 
state = enabled

[package] 
check_for_updates = false 

[ui] 
is_visible = false 
is_manageable = false

• metadata/local.meta

[] 
access = read : [ * ], write : [ admin ] 
export = system

example_forwarder_deploymentclient​​

The app is used to define the initial target for the deployment client. The DS should deploy this app and overwrite the local version with the first download of forwarder apps. The app contains the following files:

• local/deploymentclient.conf

[deployment-client]
sslVerifyServerCert = true
# Optional | sslCommonNameToCheck = <common name of DS> 
# Optional - in case of DNS based DS load balancing | sslAltNameToCheck = <san of DS>

[target-broker:deploymentServer]
targetUri = <IP>:8089

• local/app.conf

# 
# example_forwarders_deploymentclient 
# Deployed by: ds.splunk.example.com 
# 

[install] 
state = enabled 

[package] 
check_for_updates = false 

[ui] 
is_visible = false 
is_manageable = false

• metadata/local.meta

[] 
access = read : [ * ], write : [ admin ] 
export = system

example_forwarder_outputs​​

• local/outputs.conf

[tcpout] 
defaultGroup = primary_indexers 
useACK = true 
forceTimebasedAutoLB = true 
forwardedindex.2.whitelist = (_audit|_introspection|_internal) 

[tcpout:primary_indexers] 
server = <server>:<port>, <server>:<port> 
compressed = true 
sslPassword = <password> 
clientCert = $SPLUNK_HOME/etc/apps/<example_forwarder_ssl>/ssl/fowarder.pem 
sslRootCAPath = $SPLUNK_HOME/etc/apps/<example_forwarder_ssl>/ssl/ca.pem 
sslVerifyServerCert = true 
# Optional | sslAltNameToCheck = <SAN of all indexers>

• local/app.conf

# 
# example_forwarder_outputs 
# Deployed by: ds.splunk.example.com
# 

[install] 
state = enabled 

[package] 
check_for_updates = false 

[ui] 
is_visible = false 
is_manageable = false

• metadata/local.meta

[] 
access = read : [ * ], write : [ admin ] 
export = system

example_forwarder_outputs_idxc​

• local/outputs.conf

[indexer_discovery:discovery] 
pass4SymmKey = <pass4SymmKey> 
master_uri = <CM API URI> 

[tcpout] 
defaultGroup = idxc_indexers 
useACK = true 
forceTimebasedAutoLB = true 
forwardedindex.2.whitelist = (_audit|_introspection|_internal) 

[tcpout:idxc_indexers] 
indexerDiscovery = discovery 
compressed = true 
sslPassword = <password> 
clientCert = $SPLUNK_HOME/etc/apps/<example_forwarder_ssl>/ssl/fowarder.pem 
sslRootCAPath = $SPLUNK_HOME/etc/apps/<example_forwarder_ssl>/ssl/ca.pem 
sslVerifyServerCert = true 
# Optional | sslAltNameToCheck = <SAN of all indexers>

• local/app.conf

# 
# example_forwarder_outputs 
# Deployed by: ds.splunk.example.com 
# 

[install] 
state = enabled 

[package] 
check_for_updates = false 

[ui] 
is_visible = false 
is_manageable = false

• metadata/local.meta

[] 
access = read : [ * ], write : [ admin ] 
export = system

example_indexer_inputs​

• local/inputs.conf

[splunktcp-ssl://9996] 
compressed = true 

[SSL] 
serverCert = $SPLUNK_HOME/local/certs/splunkd/<server>.pem
requireClientCert = true 
# Optional | sslCommonNameToCheck = <generic forwarder cert>

• local/app.conf
• local/app.conf

# 
# example_forwarder_outputs 
# Managed by: cm.splunk.example.com 
# 

[install] 
state = enabled 

[package] 
check_for_updates = false 

[ui] 
is_visible = false 
is_manageable = false

• metadata/local.meta

[] 
access = read : [ * ], write : [ admin ] 
export = system

Troubleshooting​

• It is currently not possible to userequireClientCert in server.conf together with anencrypted private key in web.conf
• Set password to „password“ if no password is set for the private key
• Some parts of Splunk Web expect privKeyPath andserverCert to be a relative path starting in/opt/splunk
• In the case of older Splunk versions, set the following additional settings:

sslVersions = tls, -ssl* 
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 
ecdhCurves = prime256v1, secp384r1, secp521r1

• grep -E "ERROR\s(SSLCommon|HTTPServer)" $SPLUNK_HOME/var/log/splunk/splunkd.log
  • errno=185073780 → ensure that serverCert contains the whole certificate chain, the certs are in the correct order and the certificates are not corrupt
  • errno=101077092 → ensure that sslPassword is set correctly
  • errno=151404653 → corrupt value in sslPassword
  • errno=33558530 → check the configured paths for the certificates are correct
  • errno=218570875 → the key is corrupt
• Splunk Web in combination with a certificate signed by a CA not trusted by Usersbrowsers, leads to many log lines in splunkd.log regardingsslv3 alert certificate unknown
• In the case of enabled Splunk Instrumentation and configuredrequireClientCert, various errors are visible in splunkd.log as theinstrumentation fails to use certificates while connecting to the API
• In the case of enabled splunk_secure_gateway and configuredrequireClientCert, various errors are visible in splunkd.log as theapp fails to use certificates while connecting to the API
• In case of enabled requireClientCert errors are found insplunkd.log from dmc_config.py as the script fails to usecertificates while connecting to the API

How do I get SSL certificates?​​

Depending on the environment all certificates are provided and signed by an enterprise CA.

If no certificates are provided a Splunk internal PKI has to be established. Host based configuration files are used to ensure consistent and repeatable certificates.

Create a custom CA authority​​

Your CA has to be stored at a safe location. To prevent unauthorized parties from signing certificates the private key is stored in an encrypted format. First, create a configuration file for your CA called
ca.cnf based on following template:

[req]
default_bits = 2048 
distinguished_name = req_distinguished_name 
prompt = no 
x509_extensions = v3_ca 

[req_distinguished_name] 
C = CH 
ST = Zurich 
L = Zurich 
O = Datapunctum AG 
OU = Splunk 
CN = CA 

[v3_ca] 
subjectKeyIdentifier=hash 
authorityKeyIdentifier=keyid:always,issuer 
basicConstraints = critical, CA:TRUE, pathlen:3 
keyUsage = critical, cRLSign, keyCertSign

After creating the configuration file, a Private/Public Keypair is generated using the following commands

export CA_FILE_NAME=<ca_file_name>
openssl genrsa -aes256 -out $CA_FILE_NAME.key 2048 
openssl req -new -x509 -extensions v3_ca -days 3650 -key $CA_FILE_NAME.key -sha256 -out $CA_FILE_NAME.pem -config $CA_FILE_NAME.cnf

Finally run the following command to verify the creation of the CA:

openssl x509 -in $CA_FILE_NAME.pem -noout- text

Create a Public/Private key pair​​

All certificates used in Splunk are signed by the CA just created. As for the CA, a configuration file is used. Usually, the certificates are have the hostname as the common name. Any additional names must be added as a SAN. Due to some quirks in how Splunk handles certificates, the hostname is again given as SAN.

[req]
default_bits = 2048 
distinguished_name = req_distinguished_name 
prompt = no 
req_extensions = v3_req 

[req_distinguished_name] 
C = CH 
ST = Zurich 
L = Zurich 
O = Datapunctum AG 
OU = Splunk 
CN = <hostname> 

[v3_req] 
basicConstraints = CA:FALSE 
keyUsage = nonRepudiation, digitalSignature, keyEncipherment 
subjectAltName = @alt_names 

[alt_names] 
DNS.1 = <hostname> 
DNS.2 = <fqdn> 
DNS.3 = <san> 
IP = <ip>

After creating the configuration file, a Private/Public Keypair is generated using the following commands:

export CERT_FILE_NAME=idx1
openssl genrsa -aes256 -out $CERT_FILE_NAME.key 2048 
openssl req -extensions v3_req -sha256 -new -key $CERT_FILE_NAME.key -out $CERT_FILE_NAME.csr -config $CERT_FILE_NAME.cnf 
openssl x509 -req -extensions v3_req -days 3650 -sha256 -in $CERT_FILE_NAME.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out $CERT_FILE_NAME.pem -extfile $CERT_FILE_NAME.cnf
cp $CERT_FILE_NAME.pem $CERT_FILE_NAME.web.pem
openssl rsa -in $CERT_FILE_NAME.key -out $CERT_FILE_NAME.web.key
cat $CERT_FILE_NAME.key >> $CERT_FILE_NAME.pem

Finally run the following command to verify the creation of the CA

openssl x509 -in $CERT_FILE_NAME.pem -noout- text

The create .pem file can now be referenced from your Splunk configuration. For Splunk Web, just use the .pem file for serverCert and the .key file for
privKeyPath.

Use XCA​​

X -Certificate and Key management (XCA in short) or get it from
GitHub.

• Create a private key for your CA certificate
• Create a certificate based on the create CA private key
• Create a Template with all required information as country, organisation and soon
• Create a private key for each server that needs a certificate
• Create a certificate based on the created template and private key
• Export the CA certificate and server keypair to each server

XCA is easy to use but violates the best practice that a private key never leaves the server where it is used. So again, check the requirements of the customer before using XCA as a central tool for managing certificates.