Everything You Need to Know About SSL for Splunk in 2022

This blog post provides the definitive answer to all questions regarding SSL usage in the Splunk Enterprise product suite. The blog describes every possible SSL configuration in the Splunk configurations and helpful tips and tricks. With the release of Splunk 9.0 on June 14, 2022, new configuration options were introduced. The configuration that only applies to Splunk versions after 9.0 are prefixed with an according prefix.

A huge thank you goes to Duane Waddle & George Starcher for their .conf 2015 talk Best Practices for Splunk SSL (TheSSLippery Slope Revisited) that served as a reference book for all things SSL for many years.

Configuring strict SSL in Splunk, including verifying all certs that a Splunk instance sees, is a pain point for many Splunkers. This is due to some quirks in how SSL is configured for Splunk and undocumented particularities that have to be known by the Splunker. This blog post wants to resolve any ambiguities left after reading the Splunk documentation and help anyone configuring SSL successfully.

What is SSL?

Secure Socket Layer (SSL) is the defacto standard for encrypting data on the internet. The term SSL is still widely used even though Transport Layer Security (TLS) replaced SSL in 1999 with the release of TLS 1.0. Therefore, SSL and TLS can be used interchangeably for this blog.

SSL is based on Public Key cryptography and uses multiple keys and a Public Key Infrastructure (PKI). There are different Public Key cryptography algorithms with distinct advantages and disadvantages. The process of secure key generation is not explained in this post.

Every system that wants to communicate securely using SSL needs to possess two keys:

• Public Key
• Private Key

Usage of these two keys can solve the problem of Confidentiality. First, a Public-Key Infrastructure (PKI) has to get established for the communicating parties to verify each other. A PKI consists of a Root-CA and one to many intermediate CAs. A CA is nothing different than a Public/Private Key pair that signs other Public/Private Key pairs. Before a CA signs another Public Key, the CA checks if the requestor is legitimate and the system/domain claiming to be.

The whole chain is presented to a communication partner after a CA signs the public key. First, the communication partner can verify if a trusted CA signed the public key and whether the public key is legitimate. If the communication partner trusts the CA, the legitimacy of the communication partner is taken for granted.

Where is SSL Used in the Splunk Product?

Splunk uses the Confidentiality and Integrity provided by a PKI to encrypt communication and verify the communication partners.

Every communication inside a Splunk environment can be secured using SSL. This chapter gives an overview of all communication to, from, and between Splunk servers. The next chapter provides example configurations for each communication type, including configuration snippets. Splunk provides a table-based overview of all communication and the default setting in the Splunk Documentation. As pointed out by the documentation, Splunk only enables SSL on the splunkd API port by default.

Name	Component A	Component B	Default Port
User to Splunk Web	Browser	Splunk Web	8000
Splunk Web to splunkd	Splunk Web	any splunkd	8089
Forwarding	splunkd on UF/HF	splunkd on IDX/HF	9997
Deployment Server	any splunkd	splunkd on DS	8089
Distributed Search	splunkd on SH	splunkd on IDX	8089
KV Store	splunkd on SH	mongod on SH	8191
SH Cluster API	splunkd on SH	splunkd on SH	8089
SH Cluster KV Store	mongod on SH	mongod on SH	8191
SH Cluster Deployment	splunkd on DLY	splunkd on SH	8089
SH Cluster LB	LB	Splunk Web	443
Indexer Cluster Replication	splunkd on IDX	splunkd on IDX	9887
Indexer Cluster Manager	splunkd on SH/IDX	splunkd on IDXCM	8089
Licensing	any splunkd	splunkd on LM	8089
HEC	any	splunkd on HF/IDX	8088
LDAP	splunkd on SH	LDAPS	636

Following ports are used in the table and should be secured by a certificate in Splunk:

• 8000 (Splunk Web)
• 8089 (splunkd API)
• 9997 (splunkd splunktcp-ssl)
• 8191 (mongod)
• 9887 (splunkd replication_port-ssl)
• 8088 (splunkd HEC)

Additionally following ports that are not managed by a Splunk component should be secured:

• 443 (LB)
• 636 (LDAPS)

How is SSL Configured in the Splunk Product?

This chapter shows the default configuration when installing a new instance of Splunk Enterprise. Afterward, we offer what is recommended based on our experience managing large-scale Splunk Enterprise installations.

Default SSL Version and Cipher Suites

Splunk enforces the usage of TLS1.2 beginning with version 6.6. TLS1.3, released in 2018, is not yet supported. Following default configuration is found in server.conf, web.conf, and every other file that supports SSL.

sslVersions = tls1.2
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ecdhCurves = prime256v1, secp384r1, secp521r1

This configuration drops support for SSL3.0, TLS1.0 and TLS1.1 and defines a set of cipher suites to be used.

Default splunkd API

By default, Splunk enables SSL on port 8089 with a self-signed certificate created at installation.

Excerpt of $SPLUNK_HOME/etc/system/default/server.conf:

[sslConfig]
enableSplunkdSSL = true
serverCert = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = password
caCertFile = $SPLUNK_HOME/etc/auth/cacert.pem

Excerpt of $SPLUNK_HOME/etc/system/local/server.conf:

[sslConfig]
sslPassword = <encrypted>

As we can see, the Splunk installation provides a default CA cert available for all Splunk installations and is located in $SPLUNK_HOME/etc/auth/cacert.pem. During the initial startup, Splunk creates a certificate stored at $SPLUNK_HOME/etc/auth/server.pem using the default CA. The sslPassword is encrypted using the splunk.secret of the instance and saved in $SPLUNK_HOME/etc/system/local/server.conf. The certificate server.pem contains the whole certificate chain, including cacert.pem.

Default Splunk Web

Additionally to the splunkd API port, Splunk prepares some default configurations for Splunk Web.

Excerpt of $SPLUNK_HOME/etc/system/default/web.conf:

[settings]
privKeyPath = $SPLUNK_HOME/etc/auth/splunkweb/privkey.pem
serverCert = $SPLUNK_HOME/etc/auth/splunkweb/cert.pem

Splunk creates an additional set of Public and Private Keys to be used with Splunk Web. They reside in a different directory, $SPLUNK_HOME/etc/auth/splunkweb, and the two keys are split up into public and private keys. If SSL is enabled using the GUI following additional configuration is written to $SPLUNK_HOME/etc/system/local/web.conf, which enables the preconfigured certificates:

[settings]
enableSplunkWebSSL = 1

Compared to the certificate used by splunkd, the Splunk Web certificate is stored in plaintext and does not contain the whole chain.

Updating Default Certs

Splunk ships with a script that allows renewing the default certs using the Splunk CA from the CLI. To update the certificate used by splunkd API, use the following command:

$SPLUNK_HOME/bin/splunk createssl server-cert 4096 -d etc/auth -n server -c <hostname>

To update the certificates used by Splunk Web, you can use the following command:

$SPLUNK_HOME/bin/splunk createssl web-cert 4096 -n <hostname>

This command will generate new certificates in $SPLUNK_HOME/etc/auth/splunkweb.

SSL All the Things!

General

Some key points to consider for using certificates with Splunk:

• Always check the requirements of the customer regarding certificates. Some customers might have the policy that all certificates have to be signed by their enterprise CA and others only need certificates presented to end-users to be valid.
• Always use encrypted private keys and set restrictive access permissions
• Usually, sslCommonNameToCheck and sslAltNameToCheck do not need to be set as the default mechanism of trusting a certificate if it is from the same CA already provides good protection from 3rd parties.
• Store certificates in a different folder as $SPLUNK_HOME/etc/auth, e.g., as used for this guide, $SPLUNK_HOME/local/certs
• In case of a certificate chain in a single file, ensure the order LEAF > INTERMEDIATE2 > INTERMEDIATE1 > ROOT
• Keep certificate configuration local to a system and do not deploy (except for forwarders)
• Keep an inventory of all certificates and, if possible, get alerts when one is about to expire
• Add the certificates used by Splunk to your root CA using cat /opt/splunk/etc/auth/apps* /opt/splunk/etc/auth/cloudCA.pem >> ca.pem
• Create OpenSSL.cnf files for each system for easy renewal or use a tool like XCA
• Automate automate automate

SSL Version and Cipher Suites

The used cipher suites sometimes have to be adapted depending on requirements posed by regulators or standards (e.g. PCI DSS). The default configuration configures some ciphers marked as "weak". To enforce the usage of only modern ciphers, add the following configuration everywhere a certificate is configured:

cipherSuite = TLSv1.2:!eNULL:!aNULL
ecdhCurves = secp521r1

Splunk Web

1. Copy your certificate to the required location
2. Configure $SPLUNK_HOME/etc/system/local/web.conf:

[settings]
enableSplunkWebSSL = 1
sslPassword = <password>
privKeyPath = local/certs/splunkweb/<cert>.key
serverCert = local/certs/splunkweb/<cert>

3. Restart Splunk
4. Validate the certificate used by Splunk Web and make sure the entire certificate chain is presented:

openssl s_client -connect localhost:8000 | openssl x509 -noout -text

Splunkd

1. Copy your certificate to the required location
2. Configure $SPLUNK_HOME/etc/system/local/server.conf:

[sslConfig]
sslPassword = <password>
serverCert = $SPLUNK_HOME/local/certs/splunkd/<server>.pem
sslRootCAPath = $SPLUNK_HOME/local/certs/ca/ca.pem
requireClientCert = true
sslVerifyServerCert = true
# Splunk 9+ | sslVerifyServerName = true
# Splunk 9+ | cliVerifyServerName = true

3. Add the following line to $SPLUNK_HOME/etc/splunk-launch.conf:

PYTHONHTTPSVERIFY = 1

4. Restart Splunk
5. Validate the certificate used by splunkd and make sure the entire certificate chain is presented:

openssl s_client -connect localhost:8089 | openssl x509 -noout -text

6. If requireClientCert is enabled, the certs for Splunk Web have to be configured as well
7. Check for errors in splunkd.log and web_service.log

Deployment Server

1. Make sure splunkd uses SSL on the DS
2. Make sure the forwarder has a generic forwarder certificate from the same CA as the certificate on the DS as well as the CA certificate configured
3. Configure deploymentclient.conf on the forwarder
4. Restart the forwarder
5. Ensure the app containing the certificates and deploymentclient.conf is overwritten by the DS

Forwarding: Classic

1. Make sure the forwarder has a generic forwarder certificate from the same CA as the certificate on the indexers and intermediate forwarders as well as the CA certificate configured
2. Ensure the indexers and intermediate forwarders have a cert present locally and installed at Splunkd with a Subject Alternative Name identical for all indexers and intermediate forwarders or uses an identical subdomain
3. Configure outputs.conf on the forwarder
4. Configure inputs.conf on the indexers and intermediate forwarders

Forwarding: Indexer Discovery

1. Make sure the forwarder has a generic forwarder certificate from the same CA as the certificate on the indexers and intermediate forwarders as well as the CA certificate configured
2. Ensure the indexers and intermediate forwarders have a cert present locally and installed at Splunkd with a Subject Alternative Name identical for all indexers and intermediate forwarders or uses an identical subdomain
3. Configure outputs.conf on the forwarder with indexer discovery
4. Check that the common name presented by the CM is given in the server.conf of the forwarder if sslCommonNameToCheck is set
5. Configure inputs.conf on the indexers and intermediate forwarders

Indexer Replication

1. Make sure the indexers have a cert present locally and installed at Splunkd
2. Configure the following in server.conf of the clustering app:

[replication_port-ssl:9887]
disabled = false
sslPassword = <password>
serverCert = $SPLUNK_HOME/local/certs/splunkd/<server>

3. Add the SAN that is identical for all indexers to sslAltNameToCheck or all common names to sslCommonNameToCheck in $SPLUNK_HOME/etc/system/local/server.conf if sslVerifyServerCert is set
4. Restart Splunk
5. Look for errors in $SPLUNK_HOME/var/log/splunk/splunkd.log
6. Check cluster status on the cluster manager running splunk show cluster-status --verbose

KV Store

Distinct configuration is required for the KVStore starting with Splunk 9.0.0. This configuration has to be added to server.conf. The certificate configured for the KVStore has to be encrypted. Therefore, if the certificate for Splunkd is not encrypted, an additional, encrypted certificate is required.

1. Create an encrypted .pem out of an existing .pem containing the certificate chain and the private key
2. Add following stanza to $SPLUNK_HOME/etc/system/local/server.conf:

[kvstore]
sslVerifyServerCert = true
sslVerifyServerName = true
sslPassword = <password>
serverCert = $SPLUNK_HOME/local/certs/kvstore/<server>

3. Restart Splunk

HTTP Event Collector (HEC)

HEC uses the same certificate as configured for Splunkd. Therefore no additional configuration is required if the splunkd cert can be used for HEC.

How Do I Get SSL Certificates?

Depending on the environment all certificates are provided and signed by an enterprise CA. If no certificates are provided a Splunk internal PKI has to be established. Host-based configuration files are used to ensure consistent and repeatable certificates.

Create a Custom CA Authority

Your CA has to be stored at a safe location. To prevent unauthorized parties from signing certificates the private key is stored in an encrypted format. First, create a configuration file for your CA called ca.cnf:

[req]
default_bits = 2048
distinguished_name = req_distinguished_name
prompt = no
x509_extensions = v3_ca

[req_distinguished_name]
C = CH
ST = Zurich
L = Zurich
O = Datapunctum AG
OU = Splunk
CN = CA

[v3_ca]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:TRUE, pathlen:3
keyUsage = critical, cRLSign, keyCertSign

Then generate the CA key pair:

export CA_FILE_NAME=<ca_file_name>
openssl genrsa -aes256 -out $CA_FILE_NAME.key 4096
openssl req -new -x509 -sha256 -days 3650 -key $CA_FILE_NAME.key -out $CA_FILE_NAME.pem -config ca.cnf

Verify the CA:

openssl x509 -in $CA_FILE_NAME.pem -noout -text

Create a Public/Private Key Pair

All certificates used in Splunk are signed by the CA just created. Create a configuration file per host (e.g. idx1.cnf):

[req]
default_bits = 2048
distinguished_name = req_distinguished_name
prompt = no
req_extensions = v3_req

[req_distinguished_name]
C = CH
ST = Zurich
L = Zurich
O = Datapunctum AG
OU = Splunk
CN = <hostname>

[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = <hostname>
DNS.2 = <fqdn>
DNS.3 = <san>
IP = <ip>

Generate the key pair:

export CERT_FILE_NAME=idx1
openssl genrsa -aes256 -out $CERT_FILE_NAME.key 2048
openssl req -extensions v3_req -sha256 -new -key $CERT_FILE_NAME.key -out $CERT_FILE_NAME.csr -config $CERT_FILE_NAME.cnf
openssl x509 -req -extensions v3_req -days 3650 -sha256 -in $CERT_FILE_NAME.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out $CERT_FILE_NAME.pem -extfile $CERT_FILE_NAME.cnf
cp $CERT_FILE_NAME.pem $CERT_FILE_NAME.web.pem
openssl rsa -in $CERT_FILE_NAME.key -out $CERT_FILE_NAME.web.key
cat $CERT_FILE_NAME.key >> $CERT_FILE_NAME.pem

Verify:

openssl x509 -in $CERT_FILE_NAME.pem -noout -text

The created .pem file can now be referenced from your Splunk configuration. For Splunk Web, use the .pem file for serverCert and the .key file for privKeyPath.

Use XCA

X Certificate and Key management (XCA in short) or get it from GitHub.

• Create a private key for your CA certificate
• Create a certificate based on the created CA private key
• Create a Template with all required information (country, organisation, etc.)
• Create a private key for each server that needs a certificate
• Create a certificate based on the created template and private key
• Export the CA certificate and server keypair to each server

XCA is easy to use but violates the best practice that a private key never leaves the server where it is used. So again, check the requirements of the customer before using XCA as a central tool for managing certificates.

Troubleshooting

• It is currently not possible to use requireClientCert in server.conf together with an encrypted private key in web.conf
• Set password to password if no password is set for the private key
• Some parts of Splunk Web expect privKeyPath and serverCert to be a relative path starting in /opt/splunk
• In the case of older Splunk versions, set the following additional settings:

sslVersions = tls, -ssl*
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ecdhCurves = prime256v1, secp384r1, secp521r1

• grep -E "ERROR\s(SSLCommon|HTTPServer)" $SPLUNK_HOME/var/log/splunk/splunkd.log
  • errno=185073780: ensure that serverCert contains the whole certificate chain, the certs are in the correct order and the certificates are not corrupt
  • errno=101077092: ensure that sslPassword is set correctly
  • errno=151404653: corrupt value in sslPassword
  • errno=33558530: check the configured paths for the certificates are correct
  • errno=218570875: the key is corrupt
• Splunk Web in combination with a certificate signed by a CA not trusted by users' browsers leads to many log lines in splunkd.log regarding sslv3 alert certificate unknown
• In the case of enabled Splunk Instrumentation and configured requireClientCert, various errors are visible in splunkd.log as the instrumentation fails to use certificates while connecting to the API
• In the case of enabled splunk_secure_gateway and configured requireClientCert, various errors are visible in splunkd.log as the app fails to use certificates while connecting to the API
• In case of enabled requireClientCert errors are found in splunkd.log from dmc_config.py as the script fails to use certificates while connecting to the API