In our splunk deployment, we have about 100 universal forwarders installed on PCs and forward data to a splunk server (port: 998) that will further forward the data to another splunk server. The intermediate splunk server also serves as a deployment server (port: 8089) to deploy apps to the PCs. We tested this on about 20 machines initially and it worked fine. But after we expanded the deployment to the rest of the machines, we noticed the intermediate splunk server is rejecting the connections from the clients, citing SSL error.
ERROR TcpInputFd - SSL_ERROR_SYSCALL ret errno:0
ERROR TcpInputFd - SSL Error = error:00000000:lib(0):func(0):reason(0)
ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
ERROR TcpInputFd - SSL Error for fd from HOST:x.x.x.x, IP:x.x.x.x, PORT:2933
We looked at the splunk log at one of the PCs. The log clearly said that it attempts to open a non-SSL connection "INFO TcpOutputProc - Initializing connection for non-ssl forwaring to x.x.x.x". We are puzzled that why the error is related to SSL at the receiver end.
Any advice?