I'm getting quite a few "Unable to distribute to peer..." messages when searching in splunk.
The reasons given tend to be '...because peer has status = "Down".' or Authentication Failed.
Sometimes just reloading the page will get through to the search peers. Sometimes it gives me that error a number of times in a row. But I've verified that the peer is not down, and I can connect to it from the search head with no problems.
The splunk servers are in different datacenters, and all I can think of is that there's a little bit of network lag and the connections aren't being made quickly enough?
Is there a config option to alter whatever timeout there is for this? Am I on the right track, or can someone suggest what else to look at?