Hello, I'm wondering if anyone else has run into this. I consistent see about 60% of our DC's with Splunk Forwarders not properly restarting. This always happens when I adjust a deployment app to push out that affect the DC's. The DC's all have the DNS/DC TA from Splunk for Active Directory.
The forwarders are all running 4.3.5 and Windows 2008 R2 x64. The result is the same on all of them: 1) Splunk.exe is running 2) Splunkd.exe is not running (nor any other splunk related process) 3) splunkd.log shows the following log entries.
Any idea what might be the cause here?
03-14-2013 14:48:51.890 -0700 WARN DeploymentClient - Restarting Splunkd...
03-14-2013 14:48:53.669 -0700 WARN HTTPServer - Disabling all acceptors due to shutdown
03-14-2013 14:48:53.669 -0700 WARN HTTPServer - Shutdown triggered via callback.
03-14-2013 14:48:53.669 -0700 WARN HTTPServer - HTTPServer shutting down. Terminating eloop.
03-14-2013 14:48:53.669 -0700 WARN HTTPServer - Closing all acceptors due to shutdown
03-14-2013 14:48:53.669 -0700 INFO loader - Shutdown HTTPDispatchThread
03-14-2013 14:48:53.731 -0700 INFO ShutdownHandler - Shutting down splunkd
03-14-2013 14:48:53.731 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_Thruput"
03-14-2013 14:48:53.731 -0700 INFO ShutdownHandler - Component 'ThruputProcessor' completed shutdown immediately
03-14-2013 14:48:53.731 -0700 INFO ShutdownHandler - Component 'ThruputProcessor' completed shutdown immediately
03-14-2013 14:48:53.731 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpInput"
03-14-2013 14:48:54.885 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpOutput"
03-14-2013 14:48:54.885 -0700 INFO TcpOutputProc - begin to shut down auto load balanced connection strategy
03-14-2013 14:48:54.885 -0700 INFO ShutdownHandler - Component 'TcpOutputProcessor' completed shutdown immediately
03-14-2013 14:48:54.885 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_UdpInput"
03-14-2013 14:48:54.885 -0700 INFO ShutdownHandler - Component 'UDPInputProcessor' completed shutdown immediately
03-14-2013 14:48:54.885 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_FifoInput"
03-14-2013 14:48:54.885 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_WinEventLogInput"
03-14-2013 14:48:54.885 -0700 INFO ShutdownHandler - Component 'WinEventLogInputProcessor' completed shutdown immediately
03-14-2013 14:48:54.885 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_Scheduler"
03-14-2013 14:48:54.885 -0700 INFO UDPInputProcessor - Unregistering metrics callback for: udpin_connections
03-14-2013 14:48:54.885 -0700 INFO ShutdownHandler - Component 'DispatchReaper' completed shutdown immediately
03-14-2013 14:48:54.885 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_Tailing"
03-14-2013 14:48:54.885 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_SyslogOutput"
03-14-2013 14:48:54.885 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_HTTPOutput"
03-14-2013 14:48:54.885 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_TailingXP"
03-14-2013 14:48:54.885 -0700 INFO TailingProcessor - Received shutdown signal.
03-14-2013 14:48:54.885 -0700 INFO TailingProcessor - Will reconfigure input.
03-14-2013 14:48:54.885 -0700 INFO TailingProcessor - Calling addFromAnywhere in TailWatcher=000000000831F350.
03-14-2013 14:48:54.885 -0700 INFO TailingProcessor - Shutting down with TailingShutdownActor=0000000001F34BF0 and TailWatcher=000000000831F350.
03-14-2013 14:48:54.885 -0700 INFO TailingProcessor - Pausing BatchReader module...
03-14-2013 14:48:54.885 -0700 INFO BatchReader - State transitioning from 0 to 1 (pseudoPause).
03-14-2013 14:48:54.885 -0700 INFO BatchReader - State transitioning from 1 to 2 (pseudoPause).
03-14-2013 14:48:54.885 -0700 INFO TailingProcessor - Removing TailWatcher from eventloop...
03-14-2013 14:48:54.885 -0700 INFO TailingProcessor - ...removed.
03-14-2013 14:48:54.885 -0700 INFO TailingProcessor - Eventloop terminated successfully.
03-14-2013 14:48:54.885 -0700 INFO TcpInputProc - Finishing shutdown
03-14-2013 14:48:54.885 -0700 WARN pipeline - Exiting pipeline tcp gracefully: got eExit from processor tcp
03-14-2013 14:48:54.885 -0700 INFO TailingProcessor - Signalling shutdown complete.
03-14-2013 14:48:54.885 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_BatchReader"
03-14-2013 14:48:54.885 -0700 INFO BatchReader - State transitioning from 2 to 3 (signalShutdown).
03-14-2013 14:48:54.885 -0700 INFO BatchReader - Terminating threadpool.
03-14-2013 14:48:54.901 -0700 INFO ShutdownHandler - Component 'BatchReader' completed shutdown immediately
03-14-2013 14:48:54.901 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_Archive"
03-14-2013 14:48:55.447 -0700 WARN pipeline - Exiting pipeline wineventlog gracefully: got eExit from processor wineventlog
03-14-2013 14:48:55.806 -0700 WARN pipeline - Exiting pipeline udp gracefully: got eExit from processor udp
03-14-2013 14:48:55.899 -0700 WARN pipeline - Exiting pipeline archivePipe gracefully: got eExit from processor archiveProcessor
03-14-2013 14:48:55.899 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_AuditTrailManager"
03-14-2013 14:48:55.899 -0700 INFO ShutdownHandler - Component 'AuditTrailManager' completed shutdown immediately
03-14-2013 14:48:55.899 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_AuditTrailQueueServiceThread"
03-14-2013 14:48:55.899 -0700 INFO ShutdownHandler - Component 'QueueServiceThread' completed shutdown immediately
03-14-2013 14:48:55.899 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_FSChangeMonitor"
03-14-2013 14:48:56.133 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_FSChangeManagerProcessor"
03-14-2013 14:48:56.133 -0700 INFO ShutdownHandler - Component 'FSChangeManagerProcessor' completed shutdown immediately
03-14-2013 14:48:56.133 -0700 INFO ShutdownHandler - shutting down level "ShutdownLevel_HttpClientPollingThread"
03-14-2013 14:48:57.070 -0700 WARN pipeline - Exiting pipeline fschangemanager gracefully: got eExit from processor fschangemanager