Indexer – 2K8R2-64, Splunk 5.0.3, DB Connect 1.0.9, Latest JDK, ojdbc6,
Oracle 11 Server – 2K8R2-64
The problem is that after a Splunk restart (from manager, CLI, server power button) the tailing inputs fail.
This problem only occurs when there are two or more database connections either on the same Oracle server or one connection each on two different oracle servers.
There are two ways to recover from the problem:
1) Delete and recreate the inputs in DB Connect (this results in complete dump of the records causing duplicates)
2) (This is the weird one) Run a DB Connect query on one of the databases and kill Java in Task Manager, then re-run the query. All connections will start working until the next Splunk restart.
I’ve been trying to figure this out for a long time, and Splunk support has not yet found a solution. It does not matter what order Java/Splunk are started, or how the server is restarted.
A DB Connect query must be running when Java is killed to restore the inputs.
There are no errors in any log at debug level. The logs indicate that everything is normal, but log sequences that would normally cycle to completion (scheduled tailing event for example) simply stop.
I have recreated this problem in two separate networks, but Splunk support has not been able to recreate it. Obviously it is me, what am I doing wrong?
Thanks