I'm getting the message "maximum number of historical concurrent system-wide searches has been reached current=10 maximum=8", and searches go to paused without running.
When I navigate to Activity->Jobs and set App=All I can see about a dozen jobs with "Dispatched at" dates of 24 hours ago or more, up to 2 months in some cases. The Status of all of these jobs is Done. The jobs are all of the form "| summarize maintain=..." or "| summarize tstats=...".
There are also some with a Dispatched at date of 1/1/70 - I'm guessing that these are actually new jobs with a null Dispatched at date. The status of all of these jobs is "Running(0%)".
By deleting some of the "summarize ..." jobs, I can get the paused jobs to start - but why is this necessary? Why are these old jobs still alive? And why was current more than the maximum?
Search job inspector
This search has completed, but did not match any events. The terms specified in the highlighted portion of the search:
None | summarize tstats=t maintain="" summaryprefix="D785B310-3FEB-4653-AE1B-BD922FC9E375"
over the time range:
(earliest indexed event) – 1/1/01 12:00:00.000 AM
did not return any data. Possible solutions are to:
relax the primary search criteria
widen the time range of the search
check that the default search indexes for your account include the desired indexes
The following messages were returned by the search subsystem:
DEBUG: search context: user="splunk-system-user", app="system", bs-pathname="/opt/mcapps/splunk/etc"
Learn more about troubleshooting empty search results at Splunk Documentation.
(SID: SummaryDirector_1384706042.531)
Execution costs
Duration (seconds) Component Invocations Input count Output count
0.05 command.presummarize 1 - -
0.002 command.summarize.execute_input 2 - -
0.001 command.summarize.execute_output 1 - -
0.001 dispatch.check_disk_usage 1 - -
0.033 dispatch.createProviderQueue 1 - -
0.002 dispatch.evaluate 1 - -
0.001 dispatch.evaluate.summarize 1 - -
0.051 dispatch.fetch 2 - -
0.05 dispatch.stream.local 1 - -
0.053 dispatch.writeStatus 5 - -
0.122 startup.handoff 1 - -
Search job properties
canSummarize 0
createTime 2013-11-18T03:34:02.000+11:00
cursorTime 1970-01-01T10:00:00.000+10:00
defaultSaveTTL 604800
defaultTTL 600
delegate None
diskUsage 53248
dispatchState DONE
doneProgress 1.0
dropCount 0
eai:acl {'ttl': '30', 'modifiable': '1', 'app': 'system', 'sharing': 'global', 'owner': 'splunk-system-user', 'perms': {'read': ['splunk-system-user'], 'write': ['splunk-system-user']}, 'can_write': '1'}
earliestTime 1970-01-01T10:00:00.000+10:00
eventAvailableCount 0
eventCount 0
eventFieldCount 0
eventIsStreaming True
eventIsTruncated True
eventSearch None
eventSorting desc
isBatchModeSearch False
isDone True
isFailed False
isFinalized False
isPaused False
isPreviewEnabled False
isRealTimeSearch False
isRemoteTimeline False
isSaved False
isSavedSearch False
isZombie False
keywords None
label None
messages {'debug': ['search context: user="splunk-system-user", app="system", bs-pathname="/opt/mcapps/splunk/etc"']}
modifiedTime 2014-01-30T13:47:53.000+11:00
normalizedSearch presummarize tstats=t maintain="" summaryprefix="D785B310-3FEB-4653-AE1B-BD922FC9E375"
numPreviews 0
performance {'dispatch.createProviderQueue': {'duration_secs': '0.033000', 'invocations': '1'}, 'startup.handoff': {'duration_secs': '0.122000', 'invocations': '1'}, 'command.summarize.execute_input': {'duration_secs': '0.002000', 'invocations': '2'}, 'command.presummarize': {'input_count': '0', 'duration_secs': '0.050000', 'output_count': '0', 'invocations': '1'}, 'dispatch.stream.local': {'duration_secs': '0.050000', 'invocations': '1'}, 'dispatch.evaluate.summarize': {'duration_secs': '0.001000', 'invocations': '1'}, 'command.summarize.execute_output': {'duration_secs': '0.001000', 'invocations': '1'}, 'dispatch.check_disk_usage': {'duration_secs': '0.001000', 'invocations': '1'}, 'dispatch.evaluate': {'duration_secs': '0.002000', 'invocations': '1'}, 'dispatch.writeStatus': {'duration_secs': '0.053000', 'invocations': '5'}, 'dispatch.fetch': {'duration_secs': '0.051000', 'invocations': '2'}}
pid 9770
priority 5
remoteSearch presummarize tstats=t maintain="" summaryprefix="D785B310-3FEB-4653-AE1B-BD922FC9E375"
reportSearch summarize tstats=t maintain="" summaryprefix="D785B310-3FEB-4653-AE1B-BD922FC9E375"
resultCount 0
resultIsStreaming False
resultPreviewCount 0
runDuration 0.137
scanCount 0
search summarize tstats=t maintain="" summaryprefix="D785B310-3FEB-4653-AE1B-BD922FC9E375"
searchProviders ['d26li198']
sid SummaryDirector_1384706042.531
statusBuckets 0
ttl 30
Additional info search.log
Server info: Splunk 6.0, d26li198:8000, Thu Jan 30 13:47:53 2014 User: admin