Hi,
We noticed an issue in one of our developers system. He setup the application 30 days back, but did not pump any data. On 26th of this month he put log files of 24th in forwarder and started running back-fill commands.
In one of our back-fill commands we have 8 searches. All these 8 searches have collect command in the end to collect summarized events into summary index.
Out of the 8, 6 searches results are collected into summary index. Where as the first two searches results are not collected.
We noticed the following in our logs:
The first two searches started at 2013-06-26 15:10:39,219 and 2013-06-26 15:10:39,317 respectively.
The first two searches ended at 2013-06-26 15:10:39,220 and 2013-06-26 15:10:39,318 respectively.
In splunkd log of indexer node:
06-26-2013 15:10:39.349 INFO HotDBManager - index=summary_XXXX No hot found for event ts=1372012200, closest match=null [expanded span=0] hotbucketsize=0 numbucks=1 maxhot=3
06-26-2013 15:10:39.349 INFO databasePartitionPolicy - creating new bucket /opt/splunk/var/lib/splunk/XXXX/db/hot_v1_0
06-26-2013 15:10:39.350 INFO databasePartitionPolicy - lazy loading database for: /opt/splunk/var/lib/splunk/XXXX/db/hot_v1_0, id=0, ts=1372012200 dirMgr::nextId=0]
06-26-2013 15:10:39.350 INFO HotDBManager - index=summary_XXXX Creating new hot (id=0, time=1372012200)
All the other searches started and ended after 2013-06-26 15:10:39,426
How long the results of first two searches would have stayed in stash? What is the reason for the first two searches results not being collected in summary index?
What can we do to overcome this issue? Can we ensure that the hot dbs are available before we start executing searches to collect summarized data.
Thanks
Strive