Hello everyone,
I'm having issues keeping my dispatch directory down to a manageable level. What I mean by that is for the past week, every two days I log in to do a manual search and I cannot because the dispatch directory has some 30000 jobs where the warning level is 2000.
So I go in and clean out the dispatch directory, restart splunk and we are back in business. My issue is that sometimes I don't touch splunk for a few weeks, and if splunkd stops, we will lose the alerts that we get for certain situations.
About the alerts, I have about 6 rt scheduled searches that run rt-1m to rt-0m checking in the last minute for a set of alert conditions. Usually they are quiet but sometimes we get many, this is intentional, and of course they are throttled to be reasonable.
I would like to fix the dispatch issue some way in splunk. My other solution is to set a script in windows task scheduler to clear the dispatch directory once per night.
Any help is appreciated!