Deleting old cold indexes

Hi, All.

I was running the following query[1] on one of my indexers, getting an overview of how many buckets are in play at the moment. Most of the indexes fell into roughly the same number of hot, warm, and cold buckets, like I expected them to.

However, I did run across one index which has over 31,000 cold buckets, and over 14,000 warm buckets- above and beyond any other of the 15 or so indexes I was looking at.

It makes it confusing since all of my indexes on this indexer are setup very generically- hot buckets age one day, warm 30-90, frozen on day 91. Here’s a generic example of the configuration for each of the indexes.

[generic_index_name]
homePath   = $SPLUNK_DB/generic_index_name/db
coldPath   = $SPLUNK_DB/ generic_index_name/colddb
thawedPath = $SPLUNK_DB/ generic_index_name/thaweddb
maxMemMB = 10
frozenTimePeriodInSecs = 7776000
maxConcurrentOptimizes = 6
maxHotIdleSecs = 86400
maxHotSpanSecs = 86400
maxWarmDBCount = 30
maxHotBuckets = 10
maxDataSize = auto
maxTotalDataSizeMB = 102400

Any recommendations on removing all the extra data without causing an exaggerated load on the indexer? I considered cutting the frozenTimePeriodInSecs in half on the index and letting Splunk delete out the old data. However, since this one particular index has about 30 times what is normal, I thought it best to run it by the community for any recommendations first. There's quite a bit written about moving indexes from warm to cold, but didn't really see anything fitting when handling cold to frozen.

Thanks for any input you may have!

[1] here’s the query. set search for ‘all time’

|dbinspect index=(indexname) |convert timeformat=""%m/%d/%Y:%H:%M:%S"" mktime(earliestTime) as earliestTime|convert timeformat=""%m/%d/%Y:%H:%M:%S"" mktime(latestTime) as latestTime|stats min(earliestTime) as earliestTime max(latestTime) as latestTime sum(sizeOnDiskMB) as sizeOnDiskMB dc(path) as NumberOfBuckets by state|eval diff_seconds=(latestTime-earliestTime)/3600|eval earliestTime=strftime(earliestTime,"%m/%d/%Y:%H:%M:%S")|eval latestTime=strftime(latestTime,"%m/%d/%Y:%H:%M:%S")