I have an accelerated search which is set for a 3 months time range. The acceleration works, I can get a whole day's logs in the past in an average of 10 seconds, where it would take forever otherwise. I need to be able to see the data for all the same day of the week. But, since you can't specify a time range before an accelerated search query, you can't use "date_wday=Thursday". And doing this: | savedsearch "my_saved_search_name" | date_wday=Thursday won't help since it will force the acceleration to get all the records for the whole week so as to filter them afterward. This results in again an extremely lengthy search. My experiments show that the time it takes for acceleration increases exponentially with the time range you are looking at. Here is a little table to give you an idea of what I mean: Days search time 1 4 2 13 3 31 4 65 5 104 6 207 7 216 8 246
So, as I need to look at all the Thursdays for the last 6 weeks, I end up with a search that takes more than an hour to complete.
Any suggestion on how to get this working will be very appreciated.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Here is my answer regarding martin_mueller & lguinn's requests for the exact searches:
Actually I disagree. The principle should be applicable to any searches, it should not be dependent on my specific search. I have a search that's accelerated. I want to get the accelerated data only for a specific week day, say Thursday (this means all the Thursdays), in the past 6 weeks. And as I said earlier, the way I understand the usage of accelerated searches, you can't do this without looking at the whole 6 weeks worth of data. Unfortunately this nulls the value of accelerated reports.
But just to make you happier:
Accelerated search (3 months) - Name: acc_metric_ps4_create_account_all_history Query: index=apache uri="*/user/accounts.json" method=POST | bin _time span=1m | rex field=_raw "(?<response_time>d+) d+ "ajp_resource"" | stats count(eval(status=201)) as "Succ", count(eval(NOT status=201)) as "Fail", count as Total, avg(eval(response_time/1000000)) as Latency by _time
six weeks expected data for next day of the week: | savedsearch acc_metric_ps4_create_account_all_history [search earliest=-1s | head 1 | eval date_wday=strftime(relative_time(now(), "+1d@d"), "%A") | fields date_wday | format] | eval lat=round(Latency,2) | eval tot=round(Total) | eval succ=round(100-(Fail/Total*100),1) | eval _time=strptime(strftime(relative_time(now(), "+1d@d"), "%m/%d/%Y").strftime(_time,":%H:%M:%S"), "%m/%d/%Y:%H:%M:%S") | bucket _time span=1h | stats max(lat) as LATENCY_MAX_100, perc99(lat) as LATENCY_MAX_99, perc98(lat) as LATENCY_MAX_98, perc97(lat) as LATENCY_MAX_97, perc95(lat) as LATENCY_MAX_95, perc90(lat) as LATENCY_MAX_90, perc80(lat) as LATENCY_MAX_80, perc70(lat) as LATENCY_MAX_70, perc30(lat) as LATENCY_MIN_30, perc20(lat) as LATENCY_MIN_20, perc10(lat) as LATENCY_MIN_10, perc5(lat) as LATENCY_MIN_5, perc3(lat) as LATENCY_MIN_3, perc2(lat) as LATENCY_MIN_2, perc1(lat) as LATENCY_MIN_1, min(lat) as LATENCY_MIN_0, stdevp(lat) as LATENCY_STD_DEV, max(tot) as TOTAL_MAX_100, perc99(tot) as TOTAL_MAX_99, perc98(tot) as TOTAL_MAX_98, perc97(tot) as TOTAL_MAX_97, perc95(tot) as TOTAL_MAX_95, perc90(tot) as TOTAL_MAX_90, perc80(tot) as TOTAL_MAX_80, perc70(tot) as TOTAL_MAX_70, perc30(tot) as TOTAL_MIN_30, perc20(tot) as TOTAL_MIN_20, perc10(tot) as TOTAL_MIN_10, perc5(tot) as TOTAL_MIN_5, perc3(tot) as TOTAL_MIN_3, perc2(tot) as TOTAL_MIN_2, perc1(tot) as TOTAL_MIN_1, min(tot) as TOTAL_MIN_0, stdevp(tot) as TOTAL_STD_DEV, max(succ) as SUCCESS_MAX_100, perc99(succ) as SUCCESS_MAX_99, perc98(succ) as SUCCESS_MAX_98, perc97(succ) as SUCCESS_MAX_97, perc95(succ) as SUCCESS_MAX_95, perc90(succ) as SUCCESS_MAX_90, perc80(succ) as SUCCESS_MAX_80, perc70(succ) as SUCCESS_MAX_70, perc30(succ) as SUCCESS_MIN_30, perc20(succ) as SUCCESS_MIN_20, perc10(succ) as SUCCESS_MIN_10, perc5(succ) as SUCCESS_MIN_5, perc3(succ) as SUCCESS_MIN_3, perc2(succ) as SUCCESS_MIN_2, perc1(succ) as SUCCESS_MIN_1, min(succ) as SUCCESS_MIN_0, stdevp(succ) as SUCCESS_STD_DEV by _time | collect marker="bw_metric_ps4_create_account_all_expected"
This "expected" search takes almost 2 hours to complete. However I have devised a new technique which doesn't use the accelerated reports, and yet gets me the same results in 20 to 30 minutes. But I still would like to know if there is something I am missing here. Thank you very much for your interest and suggestions.