We currently have the limits.conf max_mem_usage_mb parameter value set to 2000, which is 10x the default value (200). We have noticed incidences of splunk helper processes being killed due to OOM, and is suspecting that it might have something to do with max_mem_usage_mb value being too high:
01-27-2014 15:54:18.645 -0500 ERROR STMgr - dir='/opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_43' out of memory failure rc=1 warm_rc[-2,12] from st_txn_start 01-27-2014 15:54:18.645 -0500 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index 01-27-2014 15:54:18.694 -0500 FATAL ProcessRunner - Unexpected EOF from process runner child!
This limit was set to a high value to accommodate the end user running extremely large ad hoc queries. This decision was taken since the system has an extremely large amount of memory (160 GB) and a large number of cores (32) available.
As a rule of thumb, what is the maximum range of values that can be assigned to max_mem_usage_mb? Does it have a relation to the max available memory (160 GB in this case)?
http://docs.splunk.com/Documentation/Splunk/5.0.7/Admin/Limitsconf