Splunk Crashing once in 10min

Hi All,

My Splunk instance 5.0.1 running in Solaris 10 is crashing. I have updated with the latest Splunk 5.0.3 but made it worse. It used generate crash logs and crash files (in /var/core directory) once in every 10 min. Not its twice in 10 min.

Can anyone help ?

Crash log is added below. Looks like the Report Acceleration is causing the crash.

Thanks

Kind Regards

KK

bash-3.2# more crash-2013-06-27-15:13:19.log
[build 163460] 2013-06-27 15:13:19
Received fatal signal 6 (Abort).
 Cause:
   Unknown signal origin (si_code=-1).
 Crashing thread: dispatch
 Registers:
    RIP:  [0xFFFFFD7FFEAE2CEA] __lwp_kill + 10 (/lib/amd64/libc.so.1)
    RDI:  [0x0000000000000003]
    RSI:  [0x0000000000000006]
    RBP:  [0xFFFFFD7FFE3FD3A0]
    RSP:  [0xFFFFFD7FFE3FD398]
    RAX:  [0x0000000000000000]
    RBX:  [0x0000000000000006]
    RCX:  [0x0000000000000005]
    RDX:  [0xFFFFFFFF83986C80]
    R8:  [0x000000000000002D]
    R9:  [0x0000000000000000]
    R10:  [0x0000000000000005]
    R11:  [0x0000000000000000]
    R12:  [0x0000000002CDB1B8]
    R13:  [0x0000000002CDB010]
    R14:  [0x0000000002CDB1E8]
    R15:  [0x0000000002CB9210]
    RFL:  [0x0000000000000286]
    TRAPNO:  [0x000000000000000E]
    ERR:  [0x0000000000000014]
    CS:  [0x000000000000004B]
    GS:  [0x0000000000000000]
    FS:  [0x0000000000000000]

 OS: SunOS
 Arch: x86-64

 Backtrace:
  [0xFFFFFFFFFFFFFFFF] ?
  [0xFFFFFD7FFEA87E99] raise + 25 (/lib/amd64/libc.so.1)
  [0xFFFFFD7FFEA6694E] abort + 94 (/lib/amd64/libc.so.1)
  [0x0000000001A0161F] _ZN9__gnu_cxx27__verbose_terminate_handlerEv + 351 (/opt/splunk/bin/splunkd)
  [0x0000000001A002A6] _ZN10__cxxabiv111__terminateEPFvvE + 6 (/opt/splunk/bin/splunkd)
  [0x0000000001A002D3] _ZSt9terminatev + 19 (/opt/splunk/bin/splunkd)
  [0x0000000001A0065F] __cxa_pure_virtual + 31 (/opt/splunk/bin/splunkd)
  [0x0000000000D1E73E] _ZN15SearchEvaluator10lispyQueryER3StrR7TimevalS3_R9StrVectorRKS2_S7_b + 414 (/opt/splunk/bin/splunkd)
  [0x0000000000B3644C] _ZN17IndexScopedSearch4initERK7TimevalS2_bP14LookupOperatorP12FieldAliaserP18CalcFieldProcessorPKSt3setI10CMBucketIdSt4lessISA_ESaISA_EE + 588 (/opt/splunk/bin/splunkd)
  [0x0000000000B26C9E] _ZN14SearchOperator8evalArgsER17SearchResultsInfo + 9006 (/opt/splunk/bin/splunkd)
  [0x0000000000FA901A] _ZN14SearchPipeline8evalArgsER17SearchResultsInfo + 90 (/opt/splunk/bin/splunkd)
  [0x0000000000B572B9] _ZN22BucketSummaryProcessor8evalArgsER17SearchResultsInfo + 8713 (/opt/splunk/bin/splunkd)
  [0x0000000000FA901A] _ZN14SearchPipeline8evalArgsER17SearchResultsInfo + 90 (/opt/splunk/bin/splunkd)
  [0x000000000103B120] _ZN14DispatchThread8evaluateEbb + 15264 (/opt/splunk/bin/splunkd)
  [0x0000000001033981] _ZN14DispatchThread8mainImplEv + 4321 (/opt/splunk/bin/splunkd)
  [0x00000000010368C2] _ZN14DispatchThread4mainEv + 226 (/opt/splunk/bin/splunkd)
  [0x0000000000F37352] _ZN6Thread8callMainEPv + 98 (/opt/splunk/bin/splunkd)
  [0xFFFFFD7FFEADD1AB] _thr_slot_offset + 795 (/lib/amd64/libc.so.1)
  [0xFFFFFD7FFEADD3E0] smt_pause + 96 (/lib/amd64/libc.so.1)
 SunOS / splunk / 5.10 / Generic_147441-07 / i86pc
 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    2013-06-26 17:19:51.400 +1000 splunkd started (build 143156)
    2013-06-26 17:25:11.350 +1000 Interrupt signal received
    2013-06-26 17:27:59.775 +1000 splunkd started (build 143156)
    2013-06-27 12:21:03.153 +1000 Interrupt signal received
    2013-06-27 12:21:56.892 +1000 splunkd started (build 143156)
    2013-06-27 13:21:08.304 +1000 Interrupt signal received
    2013-06-27 13:37:12.340 +1000 splunkd started (build 163460)
    2013-06-27 13:39:12.006 +1000 Interrupt signal received
    2013-06-27 13:39:59.495 +1000 splunkd started (build 163460)
    2013-06-27 13:52:08.211 +1000 Interrupt signal received
    2013-06-27 13:52:58.376 +1000 splunkd started (build 163460)
    2013-06-27 14:50:25.221 +1000 Interrupt signal received
    2013-06-27 15:04:15.911 +1000 splunkd started (build 163460)

Threads running: 3
argv: [splunkd -p 8089 start]
Process renamed: [splunkd pid=3972] splunkd -p 8089 start [process-runner]
Process renamed: [splunkd pid=3972] search --id=SummaryDirector_1372309985.40 --maxbuckets=0 --ttl=30 --maxout=50000 --maxtime=8640000 --lookups=0 --reduce_freq=10 --user=splunk-system-user --pro --roles=admin:can_delete:cds:power:splunk
-system-role:user
terminating...
bash-3.2#