We have performance problems. Looking at one of the search logs, I see that it ends with
08-16-2013 14:00:55.172 INFO NewTransam - Finalizing. Committing all open txns withheld
08-16-2013 14:01:26.542 INFO NewTransam - evicted=0, closed=554, event_count=163158, candidate_events=162604, comparison_count=974503
08-16-2013 14:01:26.542 INFO NewTransam - Finalizing/previewing. Committing all open txns withheld
08-16-2013 14:01:26.547 INFO NewTransam - evicted=0, closed=554, event_count=163158, candidate_events=162604, comparison_count=974503
08-16-2013 14:01:26.555 INFO NewTransam - Finalizing/previewing. Committing all open txns withheld
08-16-2013 14:11:27.427 INFO NewTransam - evicted=0, closed=8, event_count=554, candidate_events=546, comparison_count=670
08-16-2013 14:11:35.686 INFO script - Invoked script vgformatmaintable with 71613951 input bytes (8 events). Returned 496 output bytes in 1049 ms.
08-16-2013 14:11:35.692 INFO UserManager - Unwound user context: spotter -> NULL
08-16-2013 14:11:35.694 INFO UserManager - Setting user context: spotter
08-16-2013 14:11:35.694 INFO UserManager - Done setting user context: NULL -> spotter
08-16-2013 14:11:35.694 INFO UserManager - Unwound user context: spotter -> NULL
08-16-2013 14:11:35.694 INFO DispatchCommand - DispatchManager::dispatchHasFinished(id='1376686831.442', username='spotter')
08-16-2013 14:11:35.830 INFO UserManager - Unwound user context: spotter -> NULL
Notice the jump from 14:01 to 14:11, when it is supposedly "Finalizing/previewing". What is it actually doing during those ten minutes? Any suggestions for debugging this?
UPDATE: I just attached to the search process with 'strace'. For 10 minutes the only system calls the process made were 'madvise MADV_DONTNEED' and 'mmap'.