My custom search command is getting Memory Error while out putting data.
splunk.Intersplunk.outputResults(output, None, fields)
File "/opt/splunk/lib/python2.7/site-packages/splunk/Intersplunk.py", line 181, in outputResults dw.writerows(results) File "/opt/splunk/lib/python2.7/csv.py", line 153, in writerows rows.append(self._dict_to_list(rowdict)) File "/opt/splunk/lib/python2.7/csv.py", line 145, in _dict_to_list return [rowdict.get(key, self.restval) for key in self.fieldnames] MemoryError
The search command uses splunk.search.dispatch() to call a search which can generate tons of events. What is the proper way to call splunk.Intersplunk.outputResults? can I call this function multiples times with smaller window chunk size?
--Edit
I have change the code to output in chunks by calling the outputResults multiple times, but now i get this error instead
splunk.Intersplunk.outputResults(output, None, fields)
File "/opt/splunk/lib/python2.7/site-packages/splunk/Intersplunk.py", line 181, in outputResults dw.writerows(results) File "/opt/splunk/lib/python2.7/csv.py", line 154, in writerows return self.writer.writerows(rows) IOError: [Errno 32] Broken pipe