splunk is trigerring duplicate events from syslog.

Hi

I have been using syslog to store my server logs and splunk will be monitoring the syslog.log file located at /opt/splunk/var/syslog-ng/ path. Now while splunk montoring the files i could see duplicate events in my logs. when i checked the splunkd log file i could see at partiucular timestamps i.e

06-17-2013 07:18:48.691 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'. 06-17-2013 07:18:48.691 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.

i could see splunk reading the file twice ..hence i could see duplicates events in my index. Posted you the snippet of splunkd log file.

06-17-2013 07:18:30.689 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:33.690 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:36.690 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:39.690 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:42.690 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:45.692 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:48.691 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:48.691 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:50.551 +0100 INFO  BatchReader - Removed from queue file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:56.561 +0100 INFO  TcpOutputProc - Connected to idx=host1:8089
06-17-2013 07:19:26.563 +0100 INFO  TcpOutputProc - Connected to idx=host2:8089
06-17-2013 07:19:56.576 +0100 INFO  TcpOutputProc - Connected to idx=host3:8089

Can any one help me.. wats happening here .why splunk is reading a file a twice and generating duplicate events ??

for Syslog-log rotation i have defined the following configuration in syslog-ng file

//syslog-ng logrotation configuration

/etc/logrotate.d/syslog-ng

/opt/splunk/var/syslog-ng/syslog.log {
        size 30M
        copytruncate
        create 750 splunk splunk
        rotate 500
}

crontab - entry to check the syslog size every 5 min and rotate

// crontab

#Added entry to rotate logs generated from syslog-ng
*/5 * * * * /usr/sbin/logrotate /etc/logrotate.d/syslog-ng

I cleary see duplicates . You can find the same with the screenshot below.