I'm trying to setup shuttl for the first time on Splunk 6, all the config files are populated per the documentation but when I look at the splunkd.log file I see the following and no buckets are actually archiving either:
01-11-2014 05:20:46.180 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/shuttl/bin/warmToColdRetry.sh" [Fatal Error] :1:3: The markup in the document preceding the root element must be well-formed. 01-11-2014 05:20:46.196 +0000 ERROR BucketMover - coldToFrozenScript [Fatal Error] :1:3: The markup in the document preceding the root element must be well-formed. 01-11-2014 05:20:46.197 +0000 ERROR BucketMover - coldToFrozenScript Exception in thread "main" com.splunk.HttpException: HTTP 400 01-11-2014 05:20:46.198 +0000 ERROR BucketMover - coldToFrozenScript at com.splunk.HttpException.create(HttpException.java:59) 01-11-2014 05:20:46.198 +0000 ERROR BucketMover - coldToFrozenScript at com.splunk.HttpService.send(HttpService.java:355) 01-11-2014 05:20:46.198 +0000 ERROR BucketMover - coldToFrozenScript at com.splunk.Service.send(Service.java:1211) 01-11-2014 05:20:46.199 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/shuttl/bin/warmToColdRetry.sh" Exception in thread "main" com.splunk.HttpException: HTTP 400 01-11-2014 05:20:46.200 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/shuttl/bin/warmToColdRetry.sh" at com.splunk.HttpException.create(HttpException.java:59) 01-11-2014 05:20:46.200 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/shuttl/bin/warmToColdRetry.sh" at com.splunk.HttpService.send(HttpService.java:355) 01-11-2014 05:20:46.200 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/shuttl/bin/warmToColdRetry.sh" at com.splunk.Service.send(Service.java:1211) 01-11-2014 05:20:46.200 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/shuttl/bin/warmToColdRetry.sh" at com.splunk.HttpService.post(HttpService.java:212) 01-11-2014 05:20:46.200 +0000 ERROR BucketMover - coldToFrozenScript at com.splunk.HttpService.post(HttpService.java:212) 01-11-2014 05:20:46.201 +0000 ERROR BucketMover - coldToFrozenScript at com.splunk.Service.login(Service.java:1044) 01-11-2014 05:20:46.201 +0000 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.thaw.SplunkIndexedLayerFactory.getLoggedInSplunkService(SplunkIndexedLayerFactory.java:33) 01-11-2014 05:20:46.201 +0000 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.copy.EntryPointUtil.getSplunkService(EntryPointUtil.java:41) 01-11-2014 05:20:46.201 +0000 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.copy.EntryPointUtil.getIndexNameForBucketDir(EntryPointUtil.java:34) 01-11-2014 05:20:46.201 +0000 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.archive.BucketFreezer.runMainWithDependencies(BucketFreezer.java:118) 01-11-2014 05:20:46.201 +0000 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.archive.BucketFreezer.main(BucketFreezer.java:106) 01-11-2014 05:20:46.204 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/shuttl/bin/warmToColdRetry.sh" at com.splunk.Service.login(Service.java:1044) 01-11-2014 05:20:46.208 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/shuttl/bin/warmToColdRetry.sh" at com.splunk.shuttl.archiver.thaw.SplunkIndexedLayerFactory.getLoggedInSplunkService(SplunkIndexedLayerFactory.java:33) 01-11-2014 05:20:46.209 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/shuttl/bin/warmToColdRetry.sh" at com.splunk.shuttl.archiver.retry.WarmToColdRetrier.main(WarmToColdRetrier.java:64) 01-11-2014 05:20:46.285 +0000 ERROR BucketMover - coldToFrozenScript cmd='/opt/splunk/etc/apps/shuttl/bin/coldToFrozenScript.sh /opt/splunk/var/lib/splunk/aws-cloudtrail/db/db_1389417448_1388521045_10' exited with non-zero status='exited with code 1'
and in shuttl.log I see this:
2014-01-11 05:20:44,473 ERROR com.splunk.shuttl.archiver.copy.ColdCopyEntryPoint: did="Called main entry point for copying bucket" happened="com.splunk.HttpException: HTTP 400" expected="to eventually call copy bucket REST endpoint" main_args="[/opt/splunk/var/lib/splunk/aws-cloudtrail/colddb/db_1386822646_1386814656_3]" 2014-01-11 05:20:44,535 ERROR com.splunk.shuttl.archiver.copy.ColdCopyEntryPoint: did="Called main entry point for copying bucket" happened="com.splunk.HttpException: HTTP 400" expected="to eventually call copy bucket REST endpoint" main_args="[/opt/splunk/var/lib/splunk/aws-cloudtrail/colddb/db_1386885274_1386866626_4]"
Anyone know what could be causing this?