I have been having trouble getting my forwarders working from within an Amazon VPC environment. All the severs I want to forward logs from have to go through the proxy to access the outside world.
I have tried two topologies and neither worked correctly.
First I tried forwarding directly to the splunk storm endpoint directly with my http(s)_proxy environmental variables set when starting splunkd. No luck, I just get a bunch of
WARN TcpOutputProc - Cooked connection to ip=107.22.55.186:9997 timed out
WARN TcpOutputProc - Cooked connection to ip=50.17.98.179:9997 timed out
type events in var/log/splunk/splunkd.log which seem to indicate the forwarder is not using the proxy
I also tried setting up an intermediate forwarder in a subnet that was reachable by my servers and could also access the internet directly. This partially worked. I can get log events into storm by setting my firewalled servers to forward to the intermediate server which could forward to storm. However, in this setup everything showed up in storm as type tcp-raw with contents --splunk-cooked-mode-v3--\x00 [...], even when I set the appropriate source type in the monitor.
Does anyone have any ideas about getting forwarders to work properly -- either directly or through an intermediate forwarder in a firewalled/proxy environment?