When running splunk list deploy-clients on deployment servers, I have noticed that for some deployment-clients, the value of hostname does not match the name of the host included in the value of dns. For example, in one record for a computer whose name is truly "host1", the hostname value is accurately presented as "host1" but the hostname attribute of the dns value is inaccurately presented as "host3.domain.com".
According to http://docs.splunk.com/Documentation/Splunk/6.0.1/Admin/Serverclassconf, the dns value is supposed to be derived from a reverse dns lookup. Running splunk in debug mode on the deployment client, I can see that the deployment client is posting the dns value in question to the deployment server, so the inaccurate lookup must be occurring on the deployment client and not the deployment server.
The thing is, when running nslookup on a deployment client in question, the returned dns name is correct. Also, when reviewing the PTR record of the ip address in question, the record seems to be accurate there as well. I've even gone as far as to manipulate the reverse DNS record to an inaccurate value for another deployment-client to see if this influences what the deployment client presents to deployment server during phoneHome activity after splunk restart. -It doesn't. I'm at the point where I do not think Splunk is truly using a reverse dns lookup for the dns value provided by deployment clients to deployment servers.
On that theory I've been trying to figure out what splunk is using the derive dns value from on windows-based deployment clients. To do so, I ran process monitor during splunk startup. I filtered on events having expected domain or expected hostname or inaccurate hostname in details column of i/o request. What I found is that splunk is performing queries on a couple of registry value names for activeComputerName, hostname, and domain. I tried manipulating those values and restarting splunk to determine if those registry values truly influence what deployment-clients post during phoneHome as dns value to deployment servers.. It turns out they those registry values DO influence but instead of uploading the values contained in the registry, the deployment-clients started reporting IP address in dns field.. At this point I figure splunk is stringing together the dns name from multiple sources, performing some sort of verification and making a decision on perceived quality prior to selecting a value to post. It would be nice to know what is being factored so that I can correct the root cause of issues in our environment.
Anywho.. Guess I wanted to alert community to this situation in the case that deployment-apps are getting deployed to deployment-clients inexplicably as a result of multiple matches on serverclass whitelist entries on hostname and inaccurate dns, and to draw upon your experience for what to do moving forward.