Ping, Ping, Ping, Are You There?

As a SharePoint Technology Architect you have to know a little bit of everything in the platform stack from Windows Server to Active Directory to DNS to networking, and much more, a veritable jack-of-all-platform-trades. On a recent project I was working with an IT Pro to prepare a new server virtual machine for a proof-of-technology for SharePoint 2016 when I was reminded of this.

This project teanm is hard up for infrastructure so we are scrounging for virtual machines and have to make do with a refurbished hand-me-down VM. The server team re-imaged the VM with Windows Server 2012 R2 as is required for SharePoint 2016. The initial sanity check of the refurbished VM is good, we can RDP to it and Server Manager Dashboard shows a healthy server, all services running normally.

vm-server-manager-dashboard

Next  step is we have to request a DNS entry for the Central Admin site for SharePoint 2016, hence we need the IP address(es) of the new VM. As a quick step I ping the VM rather than RDP’ing to inspect its network adaptor configuration:

0 ping

Whoa, ping does not respond! How can that be? I just RDP’ed to the server and it was fine.

By the way, grey rectangles blot out client-specific details. Also ignore that the server name is GCDOCS, that is an artefact of the server refurbishing, the server team did not rename the refurbished VM. It will be a SharePoint Server 2016 VM and no longer has any trace of GCDOCS, the Government of Canada’s variant of OpenText’s Content Server product.

Here is that command sequence again in text format:

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\calvjoh>ping gcdocs

Pinging gcdocs.XXXXXXX.ca [XXX.XXX.43.155] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for XXX.XXX.43.155:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

C:\Users\calvjoh>ping gcdocs.XXXXXXX.ca

Pinging gcdocs.XXXXXX.ca [XXX.XXX.43.155] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for XXX.XXX.43.155:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

Oh yeah, some IT shops blog ping as a security measure, so servers don’t leak information. Could it be that? Seems unlikely since we are in the Intranet zone of a well partitioned corporate network with DMZ and other network segregations.

I drop over to my colleagues desk and ask them to run the ping to compare.

[Insert image: successful ping]

Go figure, it works fine! And it responds on a different IP address.How could that be?

Oh yeah, the server team mentioned that they changed the IP address assigned to the refurbished VM. Their old networking configuration standard involved “routable” or public IP addresses in the Intranet zone but they have phased that out so all new (or refurbished) VMs are assigned a “non-routable” or Private Network IP address, eg in the 24-bit block 10.0.0.0 – 10.255.255.255 class A IP address range. A much more robust security posture. (Networking ninjas, please excuse the terminology routable vs non-routable, I’m using this euphemistically so readers don’t have to dig into RFC 1918 – Address Allocation for Private Internets).

Perhaps my workstation has in its DNS cache an old IP address for the GCDOCS server. Time to run ipconfig /dnsflush to clear out the DNS cache and try that ping command again!

Oh dear, the result is the same, boo hoo!

Seems like it is time to check the VM’s network adaptor config to see what IP address(es) are assigned. I remote over to the VM and open up Network and Sharing Centre in Control Panel:

1 VM Network and Sharing Center

I double-click Domain network to open the Network Connections page:

2 VM Network Connections

Then double click the adaptor to open its status property sheet:

3 VM Net X Status

Then open the Properties and double click the Internet Protocol Version 4 (TCP/IPv4)) item:

5 VM Net X Properties - IPv4

To open the IPv4 properties sheet:

6 VM Net X Properties - IPv4 - Properties

Which only shows the primary IP address so I also open the Advanced property sheet to show all secondary IP addresses assigned to the adaptor:

7 VM Net X Properties - IPv4 - Properties - Advanced

Nothing but 10.204.161.11, no sign of XXX.XXX.43.155. So the failed pings returning XXX.XXX.43.155 are not related to two IP addresses bound to the network adaptor.

My next thought, surely there are multiple redundant or load balanced name servers so perhaps one is returning the correct IP address 10.204.161.11 and another one is returning an invalid deprecated IP address XXX.XXX.43.155.

I consult with my IT Pro to find out how to query the name servers to see what they doing; he recommends the nslookup GCDOCS command:

8 nslookup

Text version:

C:\Users\calvjoh>nslookup gcdocs
Server:  XXXdca4.XXXXXXX.ca
Address:  10.204.232.14

Name:    gcdocs.XXXXXXX.ca
Addresses:  10.204.161.11
XXX.XXX.43.155

Well low and behold, there are the two IP addresses bound to the name GCDOCS. Or at least that is what domain controller DCA4 says.

I pop back over to my IT Pro’s desk and get him to run the nslookup command too. Recall that his ping on GCDOCS returned correctly the 10.204.161.11 IP address. But his nslookup gives the exact same result. Go figure, what to make of that?

Time to check DNS Manager and see what is defined in there:

9 DNS Manager

BINGO! DNS has two A records for the name GCDOCS, one with the correct IP address 10.204.161.11 and the other with the deprecated IP address XXX.XXX.43.155. The VM’s Domain network adaptor is only bound to the first IP address, hence no server is listening to respond to ping on the second IP address. This must be the source of the problem. When the server team team updated the refurbished VM’s network adaptor settings perhaps they forgot to remove the old IP address binding in DNS.

Ping’s behaviour at any given moment depends on which IP address DNS serves up when asked to resolve the name GCDOCS. The exact details of DNS resolution are murky to me, but no matter, we will very likely resolve the problem by removing the errant DNS entry:

[Insert image: DNS cleaned up]

And voila, ping is now working:

ping-fixed

This might seem like a rare circumstance, DNS misconfigurations such as multiple IP addresses assigned to a DNS name but not bound to any network adaptor, but it is not so rare, I’ve seen it before, and between various team members lost several person-days of effort troubleshooting various SharePoint configuration and web application provisioning errors. So now I always make a point to check any new DNS entries with ping and nslookup to be sure the entries are as expected before I implement them in SharePoint.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s