Archive
Troubleshooting Basic Network Connectivity Using A Telnet Client
The telnet client included with Windows is a useful low level diagnostic tool. When troubleshooting a network connectivity issue, it is a simple and effective way to determine if you can establish a TCP connection with a remote host on a particular port. The command syntax is:
telnet <IP address, FQDN or hostname> <port>
For example…
telnet fileserver.celestix.net 445
This command instructs the telnet client to open a TCP connection to the server fileserver.celestix.net on port 445 (which is used by SMB). A successful TCP connection was made if the command prompt disappears and you are left with only a flashing cursor.
You can confirm this by monitoring the communication with a protocol analyzer (e.g. Network Monitor or Wireshark) when you execute the command.
Note: The example above uses a fully-qualified domain name (FQDN). When using FQDN’s or hostnames, make certain that name resolution is working properly if you are unable to establish connectivity.
Additional Examples
telnet dc01 389
This command establishes connectivity to dc01 on TCP port 389 (LDAP).
telnet 192.168.1.100 3389
This command establishes connectivity to 192.168.1.100 on TCP port 3389 (RDP).
Unable To Connect
If a connection cannot be established you will receive the following error message:
Could not open connection to the host, on port <port number>: Connect failed
This error message does not explain why the connection could not be established, however. It may be that the remote host was not listening on the specified port, the remote host was not reachable, or perhaps a firewall on the remote host or in the network path between the client and the remote host denied the connection. Looking at the communication with a protocol analyzer provides more information. For example, if the remote host is reachable but not listening on the port requested, the remote host will reply with a TCP reset.
If the remote host was not reachable or a firewall denies the communication there will be no replies.
Telnet Clients
Historically the telnet client has been installed by default on all Windows operating systems. That changed beginning with Windows Vista. With Windows Vista and Windows 7 you will need to open the ‘Programs and Features’ control panel application and in the task pane select the option to ‘Turn Windows features on or off’.
Select ‘Telnet Client’ and choose ‘Ok’ to install it.
On Windows Server 2008 systems it is even easier to install the telnet client. From an elevated command prompt enter the following command:
servermanagercmd -i telnet-client
In addition there are numerous third-party telnet clients available for Windows. Some are free such as the popular Putty telnet client, and some are commercial such as SecureCRT. Third-party telnet clients offer advanced featuers not found in the native Windows telnet client and also support additional protocols such as SSH and RLogin.
In my next post I will demonstrate how to perform basic HTTP troubleshooting using a telnet client.
DNS Security Enhancements and Web Proxy Auto Discovery
Using Web Proxy Auto Discovery (WPAD) is a simple and effective way to configure web browsers to use the ISA firewall as a proxy server. WPAD can be implemented using DNS or DHCP, with DNS being the more common of the two. For WPAD using DNS, configuration is simple and straightforward; all that is required is that you configure a host record in DNS called WPAD that resolves to the IP address of your ISA firewall’s internal network interface.
On the ISA firewall, enable the ‘Publish automatic discovery information for this network’ option on the ‘Auto Discovery’ tab for the Internal network.
For Internet Explorer, navigate to ‘Tools/Internet Options/Connections/Lan Settings’ and select the option to ‘Automatically detect settings’ and your work is done!
Unfortunately this functionality can be easily leveraged for nefarious purposes. An attacker could create their own WPAD record (which can be accomplished simply if dynamic DNS is not configured correctly) and redirect traffic through a host that they control. From there they could have full view in to all web-based communication between a client and an Internet-based remote host.
In order to address this security concern, Microsoft has made changes to the way DNS works beginning in Windows Server 2008. DNS in Windows Server 2008 now includes a feature called the global query block list. Essentially this is a list of names that the DNS server will not respond to if queried. By default this list includes two entries; WPAD and ISATAP. You can view this list by executing the following command from an elevated command prompt:
dnscmd /info /globalqueryblocklist
If you are using Windows Server 2008 DNS and you wish to leverage DNS WPAD functionality you must instruct the DNS server to respond to these requests. Simply creating the DNS record by itself is not enough. On Windows Server 2008 you can configure WPAD by creating your DNS record as usual, then remove WPAD from the global query block list by executing the following command from an elevated command prompt:
dnscmd /config /globalqueryblocklist isatap
This command replaces the existing global query block list with only isatap. Remember to execute this command on each DNS server that is authoritative for your zone.
Although not recommended, you can also disable the global query block list functionality altogether by executing the following command from an elevated command prompt:
dnscmd /config /enableglobalqueryblocklist 0
Of course this functionality can be restored by executing the following command from an elevated command prompt:
dnscmd /config /enableglobalqueryblocklist 1
The global query block list functionality is also now included in security update MS09-008 for Windows Server 2003 DNS and WINS servers. This means that everything we’ve discussed here applies to Windows Server 2003 DNS servers with the MS09-008 update installed, with the exception of how the block list is configured. With Windows Server 2003 DNS and the MS09-008 update, management of the global query block list is done through the following registry key:
HKLM\SYSTEM\CurrentControlSet\Services
\DNS\Parameters\GlobalQueryBlockList
If you have already configured WPAD record in DNS, the good news is that if you perform an in-place upgrade to Windows Server 2008, WPAD functionality will not be disabled. The same holds true if you install the security update for MS09-008. Any existing functionality will remain if it was in place prior to the upgrade or update. For additional information on the MS09-008 security update, read this blog post by the Microsoft Security Research and Defense team.
ISA Server Detected Routes…
Another common ISA alert that I often get questions about is the following:
“ISA Server detected routes through the network adapter [network interface] that do not correlate with the network to which this network adapter belongs. When networks are configured correctly, the IP address ranges included in each array-level network must include all IP addresses that are routable through its network adapters according to their routing tables. Otherwise valid packets may be dropped as spoofed. The following ranges are included in the network’s IP address ranges but are not routable through any of the network’s adapters: [address ranges];. Note that this event may be generated once after you add a route, create a remote site network, or configure Network Load Balancing and may be safely ignored if it does not re-occur.
The routing table for the network adapter [network interface] includes IP address ranges that are not defined in the array-level network [network object], to which it is bound. As a result, packets arriving at this network adapter from the IP address ranges listed below or sent to these IP address ranges via this network adapter will be dropped as spoofed. To resolve this issue, add the missing IP address ranges to the array network. The following IP address ranges will be dropped as spoofed: [network object][address ranges];”
Essentially what this alert means is that there is a there is a mismatch between the configuration of the network object in the ISA management console and the routing table of the corresponding network interface. If you are not experiencing any network connectivity issues, or you just made changes to your network configuration, you can probably safely disregard this message. You will not, in most cases, see this message again. If you are seeing this alert on a consistent basis however, you might wish to look at your networking configuration more closely. For the networking to work correctly on the ISA firewall, each network interface should contain routes that correspond to any networks that are reachable through that particular network interface. The network objects in the ISA management console should be configured to include this network and all reachable networks as well. This is critical because if the routing table is configured to include networks that are not included in the ISA network object, the ISA firewall will reject this traffic as spoofed.
There are times when you will have network objects configured with addresses or networks that do not have routes associated with them, and in which case you will see this alert. A case in point would be when a situation arises that you need to exclude a particular address , address range, or subnet that your underlying routing infrastructure routes through another gateway. A common scenario that I have encountered is when registered IP addresses are used for private connectivity over dedicated circuits when connecting to business partners. In this case I would include the IP address(es) of the business partner in the ‘Internal’ network object so that my web proxy and firewall clients would bypass the ISA firewall when communicating with hosts on these networks.
The Perils of Virtualization
With all of the talk recently regarding the availability of Microsoft’s Intelligent Application Gateway (IAG) SP2 and its official support for running in a virtual infrastructure, as well as the support for Microsoft’s Forefront Threat Management Gateway running in virtual environments, there has been a quiet debate running among my colleagues and peers regarding the virtualization of security systems.
I realize that by virtue of the fact that I work for a hardware vendor (Celestix Networks) that my opinions will be considered by many to be biased. Somewhat perhaps, but the fact remains that virtualization, especially the virtualization of a security system, can prove to be extremely risky. Without proper planning, deployment, and ongoing monitoring of the virtual infrastructure the likelihood of a configuration error leading to a total and complete compromise of your security model is extremely high.
For those of you who might think this is just an alarmist’s point of view, I would encourage you to read this story posted recently at InformationWeek.Com. The article talks about a new virtualization security product, but the interesting thing to note here is that it includes a reference to a documented case where the misconfiguration of a virtual network resulted in a sensitive internal database server being connected directly to the public Internet. According to the article, this was put in to operation and was running for some period of time, until someone later discovered the error. Can you imagine if in this case it wasn’t a database server, but a security system responsible for protecting your entire internal private network?
This of course illustrates clearly the point I am trying to make – that there are substantial security risks associated with virtualizing your security infrastructure. Yes, there are some benefits, but I firmly believe that the risks far outweigh the rewards.
Scalable Networking Pack and the ISA Firewall
If you’ve spent any time at all looking at alerts on a Microsoft ISA firewall, you have no doubt seen the following alert:
“The Windows Server 2003 Scalable Network Pack, which is included in Windows Server 2003 Service Pack 2, is enabled. Some ISA Server features will not work properly if a network adapter installed on an ISA Server computer supports and uses the Scalable Network Pack features. For more information, see the Microsoft Knowledge Base article 948496. If you do not have a network adapter that supports the Scalable Network Pack features, you can disable the Windows Server 2003 Scalable Network Pack Enabled alert.”
What does this mean? What is the Scalable Network Pack? Why are we being alerted about it? How did it get installed? Well, let me answer some of those questions for you!
The Microsoft Windows Server 2003 Scalable Networking Pack (SNP) provides support for network acceleration and offloading technologies available in today’s advanced network interface adapters to increase performance and scalability. It was made available as an update to Microsoft Windows Server 2003 SP1, and was later included in Windows Server 2003 SP2. The Scalable Networking Pack consists of the following three new features:
TCP Chimney Offload – Provides for automated, stateful offload of TCP processing to a network adapter that includes a TCP offload engine (TOE). For certain types of network communication (typically large file transfers) TCP Chimney Offload reduces CPU overhead by offloading network packet processing tasks such as packet segmentation and reassembly to the network adapter.
Receiver-side Scaling – Enables the processing of inbound networking traffic to be shared across multiple CPUs. Applications that rely heavy on inbound network communication that run on a multiprocessor system can benefit from Receiver-side Scaling.
NetDMA – Enables memory management efficiencies through direct memory access (DMA), provided the server supports this.
All of these features are wonderful, and have the ability to dramatically increase throughput and substantially reduce processor utilization for network intensive applications. At first you might think that these features would be beneficial to the ISA firewall, but unfortunately, they are not. TCP offloading is designed primarily for large file transfers, and provides little benefit for short-lived conversations that are typical of network traffic handled by the ISA firewall. There are also some incompatibilities as well. For instance, Receiver-side Scaling is incompatible with both NAT and NLB, both commonly configured on an ISA firewall.
It is for these reasons that you will see this alert generated by the ISA firewall, and as a best practice you should disable these features to prevent unexpected behavior. Microsoft Knowledge Base article 948496 includes an update to turn off the default SNP features, as well as instructions on how to disable these features by editing the registry.
ISA Behind a Cisco ASA?
Several people have written to me in response to my earlier blog post ‘HTTP 2.0 Specification?‘ asking why I would have my ISA firewall behind a Cisco ASA. The answer is simple: enhanced security! I am following a long standing security best practice by implementing security in layers; defense in depth. Now, it’s not that the ISA firewall isn’t totally and completely capable of acting as an edge firewall, because it most certainly is. In this case though, I have elected to use an ASA as my edge firewall because I don’t need any real intelligence there. All I want is to do some very simple packet filtering here; basically just filtering out the bulk of the noise from the Internet and allowing my internal ISA firewall, with its advanced deep application layer inspection capabilities and granular user and group based access controls to do the important network communication inspection.
In addition to enhanced security, there are some other benefits to using the ASA (or another firewall) at the network edge. If someone were to circumvent the access controls that are in place on that edge firewall, they would not be able to use those same methods of exploitation on the ISA firewall. If I practice security in layers but deploy the same model firewall at each layer, an attacker can use the same method used to bypass my internal firewalls as they used to bypass my edge firewall.
An additional benefit by using another firewall at the network edge is that by squelching ‘Internet noise’, the logs on the ISA firewall become much more meaningful. It allows me to find important information much more quickly than having to sift through mountains of data this is mostly port scans and probes that occur constantly on the public Internet. This also frees up resources on my ISA firewall that are better put to use on inspecting important traffic.
TCP/IP Fundamentals for Microsoft Windows
I wanted to bring to your attention today a wonderful document authored by Joe Davies called ‘TCP/IP Fundamentals for Microsoft Windows‘. This is an incredible resource, with over 500 pages of detailed information regarding TCP/IP networking in Microsoft Windows. Recently it was updated to include information for Windows Server 2008. What is really amazing is that it is a free download! Joe Davies has authored (and co-authored) many Microsoft Press books on networking, including the TCP/IP Protocols and Services books, Understanding IPv6, Windows Server 2008 Networking and Network Access Protection (NAP), and many more. Joe is also the author of ‘The Cable Guy‘ column that is featured in Microsoft Technet magazine and online.
DNS Resolver Behavior in Windows Vista
Recently while working with one of my customers I came across some odd behavior on a Windows Vista client workstation. We noticed that Windows Vista workstations that were configured as Web Proxy and Firewall Clients only (no default gateway) were unable to access non-web based remote resources (e.g. RDP and FTP) by hostname (single label or fully qualified). They could, however, connect by IP address. The odd thing was that Windows XP clients configured in the identical manner did not exhibit this behavior. Windows XP clients configured as Web Proxy and Firewall Clients without a default gateway could access non-web based remote resources without issue, by hostname or IP. In either case, web proxy communication (HTTP, HTTPS, tunneled FTP) worked perfectly.
Name resolution on the Windows Vista client worked flawlessly, so I opened a case with Microsoft so they could shed some light on this for me. After some additional research on their part they were able to determine that this was expected behavior on Windows Vista. Apparently the DNS resolver in Windows Vista filters out hostnames for destinations that are not reachable from the local host. Without a default gateway, Vista had determined that it couldn’t connect to the resource because it had no route to the remote network (obviously not aware of the Firewall Client) and so communication fails.
The resolution was simple enough…add static routes to any remote internal networks, or if Winsock access to the general internet was required, then add a default gateway. And while you may not encounter this particular scenario in complex, large scale corporate networks, it is fairly common in small, flat networks deployed at many SMB’s. If you are like me and have recommended in these scenarios that you deploy only the Firewall Client, it appears that option is no longer available.
Using DHCP to Assign Static Routes
In order to facilitate testing in a lab environment, I make extensive use of virtual machine technology. Today I employ a combination of VMWare Workstation and VMWare ESX Server in order to simulate distributing computing environments. Included with these technologies is virtual networking, which is tremendously useful when creating complex routed networks in my lab.
There are times when I will add static routes (host routes or network routes) to my local workstation in order to conduct testing. Occasionally there will be times when I forget the ‘-p’ switch to make the route persistent, then I find myself wondering why my connectivity to my test environment isn’t working. Another common scenario is when I grab another system (a laptop or another VM perhaps) and I end up having to manually add more routes before testing can resume.
Recently I decided to make my life easier and use DHCP to add these routes to all of my workstations. Configuring static routes using DHCP is quick and simple. You can add a static route by configuring a DHCP option 249 on Windows Server 2003 DHCP servers, or option 121 on Windows Server 2008 DHCP servers. You can configure the option at the server level, but it is more likely that you will configure the option at the scope level, or as I often do, on a specific reservation.
For demonstration purposes here we’ll make this change at the scope level. To begin, right-click Scope Options and choose Configure Options…. On Windows Server 2003 DHCP servers, select option 249 Classless Static Routes (Option 121 on Windows Server 2008 DHCP servers).
Click Add Route…, then enter the static route you wish to configure.
Now, regardless of which system I log on to, I am assured that I will have connectivity to all of my virtual environments.
Note: DHCP Option 121 is ignored by DHCP clients prior to Windows Server 2008 and Windows Vista. This may not work if you are using a Windows Server 2008 DHCP server to assign networking configuration to these clients. Windows Vista and Windows Server 2008 DHCP clients use both Option 121 and Option 249.
Connect
Richard Hicks' Forefront TMG Blog 