Running vCenter 5.x with SQL 2012 AlwaysOn Availability Groups

After proudly starting the listener on our shiny new SQL 2012 AlwaysOn cluster, I was very eager to get vCenter moved off the brave little single-point-of-failure that is our current SQL server (a 2008 VM sitting in the virtual environment itself). I had done some research ahead of time and thought that AlwaysOn was at least sort-of supported by VMWare for protecting vCenter workloads. However, in my haste to play with a fancy new toy, I must have missed the plethora of blog posts indicating that either a) It's not actually supported at all, or b) Only Failover Clustering (shared storage) - not Availability Groups (non-shared storage) - are supported. And if you are about to do what many have done on the forums and suggest KB1037959 as evidence that they ARE supported, think again - that article is referencing support for running various clustered workloads on vSphere, not running your vCenter DB on clustered systems. Outside of a vague mention of AlwaysOn as a possible third party clustering solution to replace vCenter Heartbeat (e.g. "Best effort support"), I haven't been able to find anything official one way or the other.

But the AlwaysOn cluster was ready to go and if no one is going to tell me explicitly that I can't do it - well, that's basically an open invitation.

Pay Me Now, or Pay Me Later: DNS Edition

I hate most low-cost hosting providers. I've rarely have a great experience with one, but Web and DNS hosting providers are usually already established when I start working with a client, and migrating to another provider is typically not marked as a high priority project. I might change my mind on that after working with a client who used iPower and suddenly lost all public DNS resolution one day.

We called up iPower twice and spoke to two oblivious techs who spouted off nonsense and promised to call back after looking in to it further. Finally, on the third call a person finally told us why the client's SOA record wasn’t propagating (causing their domain name not to resolve anywhere, so no website, no e-mail, etc. for two days). Turns out they had failed to respond to their “domain verification” e-mail, which with most providers means nothing – they are just required by ICANN to prompt you to update it once a year, but most just don’t do anything if you ignore the e-mail. iPower took it a step further and deactivated their domain when there was no response.

Paul: So, if I understand you correctly, you all sent an e-mail to the registrant e-mail address to verify the registrant details. The e-mail address was wrong, so they didn't see it. So the domain was deactivated?
Paul: So you checked to see if you had the right e-mail address by ... sending an e-mail? And if the address was not correct, you just assumed we didn't need the domain?
Deepika B: Yes, you are right.
Paul: Fantastic, well done.
Deepika B: Thank you!

Bravo, iPower. Bravo.

Enabling Office 365 Message Encryption

Back in November of 2013, Microsoft announced Office 365 Message Encryption (OME) as a way to protect the contents of outbound messages to people in other organizations. This excellent update helps fence in it's predecessor, Exchange Hosted Encryption (EHE), to the Office 365 branding and management. Users of EHE had to work through more cumbersome configuration steps and were required to purchase it open volume licensing, whereas OME comes included as part of Azure Active Directory Rights Management  (AADRM) in the E3 and E4 packages, or purchased as an add-on to other enterprise SKUs with the normal subscription licensing.

SharePoint Downloads Interrupted for Large Files

While we primarily use SharePoint Online (2013) for fairly small documents - the largest are still less than 5MB - I recently decided to start uploaded recorded team meetings from Lync 2013. The videos are about 30 minutes in length and end up being around 40-60MB in size. Although the upload runs fine, users were reporting issues when trying to stream the files or attempting to download it. The stream would simply stop and the downloads fail with a message that the download was "Interrupted". We replicated this behavior across 4 different locations with the same result - at some random point in the download, it would fail.

Repairing Mailbox Corruption in Exchange 2010

I recently got through recovering an SBS 2011 server after Active Directory face-planted in the middle of a workday. When I say recover, I mean I repeated the entire migration, using a cleaned up secondary DC - it was a fun weekend (expect another post about that experience). Although I thought we were in the clear, I got a call from the client about 24 hours after we had verified everything was working. He indicated that his iPhone had suddenly stopped receiving mail in the inbox (calendar, contacts, sent items were still fine) and throws up an error after spinning in circles for a few minutes that it "cannot connect to mail server".

Failure of vShield Edge NAT/VPN Traffic Post-5.1 Upgrade

UPDATE: Turns out this is a known issue during the 1.5 > 5.1 VSM upgrade and a fix should be released in an upcoming patch.

That's about the shortest title I could think of to be descriptive of this issue. TLDR is that NAT rules on vShield Edge appliances appear to be causing unexpected behavior on VPN traffic after a vCloud upgrade from 1.5 to 5.1.

Background: We recently upgraded from 1.5 to 5.1. For most of our vDCs, we simply have a single vSE/Routed network that connects a private subnet to a "WAN" network and pulls a public IP from a pool. We forward (NAT) and allow (firewall) selected ports (e.g. 3389 for RDP) to virtual machines. Most of these networks also have a site-to-site VPN tunnel with a physical firewall across the internet. After the upgrade, we went and converted our rules to match on original IP and then enabled "multiple interfaces" - effectively taking them out of compatibility mode. Everything looked good (even for the vSE devices still in compatibility mode)

Issue: We first noticed this when a client reported that they could not access a virtual machine via RDP using it's internal (VSE protected) IP across a VPN tunnel, but could access the VM via RDP using it's public hostname/IP address. We allow all traffic across the VPN (firewall has an any:any rule for VPN traffic). When we logged in to troubleshoot (simply thinking the VPN was down), we found that we could connect to any port on the remote VM across the VPN tunnel except 3389. I could ping from the local subnet to the troubled VM on the vApp network with no problem. I could connect to other ports that were open on the remote VM with no problem. I could not connect to 3389 across the VPN.

We thought it might be isolated, but found the issue on every VSE we have: If there existed a DNAT rule to translate inbound traffic for a particular port, that port would be unresponsive when traffic traversed the VPN tunnel destined for the target of the DNAT rule.

While vCloud Director doesn't show anything strange in the firewall section of vSE configuration, if you log in to vShield Manager and look at the firewall rules there, a "Deny" rule with the private/internal/translated IP is added for any NAT rule that exists:

This, I'm assuming, is for security reasons during the upgrade but it does not show up in vCloud Director (thus our confusion). After taking our appliances out of compatibility mode post-upgrade, the rules were still there.

Solution:  After the vSE is out of compatibility mode (see pg. 49 of the vCD 5.1 Install Guide), re-apply the service configuration (Right-Click vShield Edge Appliance in vCloud Director and select "Re-Apply Service Configuration"). You can also re-deploy the appliance or add an arbitrary rule to the firewall list - both appear to have the same effect.

Weathering the Cloud: Moving your service company ahead in the age of SaaS

With phrases like "cloud computing" making their way deeper in to the everyday dialect of non-technical business managers, it no longer comes as a surprise when decision makers approach me about cutting costs by using "the cloud". They may not know exactly (or even vaguely) what that means, but they know it might save them money, so the door is wide open for a discussion about _aaS ("Something" as a Service - Software, Infrastructure, Security, and so on).

This would appear to be a huge sales opportunity for I.T. service companies, but my observations of several have shown that adoption can be slow for a number of reasons: