KB2756920 is NOT your friend

Ah, Microsoft patches. Dislike them or hate them (notice there’s no love?), they are a necessary evil in the day to day operation of any Windows server. Under normal circumstances, framework and Windows server patch installation goes fairly well. There are those occasions, however, where all hell breaks loose. This is one of those occasions.

The Issue

After installing the latest round of security patches, our SharePoint 2010 farm went offline. According to our ELMAH error page (if you’re not using ELMAH, you SHOULD be), this is the error we were receiving when hitting any url on the farm. The key bit of info is in bold:

WebHost failed to process a request.

Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/17653682
Exception: System.ServiceModel.ServiceActivationException: The service '/SecurityTokenServiceApplication/securitytoken.svc' cannot be activated due to an exception during compilation.

The exception message is: Method not found: 'System.String System.ServiceModel.Activation.Iis7Helper.ExtendedProtectionDotlessSpnNotEnabledThrowHelper(System.Object)'.. ---> System.MissingMethodException: Method not found: 'System.String System.ServiceModel.Activation.Iis7Helper.ExtendedProtectionDotlessSpnNotEnabledThrowHelper(System.Object)'.
at System.ServiceModel.WasHosting.MetabaseSettingsIis7V2.WebConfigurationManagerWrapper.BuildExtendedProtectionPolicy(ExtendedProtectionTokenChecking tokenChecking, ExtendedProtectionFlags flags, List`1 spnList)
at System.ServiceModel.WasHosting.MetabaseSettingsIis7V2.WebConfigurationManagerWrapper.GetExtendedProtectionPolicy(ConfigurationElement element)
at System.ServiceModel.WasHosting.MetabaseSettingsIis7V2.ProcessWindowsAuthentication(String siteName, String virtualPath, HostedServiceTransportSettings& transportSettings)
at System.ServiceModel.WasHosting.MetabaseSettingsIis7V2.CreateTransportSettings(String relativeVirtualPath)
at System.ServiceModel.Activation.MetabaseSettingsIis.GetTransportSettings(String virtualPath)
at System.ServiceModel.Activation.MetabaseSettingsIis.GetAuthenticationSchemes(String virtualPath)
at System.ServiceModel.Channels.HttpChannelListener.ApplyHostedContext(VirtualPathExtension virtualPathExtension, Boolean isMetadataListener)
at System.ServiceModel.Channels.HttpTransportBindingElement.BuildChannelListener[TChannel](BindingContext context)
at System.ServiceModel.Channels.BindingContext.BuildInnerChannelListener[TChannel]()
at System.ServiceModel.Channels.MessageEncodingBindingElement.InternalBuildChannelListener[TChannel](BindingContext context)
at System.ServiceModel.Channels.BinaryMessageEncodingBindingElement.BuildChannelListener[TChannel](BindingContext context)
at System.ServiceModel.Channels.BindingContext.BuildInnerChannelListener[TChannel]()
at System.ServiceModel.Channels.Binding.BuildChannelListener[TChannel](Uri listenUriBaseAddress, String listenUriRelativeAddress, ListenUriMode listenUriMode, BindingParameterCollection parameters)
at System.ServiceModel.Description.DispatcherBuilder.MaybeCreateListener(Boolean actuallyCreate, Type[] supportedChannels, Binding binding, BindingParameterCollection parameters, Uri listenUriBaseAddress, String listenUriRelativeAddress, ListenUriMode listenUriMode, ServiceThrottle throttle, IChannelListener& result, Boolean supportContextSession)
at System.ServiceModel.Description.DispatcherBuilder.BuildChannelListener(StuffPerListenUriInfo stuff, ServiceHostBase serviceHost, Uri listenUri, ListenUriMode listenUriMode, Boolean supportContextSession, IChannelListener& result)
at System.ServiceModel.Description.DispatcherBuilder.InitializeServiceHost(ServiceDescription description, ServiceHostBase serviceHost)
at System.ServiceModel.ServiceHostBase.InitializeRuntime()
at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustServiceHost.InitializeRuntime()
at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.ServiceHostingEnvironment.HostingManager.ActivateService(String normalizedVirtualPath)
at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath)
--- End of inner exception stack trace ---
at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath)
at System.ServiceModel.ServiceHostingEnvironment.EnsureServiceAvailableFast(String relativeVirtualPath)

 

The Reason

This was due to installing the KB2756920 security patch. According to this: KB280172, the before mentioned hotfix caused the System.ServiceModel and System.ServiceModel.WasHosting assemblies to become out of sync, and surprise! They depend on each other.

The Fix

KB280172 recommends that another hotfix be installed to bring both of these assemblies into parity. I chose to simply uninstall the offending patch using ControlPanel > Programs > Programs and Features > Installed Updates. Uninstalling does require a reboot, but after this the farm was back online and no other issues have been encountered.

Reason for not installing the other hotfix: I didn’t want to introduce yet another untested hotfix into a production environment. You all know this (right?!?), but I’ll say it anyway: as a best practice, test these hotfixes on your staging or QA stack before moving into the production environment.

So, to sum up: no new hotfix to fix the bad hotfix until the two hotfixes are hotfixed together.

For good measure, I’m going to say hotfix one more time: hotifix.

A Note on the Side

We had originally installed the latest round of patches on our staging farm servers and ran through our normal tests. A few weeks went by before moving them to the production servers, however. This has brought up a vulnerability in our patching process; moving forward we need to double check that what is installed and tested on staging is exactly what is moved to production.

When triaging the issue, it seemed logical that something went to production that wasn’t installed on stage. But how do you easily reconcile what was patched on one server vs the other? Well there’s this nifty little command line you can run that will export all of your server patches to a convenient CSV file:

wmic qfe get /format:csv > C:\temp\foo.csv

Run this in a command window (as administrator) and it will export the list to CSV. Comparison between two servers at this point is as simple as importing to Excel or a DB table. In this case I knew the offending patch number, so it was a simple search in both files.

I’m not a sys/admin on a typical day, so I’d be interested in hearing if there are better ways to reconcile what is installed on two different Windows servers.

Happy hotfixing!
Jim

Advertisements
Tagged ,

The power compels you ...

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: