KB2756920 is NOT your friend

Ah, Microsoft patches. Dislike them or hate them (notice there’s no love?), they are a necessary evil in the day to day operation of any Windows server. Under normal circumstances, framework and Windows server patch installation goes fairly well. There are those occasions, however, where all hell breaks loose. This is one of those occasions.

The Issue

After installing the latest round of security patches, our SharePoint 2010 farm went offline. According to our ELMAH error page (if you’re not using ELMAH, you SHOULD be), this is the error we were receiving when hitting any url on the farm. The key bit of info is in bold:

WebHost failed to process a request.

Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/17653682
Exception: System.ServiceModel.ServiceActivationException: The service '/SecurityTokenServiceApplication/securitytoken.svc' cannot be activated due to an exception during compilation.

The exception message is: Method not found: 'System.String System.ServiceModel.Activation.Iis7Helper.ExtendedProtectionDotlessSpnNotEnabledThrowHelper(System.Object)'.. ---> System.MissingMethodException: Method not found: 'System.String System.ServiceModel.Activation.Iis7Helper.ExtendedProtectionDotlessSpnNotEnabledThrowHelper(System.Object)'.
at System.ServiceModel.WasHosting.MetabaseSettingsIis7V2.WebConfigurationManagerWrapper.BuildExtendedProtectionPolicy(ExtendedProtectionTokenChecking tokenChecking, ExtendedProtectionFlags flags, List`1 spnList)
at System.ServiceModel.WasHosting.MetabaseSettingsIis7V2.WebConfigurationManagerWrapper.GetExtendedProtectionPolicy(ConfigurationElement element)
at System.ServiceModel.WasHosting.MetabaseSettingsIis7V2.ProcessWindowsAuthentication(String siteName, String virtualPath, HostedServiceTransportSettings& transportSettings)
at System.ServiceModel.WasHosting.MetabaseSettingsIis7V2.CreateTransportSettings(String relativeVirtualPath)
at System.ServiceModel.Activation.MetabaseSettingsIis.GetTransportSettings(String virtualPath)
at System.ServiceModel.Activation.MetabaseSettingsIis.GetAuthenticationSchemes(String virtualPath)
at System.ServiceModel.Channels.HttpChannelListener.ApplyHostedContext(VirtualPathExtension virtualPathExtension, Boolean isMetadataListener)
at System.ServiceModel.Channels.HttpTransportBindingElement.BuildChannelListener[TChannel](BindingContext context)
at System.ServiceModel.Channels.BindingContext.BuildInnerChannelListener[TChannel]()
at System.ServiceModel.Channels.MessageEncodingBindingElement.InternalBuildChannelListener[TChannel](BindingContext context)
at System.ServiceModel.Channels.BinaryMessageEncodingBindingElement.BuildChannelListener[TChannel](BindingContext context)
at System.ServiceModel.Channels.BindingContext.BuildInnerChannelListener[TChannel]()
at System.ServiceModel.Channels.Binding.BuildChannelListener[TChannel](Uri listenUriBaseAddress, String listenUriRelativeAddress, ListenUriMode listenUriMode, BindingParameterCollection parameters)
at System.ServiceModel.Description.DispatcherBuilder.MaybeCreateListener(Boolean actuallyCreate, Type[] supportedChannels, Binding binding, BindingParameterCollection parameters, Uri listenUriBaseAddress, String listenUriRelativeAddress, ListenUriMode listenUriMode, ServiceThrottle throttle, IChannelListener& result, Boolean supportContextSession)
at System.ServiceModel.Description.DispatcherBuilder.BuildChannelListener(StuffPerListenUriInfo stuff, ServiceHostBase serviceHost, Uri listenUri, ListenUriMode listenUriMode, Boolean supportContextSession, IChannelListener& result)
at System.ServiceModel.Description.DispatcherBuilder.InitializeServiceHost(ServiceDescription description, ServiceHostBase serviceHost)
at System.ServiceModel.ServiceHostBase.InitializeRuntime()
at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustServiceHost.InitializeRuntime()
at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.ServiceHostingEnvironment.HostingManager.ActivateService(String normalizedVirtualPath)
at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath)
--- End of inner exception stack trace ---
at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath)
at System.ServiceModel.ServiceHostingEnvironment.EnsureServiceAvailableFast(String relativeVirtualPath)


The Reason

This was due to installing the KB2756920 security patch. According to this: KB280172, the before mentioned hotfix caused the System.ServiceModel and System.ServiceModel.WasHosting assemblies to become out of sync, and surprise! They depend on each other.

The Fix

KB280172 recommends that another hotfix be installed to bring both of these assemblies into parity. I chose to simply uninstall the offending patch using ControlPanel > Programs > Programs and Features > Installed Updates. Uninstalling does require a reboot, but after this the farm was back online and no other issues have been encountered.

Reason for not installing the other hotfix: I didn’t want to introduce yet another untested hotfix into a production environment. You all know this (right?!?), but I’ll say it anyway: as a best practice, test these hotfixes on your staging or QA stack before moving into the production environment.

So, to sum up: no new hotfix to fix the bad hotfix until the two hotfixes are hotfixed together.

For good measure, I’m going to say hotfix one more time: hotifix.

A Note on the Side

We had originally installed the latest round of patches on our staging farm servers and ran through our normal tests. A few weeks went by before moving them to the production servers, however. This has brought up a vulnerability in our patching process; moving forward we need to double check that what is installed and tested on staging is exactly what is moved to production.

When triaging the issue, it seemed logical that something went to production that wasn’t installed on stage. But how do you easily reconcile what was patched on one server vs the other? Well there’s this nifty little command line you can run that will export all of your server patches to a convenient CSV file:

wmic qfe get /format:csv > C:\temp\foo.csv

Run this in a command window (as administrator) and it will export the list to CSV. Comparison between two servers at this point is as simple as importing to Excel or a DB table. In this case I knew the offending patch number, so it was a simple search in both files.

I’m not a sys/admin on a typical day, so I’d be interested in hearing if there are better ways to reconcile what is installed on two different Windows servers.

Happy hotfixing!

Tagged ,

The Curious Case of Chrome, Content-Disposition and the Comma.

Chrome just works. It’s generally cleaner, faster and it simply feels better than IE, FireFox and Safari. So when I heard a client tell me that something wasn’t working specifically in Chrome, I thought nah, that can’t be right!

Well, turns out the client was right. File downloading in Chrome was apparently not working! So what happened?

The Issue: 

The site I am working on has a Print to PDF feature that will convert certain content types within the site to a printer friendly HTML format using a special CSS. The results of this HTML are stuffed into a PDF using a 3rd party component, then the bits are shipped the  to the browser as a file attachment. This all occurs in a single runtime operation and is pretty standard stuff.

When the bytes are sent to the browser, to get the file to download automatically we’re setting the content disposition as such:

Content-Disposition: attachment; filename=[content title].pdf

The issue lies in the replacement of the file name. For sake of argument lets say the content type is a forum post, and we’re using the title of the post as the file name. Let’s also say the title of the forum post is “Foo, How it Compels You”.

The resulting content disposition header would be:

Content-Disposition: attachment; filename=Foo, How it Compels You.pdf

In Chrome, when the content hits the client side, literally nothing happens. Network traffic shows the request was made, and data comes back, however no download occurs. On closer inspection you may get this message:

Content-Disposition headers received

This is happening because Chrome implements a stricter version of the content disposition header. Chrome pukes on the comma and considers this a possible security risk. Not sure how a comma can compromise your security, but I’ve heard stranger things.

The Fix:

Knowing that it’s the comma causing the problem, the fix was obvious: replace the comma with a space. Done.

It’s also worth noting that as a precaution additional character parsing was implemented to replace invalid Windows file characters with spaces, then replace any double spaces with single spaces.  In hind sight this should have been in place from the start. But that’s hind sight for you.


Content database restored and the site doesn’t work. What gives?

A simple procedure gone wrong.

When do simple procedures go wrong? Never …

Recently an attempt was made to restore a SharePoint 2007 content database from our production system to our staging system. The restore was necessary to fully test new customizations with an up-to-date version of the content database from production. Seems reasonable, no?

The process went a little like this:

  • Backup the content database on the production SQL Server
  • Copy it over to the staging SQL Server
  • Restore it over the top of the staging content database
  • Re-attach the content database using Central Admin:
    • Go to Application Management -> Content Databases
    • Select your Web Application
    • Click on the existing content database, check “remove” and hit OK
    • Then re-add the content database

After this restore the site did not come back up. This is a fairly simple procedure and in most cases should yield a functional SP site.  After some investigation, we noticed the following:

The current number of sites is zero? Abuuuuuuuuh?

So what happened here?

After hours of digging, it turns out our staging farm had TWO instances of the same content database. The first instance, let’s call it “foo1.foocompelsyou.com”, was the original staging SP site, which was at one point restored from the production content database using the same procedure outlined above. The second site, let’s call it “foo2.foocompelsyou.com”, was a second instance of the same site, which ORIGINALLY was created via a stsadm backup/restore operation.

Restoring the backup of production’s content DB over foo2’s content DB made the Site Ids and Database Ids in f001 and f002 identical. The farm knows that these ids in the attempted restore of foo2 is already used by another site, hence why the current number of sites is listed as 0.

The fix is in.

Ok, so there are two identical content databases on my farm server with the same ids. Now what? Well, there’s not much you can do. I thought some behind-the-scenes trickery would work, such as manually updating site id and database id values. There are various posts on the interwebs that describe nefarious procedures to manually correct the problem, none of which really fit the bill.

Ultimately this did the trick:

  • Restore the production content DB over the top of foo1.foocompelsyou.com’s content db.
  • Using central admin, remove any existing content databases from foo2.foocompelsyou.com.
  • Again using central admin, create a new content database for foo2.foocompelsyou.com.
  • STSADM -o backup -url http://foo1.foocompelsyou.com -filename c:\temp\foo1.bak
  • STSADM -o restore  http://foo2.foocompelsyou.com -filename c:\temp\foo1.bak

And *poof*, foo2 is back in business.

In an ideal world …

The second foo site on staging, foo2.foocompelsyou.com, is essentially a deployment/QA test site. If there were a completely separate farm for foo2, this would not have been an issue at all. In our case there’s not enough spare iron or virtual space to create a new farm instance.

Well, that’s all I have on this one. Happy SharePointing.

– Jim

OS X: Get more from Finder

There’s nothing worse than using a file browsing tool that requires 12 clicks when 1 will do. Finder is in this bucket. I find it to be clunky and down right frustrating. It’s even more frustrating when flipping back and forth between OS X host and Windows 2008 guest. I keep asking myself: “Windows Explorer works this way, why doesn’t finder?”

Well, a colleague of mine recently sent this life line: How to make the Mac OSX Finder Suck Less. Use these extremely handy tips to custom-tune OS X Finder so that it’s actually useful. Developers: rejoice!


Visual Studio 2010 Database Projects: Why Use Them?

Visual Studio 2010 database projects can help you. No, seriously.

Over the past year or so I’ve been using VS 2010 database projects off and on to deal with SQL Server 2008 schema management. I can’t honestly say that it’s been a painless transition, and convincing other developers to use it is, well, as close to a fools errand as you can get. That being said, there are a lot of great reasons to use these things.

In this post I’ll go into some of the reasons you should use database projects.

First of all, what the heck are they?

VS 2010 database projects attempt to bring in everything that is in SQL Server Management Studio into Visual Studio and create a fully integrated SQL development experience. You can create a database project and pull in an existing database, or you can tee one up from scratch and use Visual Studio to create everything you would normally do in SSMS.

Either way, the result is all of your SQL objects are sitting on your file system, nicely  integrated into a convenient Visual Studio project that exists within a Visual Studio solution. It’s a pretty decent setup, really. Take a look at the screen shot below:

Database Projects - Solution Explorer

Showing the stored procedures list in the database project solution explorer.

As you can see, everything is laid out in the solution explorer in a fairly straight forward structure. Each schema has it’s own folder, and all tables, views, procedures, UDFs, etc end up within the schema folder. With only a few exceptions, each SQL object will have it’s own separate file on the file system. This particular database project was populated by importing a SharePoint 2007 content database, and low and behold is riddled with build warnings … 🙂

What I’ve found most useful is to import existing databases into Visual Studio. Setting up a new database in Visual Studio from scratch is a bit too clunky. In all honesty: you’ll probably spend most of your time editing your database in SSMS, then pulling your changes into the database project.

So why then use two tools (SSMS and VS)? Well, there are some really great features built in to the Visual Studio database project that make a fairly compelling case for using them along side SSMS. Some of my favorites are below.

Why Use Database Projects?

I don’t suppose “because I told you to” is good enough?

Build-Time Validation

One of my favorite features is build-time checking and validation of the SQL objects in your solution. For example: if a stored procedure is calling a view that no longer exists, the compiler will tell you.

Here’s a super simple example:

Compilation Example: Unknown Type Error

VS data projects will compile your SQL objects and check for errors. This is an example of a datatype error. Adding this type to your project will remove the error.

The compiler checks for quite a lot of the every day SQL errors you’ll run into when developing SQL schemas:

  • References to missing tables, views, UDFs, columns and the like
  • Cross database references (these are frowned upon)
  • Functions with missing/incorrect parameters
The compiler doesn’t check for everything, though. For example, let’s say you have a procedure that inserts a new record into a table, the values of which come from several procedure parameters. If the parameter is improperly typed, the compiler won’t necessarily catch this.
Additionally, if there is a procedure that references a #TEMP table in a procedure that is created by another, a boat load of build warnings pop up. This is difficult to factor out when building your solution. Case in point:
Database Project: Stored Procedure Warnings

A ton of warnings will come up if this #TABLE ref is not created in the procedure.

Incidentally: even if you choose not to use database projects in the future, this is a super way to validate your existing schemas. How many views and procedures reference tables/columns or other objects in your database that no longer exist? Import your schema into a new Visual Studio project and find out. You might be surprised at what you find.

Schema Comparison: FINALLY

Yes, there is now a schema comparison tool built in to Visual Studio. Wait … don’t get too excited. While it does it’s job well enough, it doesn’t hold a candle to Red Gate’s SQL Compare. Here’s the real cost/benefit analysis for you: do you have licenses to Visual Studio edition that includes database projects? Then VS schema compare is free. Red Gate, however, will still set you back another $500. Don’t get me wrong, they’ll happily take your money. And given Red Gate’s stellar usability, I wouldn’t blame you for whining to the pointy haired boss about purchasing a license, anyway.

You can find this tool under the Data menu. It works about the same as Red Gate’s comparison tool or any other: you compare one database to another, see what’s different, and it can manage the upgrade/downgrade scripts auto-magically. Or you can tell it to generate an upgrade script that can be stored in the ‘scripts’ folder of your project. Some additional cool things you can do:

  • Compare Data Project to Data Project.
  • Compare Data Project to Database, and vise versa.
  • Set a plethora (yes, I said plethora) of comparison options.
  • Choose which SQL objects to ignore. I commonly ignore Users, SQL files and Database Options.
Here’s a screen cap of the comparison tool:
Schema Comparison Tool

The schema comparison tool is comparing two completely unrelated databases. Typically not something you’ll do, but it shows a lot of differences.

Source Control Your Database Along With Your Solution, Or Else

Is your database schema checked in to source control? No? Well now you have one less excuse. Creating a database project in your solution and keeping it up to date practically gives schema source control away for free, especially if you’re using TFS. If you’re using SVN, you could use AnkhSVN (though I haven’t tried it myself), and Tortoise SVN works as well. As mentioned before, you can always shell out cash money for other solutions … *cough* Red Gate *cough*. But why?

As previously mentioned, you’re probably going to continue to use SSMS to do most of your database editing. This has worked for me in the past, and I’ve utilized the Schema Comparison tool to pull changes back in to the database project with relative ease.

If you’re using SVN, here’s a tip: SVN ignore the following file extensions and folders:

  • sql and obj folders: These are analogous to the bin and obj folders of any other code solution in VS.
  • .dbmldb: This is a file used to cache the database model for build performance. It gets big, and is a per-user file. Don’t check it in.
  • .schemaview: Again, this is a per-user file and shouldn’t be checked in. It’s used by the schema viewer tool.

Schema Viewer

I love this tool. It’s arguably one of the most useful feautres. It’s similar to the Solution Explorer, though it’s strictly for browsing your database. It includes a lot of sweet little tools, like this ‘View Dependencies’ option:

Schema View: View Dependencies

Want to view what objects are dependent on a view? Open Schema Viewer, right click and hit View Dependencies. Done deal.

Click on this little gem, and you’ll get a list of all objects that depend on the “Docs” view:

Schema Viewer: Dependency Viewer

This lists all objects that reference the “Docs” view.

There is also another folder that shows all of the objects that are referenced by the Docs view. Some of the usual caveats with this tool: if you have build errors, it’s possible the objects with errors fall out of the referenced by section. Also, dependencies in dynamically generated SQL will not show up here. All in all, the view dependencies is an excellent tool that can help you with refactoring your database or just help you trace down the dependency tree when debugging.

Speaking of refactoring:

Schema Viewer: Refactoring

Need to rename a SQL object? No problem. Right click in the schema viewer, go to Refactor, and hit Rename.

This is the good stuff. I particularly like “Rename” (which I do all of the time), and “Move to Schema”, which as you work on a database over the course of a year comes in super handy.


While I’ve only scratched the surface on database projects, its easy to see some of the gains to be had by using them. I haven’t gotten into some of the issues, however, and will post on them at some point. In the mean time, if you haven’t already, I encourage you to test one out.

Happy databasing!

Tagged , ,

It’s April 2012 and this is my first blog post.

As with most of today’s news, the important information is contained in the headline. But if you haven’t figured it out already, this is literally my first blog post, ever.

Why have I waited so long? I’ve never felt a true passion to lay it all out online before. It takes time, effort, and a whole lot of writing skill to put together a post that is readable, interesting and most of all: accurate.  Accuracy is fairly important in our extremely technical field. There’s nothing worse than getting it wrong in a public venue.

Why am I blogging now? There are many answers to this, including several favorites, like:

  • “I’ve always wanted to …”
  • “It’d be a great way to organize my thoughts and document my learnings …”
  • “Being seen as an industry leader would be super green …”
  • “Everyone else is doing it, why not me?”
Want to know the real reason?
  • “Hey Jim, blogging is a requirement of your job.”

Oh, snap. And there you have it. Am I bitter? No, not in the slightest. Getting a shove is probably the best (and only) way I’d have gone as far as I have today.

So, if you’re still reading (you are still reading, right?), I encourage you to check back from time to time. Some of the more relevant things I’ll be blogging about:
  • Core technologies, such as: C#, SQL Server, and SharePoint 2007/2010.
  • Excel customization, generation, and it’s inappropriate use as a database. Seriously, people.
  • All the TLA’s we’ve come to know and respect: ORM, OOP, MVC, BYOB
  • And who knows? Suggest something.
I may also from time to time do some write ups on the serious issues of today, such as:
  • We’ll take a look at the differences between C# and Dflat.
  • SharePoint 2007: who builds this stuff, anyway? Let’s go behind the scenes and get some answers.
  • Why programmers are people, too.
  • Unit conversions: How many lines of code make up an ounce, and how many ounces fit in a decaliter?
  • And many, many others …

And with all of that, let the blathering commence. Read on, not because you want to, and certainly not because you’ll learn anything of consequence. Read on because …

The power of programming compels you, of course!